# Concurrency violation on interplay between Distributed and Base.Threads

**URL:** <https://discourse.julialang.org/t/concurrency-violation-on-interplay-between-distributed-and-base-threads/47080>\
**Category:** Julia at Scale\
**Tags:** question\
**Created:** [September 22, 2020, 4:08pm UTC](https://discourse.julialang.org/t/concurrency-violation-on-interplay-between-distributed-and-base-threads/47080 "2020-09-22T16:08:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jonas-schulze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonas-schulze/32/9228_2.png) [@jonas-schulze](https://discourse.julialang.org/u/jonas-schulze)\
**Post date:** [September 22, 2020, 4:08pm UTC](https://discourse.julialang.org/t/concurrency-violation-on-interplay-between-distributed-and-base-threads/47080/1 "2020-09-22T16:08:10Z")

</div>

I’m working on a distributed pipeline algorithm that uses several stages per worker process. IIRC tasks cannot hop between threads once they’ve been scheduled. Since I want my stages to potentially run in parallel, I tried to create non-sticky tasks by chaining each `D.@spawnat` with a `T.@spawn`. However, this setup keeps failing/crashing and I don’t understand why.

I boiled it down to a minimal example:

```julia
using Distributed, Base.Threads

const D = Distributed
const T = Threads

pids = addprocs(10)
wids = repeat(pids, inner=2)

conns = map(RemoteChannel, wids)
fst = first(conns)
lst = RemoteChannel()
push!(conns, lst)

@everywhere begin
    function stillepost(i, prev, next)
        message = take!(prev)
        put!(next, message)
        @info "Player $i done"
    end
end

# Select how to orchestrate the players:
v = :v2 # Change me!
players = []
for i in 1:length(wids)
    w = wids[i]
    c1 = conns[i]
    c2 = conns[i+1]
    v == :v1 && (p = D.@spawnat w stillepost(i, c1, c2))
    v == :v2 && (p = D.@spawnat w fetch(T.@spawn stillepost(i, c1, c2)))
    push!(players, p)
end

# Start the game:
game = @async begin
    m1 = "gibberish"
    put!(fst, m1)
    m2 = take!(lst)
    @info "'$m1' turned into '$m2'; well done!"
end

# Demonstate success/failure:
if v == :v1
    # This one "works", but it is not my intended/preferred orchestration layout:
    wait(game)
elseif v == :v2
    # Player 2 fails, mostly due to concurrency violations:
    fetch(players[2])
end

```

How do I fix that? Am I holding it wrong?

Due to the double indirection even otherwise obvious things like “unknown call to stillepost” (which is fixed by know 🙈) are not that easy anymore. I find the debugger very hard to use in this distributed/parallel setting, any help or suggestions with this regard would also be greatly appreciated. 🙂

```julia
Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.5.0)
  CPU: Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 4
  JULIA_PROJECT = @.

```

---

<div class="post-metadata">

**Author:** ![jonas-schulze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonas-schulze/32/9228_2.png) [@jonas-schulze](https://discourse.julialang.org/u/jonas-schulze)\
**Post date:** [September 22, 2020, 6:53pm UTC](https://discourse.julialang.org/t/concurrency-violation-on-interplay-between-distributed-and-base-threads/47080/2 "2020-09-22T18:53:30Z")

</div>

Interestingly, if I use `wids = fill(1, 10)` both variants work just fine. On my exact use case, I get a deadlock. So there is at least one bug more I need to understand and fix.

---

<div class="post-metadata">

**Author:** ![jonas-schulze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonas-schulze/32/9228_2.png) [@jonas-schulze](https://discourse.julialang.org/u/jonas-schulze)\
**Post date:** [September 22, 2020, 7:01pm UTC](https://discourse.julialang.org/t/concurrency-violation-on-interplay-between-distributed-and-base-threads/47080/3 "2020-09-22T19:01:40Z")

</div>

In a fresh REPL, this is the output for version 2 selected:

```julia
julia> include("stillepost.jl")
[ Info: Player 1 done
ERROR: LoadError: On worker 2:
TaskFailedException:
concurrency violation detected
error at ./error.jl:33
concurrency_violation at ./condition.jl:8
assert_havelock at ./condition.jl:25 [inlined]
assert_havelock at ./condition.jl:48 [inlined]
assert_havelock at ./condition.jl:72 [inlined]
wait at ./condition.jl:102
wait_for_conn at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:193
check_worker_state at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:168
send_msg_ at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/messages.jl:176
send_msg at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/messages.jl:134 [inlined]
#remotecall_fetch#143 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:389
remotecall_fetch at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:386
#remotecall_fetch#146 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:421
remotecall_fetch at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:421
call_on_owner at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:494
put! at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:595 [inlined]
stillepost at /Users/jonas/.../stillepost.jl:18
#3 at ./threadingconstructs.jl:169
wait at ./task.jl:267 [inlined]
fetch at ./task.jl:282 [inlined]
#2 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/macros.jl:87
#103 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:290
run_work_thunk at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:79
run_work_thunk at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:88
#96 at ./task.jl:356
Stacktrace:
 [1] #remotecall_fetch#143 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:394 [inlined]
 [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Distributed.RRID) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:386
 [3] #remotecall_fetch#146 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:421 [inlined]
 [4] remotecall_fetch at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:421 [inlined]
 [5] call_on_owner at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:494 [inlined]
 [6] fetch(::Future) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:533
 [7] top-level scope at /Users/jonas/.../stillepost.jl:48
 [8] include(::String) at ./client.jl:457
 [9] top-level scope at REPL[1]:1
in expression starting at /Users/jonas/.../stillepost.jl:44

```

Is this a mistake on my side or a might this be a bug in Julia?

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [September 22, 2020, 8:44pm UTC](https://discourse.julialang.org/t/concurrency-violation-on-interplay-between-distributed-and-base-threads/47080/4 "2020-09-22T20:44:59Z")

</div>

Does look like a bug in Distributed. Can you open an issue?

I don’t understand how that can cause data corruption, but `wait_for_conn` waiting on the non-threadsafe Worker.c\_state is definitely very smelly.

---

<div class="post-metadata">

**Author:** ![jonas-schulze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonas-schulze/32/9228_2.png) [@jonas-schulze](https://discourse.julialang.org/u/jonas-schulze)\
**Post date:** [September 23, 2020, 7:24am UTC](https://discourse.julialang.org/t/concurrency-violation-on-interplay-between-distributed-and-base-threads/47080/5 "2020-09-23T07:24:03Z")

</div>

Thanks for the quick response! I opened an issue on GitHub.  
[https://github.com/JuliaLang/julia/issues/37706](https://github.com/JuliaLang/julia/issues/37706)