# Concurrency violation during garbage collections?

**URL:** <https://discourse.julialang.org/t/concurrency-violation-during-garbage-collections/58391>\
**Category:** General Usage\
**Created:** [April 1, 2021, 8:43pm UTC](https://discourse.julialang.org/t/concurrency-violation-during-garbage-collections/58391 "2021-04-01T20:43:28Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [April 1, 2021, 8:43pm UTC](https://discourse.julialang.org/t/concurrency-violation-during-garbage-collections/58391/1 "2021-04-01T20:43:28Z")

</div>

Really, I guess my question is should I open a bug against Julia for this? It doesn’t appear to be causing issues, and I expect that this will only happen during development. First the stack trace:

```julia
error in running finalizer: ErrorException("concurrency violation detected")
error at ./error.jl:33
concurrency_violation at ./condition.jl:8
assert_havelock at ./condition.jl:25 [inlined]
assert_havelock at ./condition.jl:48 [inlined]
assert_havelock at ./condition.jl:72 [inlined]
notify at ./condition.jl:126
#notify#515 at ./condition.jl:124 [inlined]
notify at ./condition.jl:124 [inlined]
notify at ./condition.jl:124 [inlined]
send_del_client at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:265
finalize_ref at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:97
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:278
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:365
run_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:394
jl_gc_collect at /buildworker/worker/package_linux64/build/src/gc.c:3260
maybe_collect at /buildworker/worker/package_linux64/build/src/gc.c:880 [inlined]
jl_gc_pool_alloc at /buildworker/worker/package_linux64/build/src/gc.c:1204
poptask at ./task.jl:755
wait at ./task.jl:763 [inlined]
uv_write at ./stream.jl:992
unsafe_write at ./stream.jl:1064
unsafe_write at /home/josh/.julia/packages/HTTP/cxgat/src/ConnectionPool.jl:172 [inlined]
unsafe_write at /home/josh/.julia/packages/HTTP/cxgat/src/Streams.jl:98
unsafe_write at ./io.jl:646 [inlined]
write at ./io.jl:687 [inlined]
#39 at /home/josh/me/julia/ImageManagement/src/server/images/http_response.jl:93
#open#317 at ./io.jl:330
open##kw at ./io.jl:328 [inlined]
http_send_file at /home/josh/me/julia/ImageManagement/src/server/images/http_response.jl:58 [inlined]
get_image at /home/josh/me/julia/ImageManagement/src/server/images/get_image.jl:113
handle_req at /home/josh/me/julia/ImageManagement/src/server/images/server.jl:47
#35 at /home/josh/me/julia/ImageManagement/src/server/images/server.jl:64
macro expansion at /home/josh/me/julia/ImageManagement/src/tasks.jl:61 [inlined]
#3 at ./threadingconstructs.jl:169
unknown function (ip: 0x7f7e48ef0a3c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:839
unknown function (ip: (nil))
error in running finalizer: ErrorException("concurrency violation detected")
error at ./error.jl:33
concurrency_violation at ./condition.jl:8
assert_havelock at ./condition.jl:25 [inlined]
assert_havelock at ./condition.jl:48 [inlined]
assert_havelock at ./condition.jl:72 [inlined]
notify at ./condition.jl:126
#notify#515 at ./condition.jl:124 [inlined]
notify at ./condition.jl:124 [inlined]
notify at ./condition.jl:124 [inlined]
send_del_client at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:265
finalize_ref at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:97
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:278
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:365
run_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:394
jl_gc_collect at /buildworker/worker/package_linux64/build/src/gc.c:3260
maybe_collect at /buildworker/worker/package_linux64/build/src/gc.c:880 [inlined]
jl_gc_pool_alloc at /buildworker/worker/package_linux64/build/src/gc.c:1204
poptask at ./task.jl:755
wait at ./task.jl:763 [inlined]
uv_write at ./stream.jl:992
unsafe_write at ./stream.jl:1064
unsafe_write at /home/josh/.julia/packages/HTTP/cxgat/src/ConnectionPool.jl:172 [inlined]
unsafe_write at /home/josh/.julia/packages/HTTP/cxgat/src/Streams.jl:98
unsafe_write at ./io.jl:646 [inlined]
write at ./io.jl:687 [inlined]
#39 at /home/josh/me/julia/ImageManagement/src/server/images/http_response.jl:93
#open#317 at ./io.jl:330
open##kw at ./io.jl:328 [inlined]
http_send_file at /home/josh/me/julia/ImageManagement/src/server/images/http_response.jl:58 [inlined]
get_image at /home/josh/me/julia/ImageManagement/src/server/images/get_image.jl:113
handle_req at /home/josh/me/julia/ImageManagement/src/server/images/server.jl:47
#35 at /home/josh/me/julia/ImageManagement/src/server/images/server.jl:64
macro expansion at /home/josh/me/julia/ImageManagement/src/tasks.jl:61 [inlined]
#3 at ./threadingconstructs.jl:169
unknown function (ip: 0x7f7e48ef0a3c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:839
unknown function (ip: (nil))

```

I’ve tried to create a MWE but so far have failed, so I’m not 100% on what is causing this issue.

My situation is I have 2 HTTP servers running in a single Julia process. The first serves static resources and is uses threads and `Threads.@async` to process the requests. Processing a request involves querying the database then reading a file from disk and sending the contents back the browser.

The second HTTP server just handles WebSockets and doesn’t use threads. It does use `@async` for each message that comes over the WebSocket to allow multiple operations to be run asynchronously. This server will push long running operations out to a distributed process using remotecall,

So the threaded server doesn’t use Distributed while the non threaded server does. During development I have these both in the same process but normally they will be run in their own processes.

While this appears similar to:

> [@Concurrency violation on interplay between Distributed and Base.Threads](https://discourse.julialang.org/t/concurrency-violation-on-interplay-between-distributed-and-base-threads/47080):
>
> I’m working on a distributed pipeline algorithm that uses several stages per worker process. IIRC tasks cannot hop between threads once they’ve been scheduled. Since I want my stages to potentially run in parallel, I tried to create non-sticky tasks by chaining each D.@spawnat with a T.@spawn. However, this setup keeps failing/crashing and I don’t understand why. I boiled it down to a minimal example: using Distributed, Base.Threads const D = Distributed const T = Threads pids = addprocs(10)…

and the associated:

> <https://github.com/JuliaLang/julia/issues/37706>
>
> I’m working on a distributed pipeline algorithm that uses several stages per wor…ker process. IIRC tasks cannot hop between threads once they’ve been scheduled. Since I want my stages to potentially run in parallel, I tried to create non-sticky tasks by chaining each \`D.@spawnat\` with a \`T.@spawn\`. However, this setup keeps failing/crashing and I don’t understand why.
> 
> I boiled it down to a minimal example:
> 
> \`\`\`julia
> using Distributed, Base.Threads
> 
> const D = Distributed
> const T = Threads
> 
> pids = addprocs(10)
> wids = repeat(pids, inner=2)
> 
> conns = map(RemoteChannel, wids)
> fst = first(conns)
> lst = RemoteChannel()
> push!(conns, lst)
> 
> @everywhere begin
> function stillepost(i, prev, next)
> message = take!(prev)
> put!(next, message)
> @info "Player $i done"
> end
> end
> 
> players = \[\]
> for i in 1:length(wids)
> w = wids\[i\]
> c1 = conns\[i\]
> c2 = conns\[i+1\]
> p = D.@spawnat w fetch(T.@spawn stillepost(i, c1, c2))
> push!(players, p)
> end
> 
> game = @async begin
> m1 = "gibberish"
> put!(fst, m1)
> m2 = take!(lst)
> @info "'$m1' turned into '$m2'; well done!"
> end
> 
> wait.(players)
> wait(game)
> \`\`\`
> 
> Player 2 fails with a concurrency violation:
> 
> \`\`\`
> julia\> include("stillepost.jl")
> \[ Info: Player 1 done
> ERROR: LoadError: On worker 2:
> TaskFailedException:
> concurrency violation detected
> error at ./error.jl:33
> concurrency\_violation at ./condition.jl:8
> assert\_havelock at ./condition.jl:25 \[inlined\]
> assert\_havelock at ./condition.jl:48 \[inlined\]
> assert\_havelock at ./condition.jl:72 \[inlined\]
> wait at ./condition.jl:102
> wait\_for\_conn at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:193
> check\_worker\_state at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:168
> send\_msg\_ at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/messages.jl:176
> send\_msg at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/messages.jl:134 \[inlined\]
> \#remotecall\_fetch#143 at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:389
> remotecall\_fetch at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:386
> \#remotecall\_fetch#146 at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:421
> remotecall\_fetch at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:421
> call\_on\_owner at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:494
> put! at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:595 \[inlined\]
> stillepost at /Users/jonas/.../stillepost.jl:18
> \#3 at ./threadingconstructs.jl:169
> wait at ./task.jl:267 \[inlined\]
> \`\`\`
> 
> Am I holding it wrong?
> 
> \`\`\`
> Julia Version 1.5.1
> Commit 697e782ab8 (2020-08-25 20:08 UTC)
> Platform Info:
> OS: macOS (x86\_64-apple-darwin19.5.0)
> CPU: Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
> WORD\_SIZE: 64
> LIBM: libopenlibm
> LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
> Environment:
> JULIA\_NUM\_THREADS = 4
> JULIA\_PROJECT = @.
> \`\`\`
> 
> See also \[this post\](https://discourse.julialang.org/t/concurrency-violation-on-interplay-between-distributed-and-base-threads/47080) on discourse. foobar\_lv2 suggested this might be a bug in Distributed.

Those seem be to be accessing the distributed servers on the various threads. While in my situation only thread 1 should be accessing the distributed servers. In fact I usually get this crash fairly early when the threaded HTTP server is serving a bunch of static files and the distributed server is actually idle.

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [April 1, 2021, 9:13pm UTC](https://discourse.julialang.org/t/concurrency-violation-during-garbage-collections/58391/2 "2021-04-01T21:13:14Z")

</div>

What happens is that Distributed internally does buffering of sending certain kinds of messages, and that implementation uses finalizers to handle clean-up of various structures, as well as uses regular `Condition` variables to notify listeners about state updates. Unfortunately, as implemented, both of these are both not thread-safe, and can potentially end up deadlocking (or throwing the assertion you’re getting) because they don’t use a construct that properly manages the lock while in a finalizer. Normally this would only show up in threaded code, but I suspect it also can break under heavily-async single-threaded code.

Thankfully there is a [PR open](https://github.com/JuliaLang/julia/pull/38405) that would fix most of these, and it’s been working out decently well for me. Give it a try and let me know if that helps!