Deadlock when joining external threads, jl_enter_threaded_region, dynamic threading

Internals & Design
1

I’m trying to wrap my head around using Julia with some dynamic threading. Ability to handle externally-launched threads was introduced in 1.9 with jl_adopt_thread(). The main documentation seems limited to the release notes and the PR. There’re also third-party guides available, like this one

I am hitting a deadlock when joining external threads, and the workaround recommended in the guide above is to use jl_enter_threaded_region()/jl_exit_threaded_region(). I found this issue where a similar workaround is recommended, but not much explanation apart from “Don’t write buggy blocking loops, and you won’t have this bug”, which I’m not sure applies here.

function thread_create(f)
    wrapped = _ -> (f(); nothing)
    # NOTE: @cfunction also makes sure jl_adopt_thread() is called
    wrapped_c = @cfunction($wrapped, Cvoid, (Ptr{Nothing},))

    # without jl_enter_threaded_region/jl_exit_threaded_region, call to pthread_join/uv_thread_join() might deadlock
    # @ccall jl_enter_threaded_region()::Cvoid

    threadid = UInt[0]
    err = @ccall uv_thread_create(threadid::Ptr{UInt}, wrapped_c::Ptr{Cvoid}, C_NULL::Ptr{Cvoid})::Cint
    @assert err == 0

    threadid[1]
end

function thread_join(threadid)
    tid = UInt[threadid]
    # I thought it may be GC-related deadlock, but gc_safe mode doesn't help
    # gc_state = @ccall jl_gc_safe_enter()::Int8
    err = @ccall uv_thread_join(tid::Ptr{UInt})::Cint
    # @ccall jl_gc_safe_leave(gc_state::Int8)::Cvoid
    @assert err == 0

    # @ccall jl_exit_threaded_region()::Cvoid

    nothing
end

threadid = thread_create() do
    # some IO/sleep() here is required to trigger deadlock
    println("Hello from Julia")
end
thread_join(threadid)

I write it here inline, but of course any number of layers of indirection could trigger the same problem.

So my questions are:

  • is this indeed the right solution in my case? Am I using jl_adopt_thread() correctly? Is this something I can count on in the future?
  • would we all be better off doing it from jl_adopt_thread() or even switching to this mode entirely? What’s the downside?
  • if Julia de-facto already supports dynamically-created threads, why not include it in official Threads.jl API’s?
1 Like
2

Blocking outside of julia, like uv_thread_join does can deadlock on the GC, but in other places too. Julia expects to be in control of all blocking it’s threads can do. The foreign thread support is to allow from threads to call into julia and out of it. But they don’t participate in the scheduler, which is why there isn’t an official API. threaded_region can have some performance cost associated with it but we’ve experimented with it in the past and may do it again in the future.

1 Like
3

Ah, thanks! I don’t think it’s explicitly written in documentation, but the fact there is a @threadcall macro, hints that it’s not allowed. Replacing @ccall uv_thread_join(...) with an equivalent @threadcall expression fixes the deadlock:

function thread_join(threadid)
    tid = UInt[threadid]
    err = @threadcall(:uv_thread_join, Cint, (Ptr{UInt},), tid)
    @assert err == 0

    nothing
end

But they don’t participate in the scheduler, which is why there isn’t an official API.

Well, yes. In fact the whole exercise in my example code was done to launch threads that won’t participate in scheduler and remain be dedicated to the high-priority work. There’re already two threadpools launched by julia, there’s an option to launch a dedicated IO thread, so seems only logical that real-life demands are more diverse, and give users an option to launch/control their own threads.

4

Can you explain a little about the other places?

The GC deadlock is easy enough to expect: Thread A takes a lock, and stalls waiting for GC within the critical section; then Thread B tries to take the lock. If Thread B does this “normally”, it will have a GC safepoint upon blocking; but if the blocking happens in some C code that isn’t careful about that, then we’re deadlocked.

Is this the GC deadlock you alluded to? Are the other cases enumerable and also somewhat easy to understand?

5

The other case that can happen (and I think is what happens here) is you block thread 0 outside of julia so IO won’t run. And then the thing that will release thread 0 is running IO which is a deadlock

1 Like