According to this section of the docs as well as numerous discussions on discourse and the email list, it’s apparently not allowed to call julia functions from external C threads other than the one that spawned Julia.
First of all, I’d love to know why this is a problem. Does it have something to do with private memory on the spawning thread? Is it only about concurrent access to julia functions? Any pointers to documentation/source code that explains the underlying issue would be very helpful. Especially given that so much has changed with multithreading lately, I’m not sure what’s still accurate from discussions that I’ve read.
Secondly, even if it’s not possible to directly call Julia functions from external C threads, is there any way to send data back and forth? e.g. mutating data in shared memory that both Julia and C can access.
Ideally, it would be awesome to be able to send/receive on Julia channels. I know this is generally done by calling Julia functions, which leads us back to the first problem, but perhaps there’s a way to directly interact with Julia channels as if they were native C structs? I recently found this C implementation of Go channels which inspired this thought. Any chance something like this is possible?
You can share memory between Julia and C (i.e. you can allocate an array in Julia, then pass a pointer to the data to some C code where it’s accessible as a regular C array). In a multithreading context you’ll need to do all the regular concurrency stuff, mostly wrapping any non-atomic operations on the shared memory with a Mutex. I’m not sure what the best way to share the mutex so it’s usable from both contexts though…maybe if you look into the source for ReentrantLock and SpinLock there’s an underlying lock that you can from the C context in the other thread.
If you want to use a non-blocking datastructure (for instance if the other thread needs to be real-time safe so it can’t block) then you can use RingBuffers.jl, which has both a C and Julia API and is designed for just this kind of thing. Fair warning though - that package is pretty much unmaintained at this point
I’ve raised an issue here asking whether the C channel package that I linked above can be used to communicate between C and Go to see whether it may be possible to implement something like this in Julia
Update: That package cannot be used for C-Go interop, but this one was created for that purpose!
Are there any plans to support passing Julia callbacks to C libraries that can later be called by the C library from a separate thread? At the moment, this is not possible since Julia is not thread-safe, but it would be very useful for signal processing algorithms. Most of the Audio I/O low-level C libraries create a different thread for non-blocking read/write API. For now, it’s only possible to use these libraries with blocking API without callback.
Jeff:
Yeah, I think it will be possible. I believe all we have to do is expose a C entrypoint to basically initialize whatever thread it’s called from, like jl_init_current_thread or something to setup the thread-local state that the runtime needs so that Julia code can run on that thread. I think that might be pretty much all that’s needed. So I do think it will be possible.
Stefan:
But if the code is written in Julia, it won’t be non-blocking, I mean it will have to interact with other Julia threads, right?
Jeff:
Yes, so well you have the normal problems of callbacks. I mean, it mentions writing callbacks in Julia which is technically possible, but, you know, callbacks are very restrictive. There are all sorts of things you don’t want to do from a callback. You don’t want it to take very long, you don’t want it to block, and you probably don’t want excessive pauses either, although we don’t have a realtime GC anyway, so maybe that’s not the concern. But yeah, you’d have to be pretty careful what you did in your Julia code because you really are in a restricted environment in a callback, so you’d have to be pretty careful anyway:
Stefan:
So in some sense, what people actually want, it seems to me, might be the ability to write Julia code in such a way that they’re guaranteed that it’s not going to do any of those dangerous things: it’s not going to allocate…, which, you know, the tricky part about that is that’s an ever-changing…, you know, who knows what the compiler’s going to do. It might suddenly decide “I don’t feel like optimizing your thing anymore”.
Jeff:
Yeah, I think people do want that, but it’s a secondary concern, though. I mean, right now the first thing is just to make it possible of course. And then if it’s too slow or something, that’s a different issue.
@jeff.bezanson - to follow up on that, it sounds like just calling jl_init (source code here) from the other thread would not work, correct? In that case, can we just pull out pieces of jl_init? What exactly would jl_init_current_thread need to do?
As far as I remember, currently each thread initializes its thread-local state but that state is intrinsically related to the thread being managed by Julia “dispatcher”.
So currently Julia as a static list of threads that it manages and which form a fixed pool of threads that can receive work. Having external threads would not only be about initializing their local state but also about letting Julia know about them and treat them differently: for example, the scheduler would not take them into account when managing work however the GC would have to take them into account.
Just to clear up some confusion. Right now you can obtain a C-callback pointer to a Julia function with @cfunction. You can use that callback in two ways:
On a Julia thread that enters a C-library. Most callbacks are fine in that situation.
On a non-Julia thread. Callbacks are severely limited in what they can do.
The prime restriction in the second case is: You are not allowed to interact with the Julia runtime, among those interactions are:
Allocating objects
Entering the runtime
IO
yields/task/etc…
Jeff’s comment was that one could setup the necessary thread-local state to allow some of these interactions, like allocating memory. as @cdsousa points out right now we assume that there is a fixed number of threads and the GC needs to have some visibility into the threads that use or allocate Julia objects.
The challenge is that even when you allow for some interactions, there are others that are detrimental to callbacks. You wouldn’t want the Julia scheduler to steal your thread, just because you entered a yield point.
Right now the only safe method to communicate with the Julia runtime from a non-Julia thread is AsyncCondition and the associated uv_async_send.