We’d very much like to have a gradual conversion of our RexDB system from Python 2.7 over to Julia 1.0 – critical to this gradual transition is having new code in Julia rather than Python. This means I need to be able to rewrite high-level functionality chunks (not small things) in Julia and then invoke them from an existing Python based server currently using uwsgi. Overall, it seems that pyJulia itself is small amount of code. Most of the heavy lifting (object conversion) is done in PyCall, which seems to be well maintained.
I’m willing to invest some time with this code, issues and documentation, but I’ll need some help. First, I don’t quite understand what fake-julia is for; it seems quite critical, but it’s just unclear to me. I could make a ticket to ask for someone to document it… but, if you’re the owner, if you’d just spend 30 minutes explaining why it’s there and what it does, it’d be immensely useful.
Second, I’m wondering how to make pyjulia thread-safe. The use of PyDLL basically converts multi-thread into Julia bound single thread backend. I figure that there’s probably a way to make “convert” hold the GIL (global interpreter lock; blocking all threads) but, I’m not sure it’s necessary to hold the GIL on all Julia calls. Does jl_exception_occurred use thread local storage? Does convert in PyCall lock the GIL? What other issues are there with thread-safe behavior? Should I hang this up?
Third, the introspection step on startup seems a bottleneck. One option is to introspect the julia functions as an invocation is attempted. You won’t have command line completion, but the entire introspection step on startup could be done dynamically as needed. Perhaps this is a flag? I can update that ticket.
Fourth, from an administrative perspective, there seem to be quite a bit of old tickets. I’d need some authority to commit and clean them up. I don’t want to step on anyone’s toes though.
Hopefully the situation is clearer now. This is a pretty subtle trick that @Keno came up with to make precompilation of Julia modules that depend on PyCall work properly from within pyjulia.
Second, I’m wondering how to make pyjulia thread-safe. The use of PyDLL basically converts multi-thread into Julia bound single thread backend. I figure that there’s probably a way to make “convert” hold the GIL (global interpreter lock; blocking all threads) but, I’m not sure it’s necessary to hold the GIL on all Julia calls. Does jl_exception_occurred use thread local storage? Does convert in PyCall lock the GIL? What other issues are there with thread-safe behavior? Should I hang this up?
I don’t think libjulia is thread-safe, so I don’t see how Python could safely call Julia functions from multiple threads simultaneously. (Of course, once it calls a Julia function, that Julia function itself may spawn multiple threads that Python doesn’t know about.) PyCall itself currently doesn’t touch the GIL.
Third, the introspection step on startup seems a bottleneck. One option is to introspect the julia functions as an invocation is attempted. You won’t have command line completion, but the entire introspection step on startup could be done dynamically as needed.
I never touched this code (added in this commit), but what you suggest sounds very reasonable to me. Rather than looping over all module symbols when the module is loaded, that information should really be loaded lazily (e.g. when a symbol is accessed or when tab-completion is requested).
Fourth, from an administrative perspective, there seem to be quite a bit of old tickets. I’d need some authority to commit and clean them up. I don’t want to step on anyone’s toes though.
Projects are usually happy to grant someone commit access after they make a few successful pull requests.