I have a Julia package which I would like to expose to python users.
Seems the most obvious way would be using juliacall, similar to how diffeqpy seems to do it. I tried working with that approach for now, but I’ve got a few questions.
Would it be best to have the python code at a separate repository (like diffeqpy), or at the same one as the Julia code? Any (dis)advantages in either way?
The package I’m working on has a GLMakieextension, which provides some very useful GUI capabilities. Seems like using GLMakie from juliacall is not seamless, and some workarounds might be required. Has anybody managed to deal with this matter in an elegant way (e.g. wrap Makie functions with juliacall.interactive() somewhere in the python codebase)?
More generally, would there be any pitfalls to avoid when working with Julia package extensions in this context?
I was wondering whether the relatively recent updates in AOT compilation would enable better solutions to this, exposing the same functionality while avoiding a Julia installation for the python user. Has anybody experimented with this?
Any other methods or resources I’m missing here?
I was also thinking, since most people are probably not going to be migrating from python to Julia anytime soon, might be useful to have such information for Julia package developers available somewhere in the documentation?
I don’t know how complex your package is or how long a single function call takes. But as long as your functions are large and take at least 5ms to execute, you could use a JSON over HTTP approach. I found this less error-prone than JuliaCall. In my case, the Python code had to use an outdated Python version that was not supported by JuliaCall.
Sounds interesting, but also more complicated than the juliacall solution.
Ideally I’d be looking for something as seamless as possible, where the python user only does a pip install (or equivalent) and forgets about the existence of julia, working only within their python comfort zone.
Either is fine, it’s up to personal preference and how you want to manage things.
Not that I’m aware of. However I suppose one thing we could do is have JuliaCall fire up a separate thread which calls Julia’s yield in a loop. This would just run continuously and allow the Julia event loop to keep looping even when back in Python.
I’m not aware of any particular pitfalls of using Julia package extensions from JuliaCall.
I’ve experimented with this a bit and you can indeed compile a Python extension module from some Julia code using juliac --trim. I meant to write a post about it but never did. The big caveat is that it still requires the Julia runtime library (i.e. essentially a full Julia install anyway) but we could wrap up into a Python package rather than install it like JuliaCall does. Then we’d need to figure out how to get a juliac-compiled module to link against that runtime when loaded. Must be doable I just haven’t tried it. The JuliaCall route is fine, but the compiled route should have really small TTFX.
I think your two options are:
The in-process option: use JuliaCall.
The separate-process option: wrap the functionality up as a REST API server that Python can call, as suggested by @ufechner7. I’m sure you could automate the launching of the server and connecting to it so that it is transparent to the user. And you can still use pyjuliapkg to handle the Julia dependencies.
Unless you’ve got a strong technical reason not to (e.g. you need really old dependencies, or multi threading) then JuliaCall will be the simplest route.
For the goal to preserve the Python (and Julia) “comfort zone” when building a
multi-language system, another approach may be Rembus.
I recently released Rembus for Python, which allows Python and Julia components to communicate seamlessly in the same distributed system. If you want a gentle introduction, I started a mini blog with worked examples: Rembus Blog – Rembus
A few things to give some more context:
Exchanged data are CBOR encoded (binary data, not JSON or base64 encoding)
DataFrame support out of the box: polars/pandas interops natively with Julia DataFrames
Rembus supports both brokered and brokerless (client/server) topologies
Optional DuckLake integration for persistent data at rest (ignore if not needed)
Without more details about your requirements it’s hard to say if it’s the right fit, but happy to answer any questions.
At the moment, Rembus only supports Python and Julia.
If it gains traction, I’d definitely consider adding support for more languages.
Also, if someone is interested in implementing a client in another language, I’d be happy to document and publish the CBOR-based protocol so it can be extended more easily.
Yes, people have been anecdotally reporting problems, and I’ve also personally experienced bugs, but I hope any bugs and inherent limitations can be documented precisely, so that you only reach for alternatives in these corner cases. “Just giving up” on PythonCall and JuliaCall is not very appealing since they’re so much more convenient and integrated than alternatives.
Well, it is documented that you can call Julia functions only from the main thread of Python, and the application I had to connect needed to call Julia from another thread.