I have a general question about the possibilities of Packaging a Julia library that is wrapped in python using PyJulia so that it can easily be put into a wheel for distribution.
To give some context, I have develop an open-source scientific library in Julia (this is an optical simulation library) but most of the user only want to use python as a scripting interface for this library so I use PyJulia to have an automatic wrapper around this library. So far everything is well. To simplify the user life I provide a wheel that can be installed using pip.
At the moment my working solution is to bundle in the wheel a full Julia installation with my Julia library and the dependencies already install. The first time the library is call from python after installation I trigger a reinstall of all the dependencies by calling Pkg.instantiate and Pkg.precompile and reconfiguring the Julia/python interface by calling julia.install() in python.
This solution is not the cleanest nor the more convenient but It work and seem to be fairly stable. Now we get to my problem.
A big chunk of the user are in defense and space where there are very strict limitations on network access and downloads. This mean that the step above does not work reliably because their network block Julia when it tries to download dependencies.
Therefore, I want to know if there is not a way to package everything in a more reliable way. I know of PackageCompiler.jl and have used it in some projects but to my knowledge it will not allow me to use PyJulia to make the Julia/Python interface. I would have to implement an interface by hand using ctypes and I donāt see myself finding the time to maintain such an interface on an open-source project.
So I would like to know if there is a way to do this that I am unaware of? For example would switching to PythonCall make it easier? Is there a way to make sure that my Julia dependencies do not need any download when it is relocated to another machine? Or is it not really doable at the moment and I should just tell those users to figure it out themself?
Some extra information:
It needs to work on windows and linux
I am using Julia 1.8 but I can switch to 1.10 if needed.
I target python > 3.7
I need to have both-ways Julia/python interface. Currently I use PyCall and PyJulia for that.
link to the library git (Sign in Ā· GitLab). It is behind a login wall because it is under ESA ESCL license which make it only open-source for people/companies within European countries.
First of I recommend calling PythonCall to call Python, over PyCall, and that project also has JuliaCall to call from Python:
Call Python code from Julia and Julia code from Python via a symmetric interface.
[ā¦]
PyCall has the corresponding Python package PyJulia for calling Julia from Python, and PythonCall similarly has JuliaCall.
One difference is between them is their code size: PyJulia is a large package, whereas JuliaCall is very small, with most of the implementation being in PythonCall itself. The practical up-shot is that PythonCall/JuliaCall have very symmetric interfaces; for example they use identical conversion policies and have the same set of wrapper types available.
Note also that JuliaCall will use a separate Julia project for each virtual/conda environment. This means that different Python environments can maintain an isolated set of Julia dependencies, including the versions of Julia and PythonCall themselves.
Some packages e.g. diffeqpy used PyJulia, but have migrated to JuliaCall. I suggested it at the time. Note, PythonCall does have a slightly slow/er startup than PyCall, but Iāve not checked if that applies for JuliaCall. Probably, but likely can be worked on, and hadnāt stopped people from adopting or even migrating. PyCall is legacy in my view, and probably PyJulia that is based on it.
JuliaCall will download Julia āif requiredā similar to PythonCall downloading Python if calling in the other direction.
PySR still uses PyCall/PyJulia right now for a couple of reasons:
These packages have simply been around for longer so more bugs have been worked out and its easier for my users to google any issues that come up (most of my users have never used Julia so I want to make it as low friction as possible), and
PythonCall/juliacall does not have a conda package (yet!), so you canāt actually distribute libraries depending on juliacall there (which for me is 30% of my downloads), whereas pyjulia has a feedstock
I havenāt used Julia much with Python (at the time PyCall, not in other direction), and I only used Python with pip. It IS possible to use PythonCall as is with pip (or pyjulia, or it with conda), and Iām not sure itās a problem conda missing is a problem, and even if, then you may want to trust it gets fixed soon, could start with pip.
Note though, you can see this issue has been stalled for over a year (and I was fist now learning of this conda issue), but it seems it only needs a trivial name change, and some final push:
Iām eager to try porting PySR to PythonCall.jl from PyJulia. However, the last major hurdle is conda-forge support.
My main objections there was the name since conda-forge is supposed to be language agnostic. There is also an R package called juliacall, thus I think we should call the Python package pyjuliacall and the R package r-juliacall.
There is a third possibility (and maybe fourth), but I didnāt want to go into that, since itās not yet been announced, so maybe immature, and it most likely also has a conda issueā¦
diffeqpy migrated as of October v2.3 from juliacall:
It changed installation in December (now only automatic, and manual install dropped):
As mentioned in this video, diffeqpy has a few remaining problems related to the Python integration still that need to be fixed up. Youāre seeing this + JIT time.
As I said diffeqpy migrated to PythonCall, but I do see it in [EDIT: a version from before the migration] [ana]conda, so how is that possible? Maybe itās an old version from before the migration? Which is more used (with pip)?
Chris, subjectively, how good is the user experience now, better than with PyCall with or without Conda?
[There are many Julia packages callable from other languages, usually with pyjulia (or usinga package for R), and I recall[ed] a few with PythonCall, actually I thought SR already had, so maybe just 2 or 1ā¦? I at least forget other names. I think diffeqpy was first migrating, but maybe not first one adopting it, skipping/never adopting pyjulia. Does anyone have a good list, or a way to know?]
We redid the whole package onto PythonCall and it fixed pretty much all of the issues. Iām happy with it now. User experience is now a lot better. Big props to @Lilith too for setting up essentially all of the details there.
It automatically installs Julia and all of the required packages and sets up some helper functions for the Python user to use the translated calls in a bit nicer way. Anyone who wants to do the same can just look at that package because itās extremely small (maybe no more than like 200 lines of code total) and is effectively just good documentation on how to make a wrapper.
Thanks both, that is extremely helpful to hear your positive experience using it in the wild. That makes me much more eager to switch. It really is just that conda compatibility that blocks it now, since conda makes up a third of PySRās user base.
