Current best practices for Julia dependencies in Python packages

General Usage
, , ,
1

I have a python package that a lot of colleagues use, and I have a julia package that does some of the same things vastly better, so I’d like to use it from the python package, as an optional install. I’m not too worried about the mechanics of how this will work out once everything is installed (I’m quite happy with PythonCall), but I am very worried about the installation process and environment handling. In particular, I’m trying to make the case to my colleagues that working with julia is awesome, to encourage them to just move over, so I don’t want their first encounters to be painful.

Obviously, the most worrying part is checking that julia exists locally and is sufficiently up to date, then installing/updating julia automatically if needed. There are a few resources I’ve looked at, but it’s not clear to me what the most maintainable option would be.

@MilesCranmer has the wonderful pysr, which does lots of cool stuff. In particular, he uses shared julia environments named f"pysr-{pysr.__version__}" to install the julia stuff, which is clever. On the other hand, he uses PyCall, which is DOA for me because of how it interacts with different python environments. (Even Miles seems to agree that any pulse is faint, as he has applied the AED.)

The diffeqpy package does similar things. It uses a shared julia environment named "diffeqpy", and uses PythonCall, but depends on jill.py. I suppose this was started at a time when jill.py was the best option, but it’s not clear to me that that’s still the case.

@jlapeyre has a nice-looking package, that basically abstracts this stuff into a separate package. I think that’s a great idea, but this one looks like it hasn’t been updated in a while. Looks like there was some cross-fertilization with diffeqpy. In particular, this also uses jill.py. Maybe the best use of my time would just be to try to pitch in on this package with any lessons I learn here.

I am happy to require some form of conda. I know there’s a julia feedstock, but it looks like a nightmare to maintain, and indeed seems to be having trouble updating to julia 1.10 (which is quite understandable given those difficulties). There’s also a juliaup feedstock, but it’s not clear to me if that’s better than the julia feedstock, or if there are advantages to one or the other.

I guess having some specific questions would be helpful:

  • Is jill.py still the best option in this context, or should it be replaced by juliaup?
  • Is either of the conda-forge options likely to be reliable going forward? Is one better than the other? Are there drawbacks with, e.g., the size of the executables if they’re installed in each python env?
  • What are the arguments for using a single shared environment vs. appending the python package version to its name? Maybe just the major or major.minor elements?
  • Since I’ll probably have to do a good deal of writing code either way, would it be more productive to try to help update @jlapeyre’s package? Or should I just roll my own for my package, and leave it at that?
2

IMO JuliaUp is without a doubt the easiest way to get started with Julia, and should be considered the default way for anyone to install Julia. It has superseded the likes of Jill.

If you are using JuliaCall/PythonCall, it will handle installing Julia for you if not already installed. If you have JuliaUp already installed (highly recommended) then it will use that to select and install an appropriate version of Julia automatically.

I’m not familiar enough with the Conda options to comment.

JuliaCall will automatically create a separate Julia environment for each Python environment (venv or Conda) that you use, keeping dependencies separate.

5 Likes
3

My latest experiment is turning juliacall into a conda package:
https://anaconda.org/julialang/pyjuliacall

That’s installable via

$ conda install julialang::pyjuliacall

I mainly have not gotten around to figuring out what went wrong with the julia-feedstock. I’m going to guess that maybe some of the patches applied are now obsolete.

2 Likes
4

@MilesCranmer told me he migrated to juliacall, and looking to confirm, I see, well there’s PR “working!”, though not yet merged:

You can try that version, or wait a bit.

I did though see: [BUG]: libstdc++ issues / `GLIBCXX` not found · Issue #347 · MilesCranmer/PySR · GitHub

For posterity I also found that juliacall runs into the exact same problem :frowning: So we aren’t automatically saved by switching Python<->Julia interfaces

and it may be solved too: Fixing incorrect "Unable to load"/"GLIBCXX not found" issue, once and for all (hopefully) · Issue #437 · JuliaPy/PythonCall.jl · GitHub

This has largely been fixed in PythonCall (i.e. calling Python from Julia) by having CondaPkg install a version of libstdc++ compatible with whatever Julia is using.

2 Likes
5

I absolutely agree… when it comes to julia users. My concern is that automatically installing it might tangle with existing installations, lead to bloat, or otherwise confuse/annoy people who aren’t already julia users.

Oh, this is excellent. I hadn’t seen that part, or JuliaPkg. This looks like it might be exactly what I need. Thanks again for all you do, @cjdoris.

6

I don’t think there’s any issue with jill.py? diffeqpy had a complete overhaul just 3 months ago to move from PyCall to PythonCall:

The jill.py part wasn’t touched because, why fix what isn’t broken? It works fine and I don’t have any issues with it. PyCall vs PythonCall though, we have had much more success with PythonCall and would definitely recommend that. It closed most of our major issues.

2 Likes
7

I’d strongly recommend removing the Jill stuff from that package and just let JuliaCall do its thing to install packages for you. You just need to create a juliapkg.json file, then you can remove all the code in __init__.py.

The reason why I recommend this is because the existing approach (installing into a package-specific environment) is not composable with other packages that also want to use JuliaCall - if they also install into their own environment, then which environment should JuliaCall use? Both packages can’t have it their own way. On the other hand, JuliaPkg merges dependencies from all the Python packages installed for you into a single environment.

I can put in a PR if you like.

2 Likes
8

I’m quite confused by this response - you first say you’re worried about JuliaUp automatically installing Julia, then you’re happy with JuliaCall automatically installing Julia.

If your goal is to hide Julia as much as possible from the user, it’s probably simplest to install Julia from conda-forge. There’s currently an effort to add JuliaCall to conda-forge too and they should work together nicely.

9

A PR would be helpful.

10

I should have been clearer that my concern is that there are ways of doing this that lead to bloat — not that it necessarily leads to bloat. Specifically, some of the automatic installation methods I found lying around the internet appear to install a new julia and depot within the python/conda env. That’s where my concerns lie. But it looks like JuliaPkg

  1. uses the currently recommended installation tool (juliaup);
  2. results in a standard installation in the user’s home directory; and
  3. maintains this as a separate package, which will hopefully make it a centralized focal point for community-driven improvement, rather than as just some hack that I’d have to try to keep up-to-date on my own.

Those are the three big points I was looking for, which is why JuliaPkg seems to fit the bill for me.

2 Likes
11

Indeed, Julia in conda-forge is configured to put a new depot in your Conda environment.

JuliaCall uses JuliaUp (which uses the default depot) IF it is already installed - you need to install it manually. Otherwise JuliaCall will download and install Julia directly into your Python environment, but still using the default depot.

2 Likes
12

Oh, man. This was insanely easy. I put off doing this for so long because I thought it would be difficult, but it’s literally 3 lines of python, 1 line in pyproject.toml, and 9 lines of juliapkg.json. This is astoundingly good. Thank you SO much, @cjdoris!

4 Likes
13

And now there’s also a conda-forge feedstock for juliacall: GitHub - conda-forge/pyjuliacall-feedstock: A conda-smithy repository for pyjuliacall.. Which I can confirm works nicely in an environment setting (PySR has switched entirely to JuliaCall)

1 Like