# Current best practices for Julia dependencies in Python packages

**URL:** https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433
**Category:** General Usage
**Tags:** python, juliacall, pythoncall, pyjulia
**Created:** [January 30, 2024, 5:43am UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433 "2024-01-30T05:43:11Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![moble](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moble/32/23535_2.png) [@moble](https://discourse.julialang.org/u/moble)
#### Post date: [January 30, 2024, 5:43am UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/1 "2024-01-30T05:43:11Z")

</div>

I have a python package that a lot of colleagues use, and I have a julia package that does some of the same things vastly better, so I’d like to use it from the python package, as an _optional_ install. I’m not too worried about the mechanics of how this will work out once everything is installed (I’m quite happy with `PythonCall`), but I am very worried about the installation process and environment handling. In particular, I’m trying to make the case to my colleagues that working with julia is awesome, to encourage them to just move over, so I don’t want their first encounters to be painful.

Obviously, the most worrying part is checking that julia exists locally and is sufficiently up to date, then installing/updating julia automatically if needed. There are a few resources I’ve looked at, but it’s not clear to me what the most maintainable option would be.

@MilesCranmer has the wonderful [`pysr`](https://github.com/MilesCranmer/PySR), which does [lots of cool stuff](https://github.com/MilesCranmer/PySR/blob/master/pysr/julia_helpers.py). In particular, he uses shared julia environments named `f"pysr-{pysr. __version__ }"` to install the julia stuff, which is clever. On the other hand, he uses PyCall, which is DOA for me because of how it interacts with different python environments. (Even [Miles seems to agree that any pulse is faint](https://discourse.julialang.org/t/is-pyjulia-org-e-g-juliacall-or-pyjulia-python-packages-maintained/109277/7), as he has applied the AED.)

The `diffeqpy` package does [similar things](https://github.com/SciML/diffeqpy/blob/master/diffeqpy/ __init__.py). It uses a shared julia environment named `"diffeqpy"`, and uses `PythonCall`, but depends on `jill.py`. I suppose this was started at a time when `jill.py` was the best option, but it’s not clear to me that that’s still the case.

@jlapeyre has [a nice-looking package](https://github.com/jlapeyre/julia_project), that basically abstracts this stuff into a separate package. I think that’s a great idea, but this one looks like it hasn’t been updated in a while. Looks like there was some cross-fertilization with `diffeqpy`. In particular, this also uses `jill.py`. Maybe the best use of my time would just be to try to pitch in on this package with any lessons I learn here.

I am happy to require some form of `conda`. I know there’s a [julia feedstock](https://github.com/conda-forge/julia-feedstock), but it looks like a nightmare to maintain, and indeed seems to be having trouble updating to julia 1.10 (which is quite understandable given those difficulties). There’s also a [juliaup feedstock](https://github.com/conda-forge/juliaup-feedstock/), but it’s not clear to me if that’s better than the julia feedstock, or if there are advantages to one or the other.

* * *

I guess having some specific questions would be helpful:

- Is `jill.py` still the best option in this context, or should it be replaced by `juliaup`?
- Is either of the conda-forge options likely to be reliable going forward? Is one better than the other? Are there drawbacks with, e.g., the size of the executables if they’re installed in each python env?
- What are the arguments for using a single shared environment vs. appending the python package version to its name? Maybe just the major or major.minor elements?
- Since I’ll probably have to do a good deal of writing code either way, would it be more productive to try to help update @jlapeyre’s package? Or should I just roll my own for my package, and leave it at that?

---

<div class="post-metadata">

### Author: ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)
#### Post date: [January 31, 2024, 8:42am UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/2 "2024-01-31T08:42:38Z")

</div>

IMO JuliaUp is without a doubt the easiest way to get started with Julia, and should be considered the default way for anyone to install Julia. It has superseded the likes of Jill.

If you are using JuliaCall/PythonCall, it will handle installing Julia for you if not already installed. If you have JuliaUp already installed (highly recommended) then it will use that to select and install an appropriate version of Julia automatically.

I’m not familiar enough with the Conda options to comment.

JuliaCall will automatically create a separate Julia environment for each Python environment (venv or Conda) that you use, keeping dependencies separate.

---

<div class="post-metadata">

### Author: ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [January 31, 2024, 8:58am UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/3 "2024-01-31T08:58:46Z")

</div>

My latest experiment is turning juliacall into a conda package:  
[https://anaconda.org/julialang/pyjuliacall](https://anaconda.org/julialang/pyjuliacall)

That’s installable via

```julia
$ conda install julialang::pyjuliacall

```

I mainly have not gotten around to figuring out what went wrong with the julia-feedstock. I’m going to guess that maybe some of the patches applied are now obsolete.

---

<div class="post-metadata">

### Author: ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [January 31, 2024, 8:14pm UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/4 "2024-01-31T20:14:14Z")

</div>

> [@moble](#):
>
> @MilesCranmer has the wonderful [`pysr`](https://github.com/MilesCranmer/PySR), which does [lots of cool stuff] […] On the other hand, he uses PyCall, which is DOA for me because of how it interacts with different python environments.

@MilesCranmer told me he migrated to juliacall, and looking to confirm, I see, well there’s PR “working!”, though not yet merged:

> <https://github.com/MilesCranmer/PySR/pull/535>
>
> I managed to get a PythonCall.jl/JuliaCall version of PySR working!
> 
> Tagging @…mkitti @cjdoris in case you are curious. Any suggestions about implementation are very appreciated :)
> 
> Detailed changes:
> 
> \- Dependencies are now managed by pyjuliapkg rather than the custom code we made. Simplifies things a lot!
> \- The user no longer needs to run \`python -m pysr install\`. The install process is done by JuliaCall at import time.
> - Removed code related to \`pysr.install()\` and \`python -m pysr install\` because JuliaCall now handles this. \`python -m pysr install\` will not give a warning and do nothing.
> \- Changed PyJulia with JuliaCall
> - Swapping \`eval\` -\> \`seval\`
> - Manually converting to \`Vector\` when calling SymbolicRegression.jl functions (otherwise would get passed as \`PyList{Any}\`; see https://github.com/JuliaPy/PythonCall.jl/issues/441)
> - Wrapped \`equation\_search\` code with \`jl.PythonCall.GC.disable()\` to avoid multithreading-related segfaults (https://github.com/JuliaPy/PythonCall.jl/issues/298)
> - Manually convert \`np.str\_\` to \`str\` before passing to \`variable\_names\`, otherwise it becomes a \`PyArray\` and not a \`String\` (@cjdoris might be worth adding a workaround, it seems like PyJulia does this automatically)
> \- Rather than storing the raw julia variables in \`PySRRegressor\`, I am now storing a serialized version of them. This means you can now pickle the search state and warm-start the search in another Python process. It feels much safer to do it like this.
> 
> The main blocker for this is conda-forge support in https://github.com/conda-forge/staged-recipes/pull/25097. Once that is done we can merge.
> 
> TODO:
> 
> \- \[\] Decide if I want to import JuliaCall lazily (at first run) or if it gets installed from simply \`import pysr\`.
> \- \[\] Add back \`pysr.install\` with a warning message.
> \- \[\] Add other unnecessary functions with simple deprecation messages.
> \- \[\] Remove click dependency.
> \- \[\] Test custom \`julia\_project\` with development version of SymbolicRegression.jl.
> \- \[\] Use the git version of SymbolicRegression.jl rather than registry version.
> \- \[\] Test large distributed run on a slurm cluster.
> \- \[\] Find other unused pieces of code and remove them.
> \- \[\] Ensure coverage is pretty good.
> \- \[\] Set environment variable for \`-O3\`.
> \- \[\] Raise warning if Julia was started without multiple threads, without optimization.
> \- \[\] Add an integration test for restarting a warm start fit in a new Python session.

You can try that version, or wait a bit.

I did though see: [[BUG]: libstdc++ issues / `GLIBCXX` not found · Issue #347 · MilesCranmer/PySR · GitHub](https://github.com/MilesCranmer/PySR/issues/347#issuecomment-1872622455)

> For posterity I also found that `juliacall` runs into the exact same problem ☹ So we aren’t automatically saved by switching Python\<-\>Julia interfaces

and it may be solved too: [Fixing incorrect "Unable to load"/"GLIBCXX not found" issue, once and for all (hopefully) · Issue #437 · JuliaPy/PythonCall.jl · GitHub](https://github.com/JuliaPy/PythonCall.jl/issues/437#issuecomment-1899031939)

> This has largely been fixed in PythonCall (i.e. calling Python from Julia) by having CondaPkg install a version of libstdc++ compatible with whatever Julia is using.

---

<div class="post-metadata">

### Author: ![moble](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moble/32/23535_2.png) [@moble](https://discourse.julialang.org/u/moble)
#### Post date: [January 31, 2024, 9:04pm UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/5 "2024-01-31T21:04:21Z")

</div>

> [@cjdoris](#):
>
> IMO JuliaUp is without a doubt the easiest way to get started with Julia, and should be considered the default way for anyone to install Julia

I absolutely agree… when it comes to julia users. My concern is that automatically installing it might tangle with existing installations, lead to bloat, or otherwise confuse/annoy people who aren’t already julia users.

> [@cjdoris](#):
>
> If you are using JuliaCall/PythonCall, it will handle installing Julia for you if not already installed.

Oh, this is excellent. I hadn’t seen that part, or `JuliaPkg`. This looks like it might be exactly what I need. Thanks again for all you do, @cjdoris.

---

<div class="post-metadata">

### Author: ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [January 31, 2024, 10:40pm UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/6 "2024-01-31T22:40:42Z")

</div>

> [@moble](#):
>
> - Is `jill.py` still the best option in this context, or should it be replaced by `juliaup`?

> [@moble](#):
>
> The `diffeqpy` package does [similar things](https://github.com/SciML/diffeqpy/blob/master/diffeqpy/ __init__.py). It uses a shared julia environment named `"diffeqpy"`, and uses `PythonCall`, but depends on `jill.py`. I suppose this was started at a time when `jill.py` was the best option, but it’s not clear to me that that’s still the case.

I don’t think there’s any issue with `jill.py`? diffeqpy had a complete overhaul just 3 months ago to move from PyCall to PythonCall:

> <https://github.com/SciML/diffeqpy/pull/118>
>
> This PR
> \- \[x\] Changes the backend from PyCall to PythonCall
> \- \[x\] Removes most… of the glue code that makes Py\[thon\]Call and DifferentialEquaitons compatible (that code has been upstreamed into package extensions https://github.com/SciML/SciMLBase.jl/pull/502 & https://github.com/SciML/SciMLBase.jl/pull/519)
> \- \[x\] Makes the install script perform a single Pkg operation instead of several to reduce noise and the risk of re-precompilation
> \- \[x\] Stops using the user's default environment (uses \`@diffeqpy\` instead)
> \- \[x\] Automatically installs Julia & packages if Julia is missing when loading \`de\` or \`ode\` (previously this only happened when loading \`de\`)
> \- \[\] Updates all examples in the readme so that they still work.
> 
> This PR is breaking because
> 
> \- \`from julia import Main\` no longer works. This can be replaced with \`from juliacall import Main\`, but \`julia.Main\` (from PyCall.jl) and \`juliacall.Main\` (from PythonCall.jl) have slightly different semantics around conversion and \`eval\`/\`seval\`. For the examples in the README, all that is needed is to replace
> \`\`\`py
> from julia import Main
> Main.eval("1+1")
> \`\`\`
> with
> \`\`\`py
> de.seval("1+1")
> \`\`\`
> 
> \- Automatic conversion semantics are different. For calls into the SciML ecosystem, this is mostly patched, but if folks are heavily utilizing passing data back and forth between Python and Julia in their own code and do not have a direct dependency on \`julia\`/\`PyCall.jl\`, this will break their code.
> \- Differential equation results are not automatically converted when passed back to Python. Specifically, \`sol.u\` is often \`Vector{Vector{T}}\` instead of a numpy array. This necessitates a bit of reshaping before passing to \`matplotlib.pyplot\`. \`de.stack\` is available for converting that \`Vector{Vector{T}}\` into a \`Matrix{T}\` which can be plotted, though even then it is transposed compared to what diffeqpy previously returned. 
> \- It breaks some numba interop. (TODO)
> 
> See changes to the README to see the impact of the breaking changes on downstream code.

The `jill.py` part wasn’t touched because, why fix what isn’t broken? It works fine and I don’t have any issues with it. PyCall vs PythonCall though, we have had much more success with PythonCall and would definitely recommend that. It closed most of our major issues.

---

<div class="post-metadata">

### Author: ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)
#### Post date: [February 1, 2024, 7:41am UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/7 "2024-02-01T07:41:52Z")

</div>

> [@ChrisRackauckas](#):
>
> The `jill.py` part wasn’t touched because, why fix what isn’t broken? It works fine and I don’t have any issues with it.

I’d strongly recommend removing the Jill stuff from that package and just let JuliaCall do its thing to install packages for you. You just need to create a `juliapkg.json` file, then you can remove all the code in ` __init__.py`.

The reason why I recommend this is because the existing approach (installing into a package-specific environment) is not composable with other packages that also want to use JuliaCall - if they also install into their own environment, then which environment should JuliaCall use? Both packages can’t have it their own way. On the other hand, JuliaPkg merges dependencies from all the Python packages installed for you into a single environment.

I can put in a PR if you like.

---

<div class="post-metadata">

### Author: ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)
#### Post date: [February 1, 2024, 8:30am UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/8 "2024-02-01T08:30:50Z")

</div>

> [@moble](#):
>
> > [@cjdoris](#):
> >
> > IMO JuliaUp is without a doubt the easiest way to get started with Julia, and should be considered the default way for anyone to install Julia
> 
> I absolutely agree… when it comes to julia users. My concern is that automatically installing it might tangle with existing installations, lead to bloat, or otherwise confuse/annoy people who aren’t already julia users.
> 
> > [@cjdoris](#):
> >
> > If you are using JuliaCall/PythonCall, it will handle installing Julia for you if not already installed.
> 
> Oh, this is excellent. I hadn’t seen that part, or `JuliaPkg`. This looks like it might be exactly what I need. Thanks again for all you do, @cjdoris.

I’m quite confused by this response - you first say you’re worried about JuliaUp automatically installing Julia, then you’re happy with JuliaCall automatically installing Julia.

If your goal is to hide Julia as much as possible from the user, it’s probably simplest to install Julia from conda-forge. There’s currently an effort to add JuliaCall to conda-forge too and they should work together nicely.

---

<div class="post-metadata">

### Author: ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [February 1, 2024, 11:12am UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/9 "2024-02-01T11:12:05Z")

</div>

A PR would be helpful.

---

<div class="post-metadata">

### Author: ![moble](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moble/32/23535_2.png) [@moble](https://discourse.julialang.org/u/moble)
#### Post date: [February 1, 2024, 3:50pm UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/10 "2024-02-01T15:50:52Z")

</div>

> [@cjdoris](#):
>
> > [@moble](#):
> >
> > I absolutely agree… when it comes to julia users. My concern is that automatically installing it might tangle with existing installations, lead to bloat, or otherwise confuse/annoy people who aren’t already julia users.
> 
> […]
> 
> I’m quite confused by this response - you first say you’re worried about JuliaUp automatically installing Julia, then you’re happy with JuliaCall automatically installing Julia.

I should have been clearer that my concern is that there are ways of doing this that lead to bloat — not that it necessarily leads to bloat. Specifically, some of the automatic installation methods I found lying around the internet appear to install a new julia and depot _within the python/conda env_. That’s where my concerns lie. But it looks like JuliaPkg

1. uses the currently recommended installation tool (juliaup);
2. results in a standard installation in the user’s home directory; and
3. maintains this as a separate package, which will hopefully make it a centralized focal point for community-driven improvement, rather than as just some hack that I’d have to try to keep up-to-date on my own.

Those are the three big points I was looking for, which is why JuliaPkg seems to fit the bill for me.

---

<div class="post-metadata">

### Author: ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)
#### Post date: [February 1, 2024, 5:07pm UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/11 "2024-02-01T17:07:35Z")

</div>

Indeed, Julia in conda-forge is configured to put a new depot in your Conda environment.

JuliaCall uses JuliaUp (which uses the default depot) **IF** it is already installed - you need to install it manually. Otherwise JuliaCall will download and install Julia directly into your Python environment, but still using the default depot.

---

<div class="post-metadata">

### Author: ![moble](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moble/32/23535_2.png) [@moble](https://discourse.julialang.org/u/moble)
#### Post date: [May 23, 2024, 9:34pm UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/12 "2024-05-23T21:34:25Z")

</div>

Oh, man. This was insanely easy. I put off doing this for so long because I thought it would be difficult, but it’s literally 3 lines of python, 1 line in pyproject.toml, and 9 lines of juliapkg.json. This is astoundingly good. Thank you SO much, @cjdoris!

---

<div class="post-metadata">

### Author: ![MilesCranmer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/milescranmer/32/21070_2.png) [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)
#### Post date: [May 24, 2024, 10:09am UTC](https://discourse.julialang.org/t/current-best-practices-for-julia-dependencies-in-python-packages/109433/13 "2024-05-24T10:09:46Z")

</div>

And now there’s also a conda-forge feedstock for juliacall: [GitHub - conda-forge/pyjuliacall-feedstock: A conda-smithy repository for pyjuliacall.](https://github.com/conda-forge/pyjuliacall-feedstock). Which I can confirm works nicely in an environment setting (PySR has switched entirely to JuliaCall)