Integrating Julia into a python poetry project

Tooling
,
1

I have an existing python poetry project that I want to integrate some Julia stuff into. I noticed that there exists a conda-forge package for Julia, but there’s no osx-arm64 build for it, which is a problem for me.

Are there alternative good ways to build it in? I noticed there’s also pyjuliapkg, but it’s not clear to me how this integrates with poetry or where it puts stuff. It’s crucial that I be able to easily start the interpreter from within a shell in the corresponding poetry environment.

It would be most helpful if I could understand what precisely has to go into my pyproject.toml to pull Julia or some supporting utility in, and then what if any additional automation needs to be added to the project to set Julia up in the environment.

2

The best toolbox for Julia-Python interoperability is

More specifically, you’re interested in the Python package called pyjuliacall, which you can install with pip

2 Likes
3

It’s not clear to me that this package manages the installation of Julia or how. It seems to be two FFIs, one from python to Julia and vis. versa.

4

It automatically takes care of downloading and installing Julia + creating an environment, see

5

A ha – ok, so this uses juliapkg. That makes sense.

1 Like
6

Oh one additional question – if I want to mix python and Julia source in my project, how is that done? That is, if I build a wheel or something like that, how does Julia find its way to the Julia source files from python?

7

I would assume you just call jl.include("myjuliafile.jl") but I have never used this direction before so no clue

8

See here: [ANN] pyjuliacall and pyjuliapkg packaged on conda-forge

FYI: Actually the py prefix there isn’t part of the package (and I looked in the daocs with the py prefix and fgot search results, but some, thus likely all bogus), and you don’t use it when installing from pip.

But in Conda you use the py prefix to install, and it gets you the same package…

I don’t know if you use with or without with Poetry. I first now installed it but don’t care enough to learn it… There seems to be too many options for Python, is that one the best? I installed it with apt-get, and then saw in docs, I could/should have used pipx, yet another option. I’ve only really used pip with Python, am aware of some others e.g. Conda. Don’t know if Julia (Python) package needs to supports the other in some way. I’t much simple in Julia, one default built-in package manager, Pkg, all use.

9

If you want to see a full example of putting everything together working you could check out PySR? It uses PythonCall.jl and pyjuliapkg to connect to SymbolicRegression.jl and then uses the standard pyproject.toml to manage the Python side. It also has a conda forge feedstock set up – GitHub - conda-forge/pysr-feedstock: A conda-smithy repository for pysr..

For including Julia code in the Python project I would do what @gdalle says. If using in a package, be sure to put it inside a new module (to prevent namespace mixing), like:

from juliacall import newmodule
jl = newmodule("MyPackage")

jl.include("myjuliafile.jl")
# ^Now all the code in `myjuliafile.jl` will be available in `MyPackage`

for shorter snippets you can just use seval:

jl.seval("""
function f(x)
    x ^ 2
end
""")
3 Likes
10

Ok I think we’re getting a lot closer to what I’d like to do. Let’s suppose what we want is a project structure like the following

project/
     project.jl/Project.toml
     project.jl/src/thing.jl
     pyproject.toml
     project/otherthing.py

so the Julia project is in a subfolder of the main python project. What’s the best way to stick these two projects together?

11

In pyjuliapkg you can specify relative local directories to packages so I would do that from within your Python package’s juliapkg.json.

See GitHub - JuliaPy/pyjuliapkg: Manage your Julia dependencies from Python.

Then you can just do jl.seval("using <YourProject>") and it will load that package.

And your pyproject.toml should go in the root directory and specify what code needs to be included in the package.

2 Likes
12

Wow, I just checked juliacall (and juliapkg) again after not looking at it for a while, and it is so seamless now! I needed the subdir option in juliapkg, which was added two weeks ago, and now everything literally installs so easily into a python env and is super usable. Thanks for the pointer!

2 Likes