Tentative Julia in Python project, any up-to-date tips e.g. managing dependencies?

General Usage
,
1

For context, I’m jotting down a list of Python packages for a project idea, and it will be beneficial to do some very simple number crunching without allocations. I considered the out argument of NumPy and Numba, but the current limitations will make some things very painful to get working. AFAIK the program will allocate everything at the start and update them at regular timepoints given touchscreen positions, so I’m thinking I could write the number crunching part in Julia (it’s an LLVM-based JIT compiler like Numba anyway) and use PythonCall to share the array with the Python side.

Before I start experimenting, I want some tips from people who’ve done Python-Julia interop. My wants:

  1. Easily reproducible environment that manages dependencies from both languages
  2. be able to import the package into either Julia or Python and run the program.
  3. probably will end up as Kivy app, so if there’s any issues with doing that from Julia or PackageCompiler, a warning would be appreciated

Initially I had considered working with Conda for multi-language support and making this a Julia-in-Python project, but I read through this thread where several people opined that Conda is still primarily a Python package manager that tacked on support for some more languages, so Julia support was still impractical then. It’s still not clear to me if Julia-in-Python or Python-in-Julia is more practical or what sort of package manager setup is smoothest right now.

It’s still tentative because:

  1. I also want to consider the relative memory overhead of the Numba compiler vs the Julia process
  2. I plan to have a customizable GUI and use higher order functions in the number-crunching side to customize the processing, but it seems unwieldy to have extensible code in two languages. This might work itself out when I get good enough at Kivy and PythonCall to make something.

, but managing dependencies the right way and getting it started comes first, hence this post.

I did find this somewhat outdated stackoverflow post about profiling the memory cost of imports in Python, but I’m not certain if it covers the full overhead. If anybody has a similar thing for the Julia side that can measure the process’s overhead or Numba usage, let me know too.

2

Using PythonCall.jl and CondaPkg.jl from Julia is pretty much seamless; the major problem is when you need a package from PyPI, in which case dealing with pip is always a nightmare.

2 Likes
3

I definitely need to do this. Is using pip from conda from Julia any worse than using pip from conda, or are they the same amount of nightmare?

4

Same amount of nightmare, then. It’s best to just use Conda whenever possible, possibly using conda skeleton to build the package from PyPI.

1 Like
5

This is the first time I’ve heard of this. Does this always work without any issues? Why on earth are people being told to tiptoe around duplication issues when using pip and conda or pip via conda when there’s a mechanism to make a conda package from a PyPI package? Why wouldn’t conda-forge just be a bot-maintained superset of PyPI by now?

6

Switching from conda to virtualenv+pip now that more often than not, the Python packages I’ve found aren’t available via conda, at least easily. This is shaping into a Julia-in-Python project now because 1) more and bigger Python dependencies so far, and 2) Julia’s PythonCall works with conda environments, whereas Python’s JuliaCall can be installed easily with pip.

Now I think about it, is it even feasible to use PythonCall to wrap a JuliaCall-in-Python package into a matching Julia package? I guess I’ll find out after I learn virtualenv and get the ball rolling. God I wish Python converged on 1 canonical package/environment manager like Pkg.

1 Like
7

You can still use PythonCall with pip.

There’s no good reason this isn’t easier, except that people whine about how binaries and packages are different things (even though end users just don’t care and this doesn’t do anything except confuse and frustrate them).

However, there are some real difficulties with it, namely that Conda and PyPI packages may have different names, or even different version numbers; python packages don’t use SemVer; and you need to figure out which versions belong together. This is all much easier if you only have to deal with Conda packages.

This is why I like to joke that Python and Julia switched package managers :wink: In Julia, everything is like this except the package manager. But if you do want a semi-canonical package manager that isn’t awful, Conda and Poetry are your two best bets (except Poetry doesn’t do Conda binaries).

4 Likes
8

How so? Reading the docs, its says using PythonCall makes a conda environment and to use CondaPkg to install packages.

9

When I used PyCall.jl I only used pip, all I knew, now know of Conda, and it seems better. One question, why would you rather use pip, does it have packages not in Conda?

What I think you’re after is:

To force PythonCall to use the same Python interpreter as PyCall, set the environment variable JULIA_PYTHONCALL_EXE to “@PyCall”. Note that this will opt out of automatic dependency management using CondaPkg.

You are always using CPython, i.e. the same interpreter, so the wording may be misleading. It’s just that without doing the right thing you can’t use both, then both will try to initialize, so e.g. using both, in (same or) different dependencies of your project wouldn’t have worked.

That’s the reason for this, more so than opting into pip, and out of Conda. When I used PyCall.jl I just used the standard Python installation, and thus had to install my Python packages there. I believe you can still use virtual envs. Is this what you’re after, only pip, or does it make sense to use pip and Conda together, i.e. for different dependencies? I think you would still be able to do that, i.e. use Conda with, just not PythonCall.jl (or rather it’s dependency CondaPkg.jl) taking care of it for you.

Note, Python 3.12 is just out, and dropping a lot of outdated stuff, e.g. modules; ensurepip (and distutils). Whatever you do I think you should use it (and going forward PythonCall.jl should only support it, if it’s already better for UTF-8, for better integration, or whatever later version that enables that).

It seems like a very improved and important release, largest update in a while? With e.g. “Per-interpreter GIL”, and while UTF-8 was actually usable internally, as a redundant representation, including in all currently supported Python, but might now be more usable as the default. It’s still not the only representation for strings in Python, that is planned, but 3.12 dropped some fields from the string object to make it possible. It’s still Python compatible (just dropping/breaking part of the C API).

I think it’s very valid to have Julia as your main language calling Python, possibly less ideal if you use pip. If you have lots of Python code you could still to that, or alternatively use Python as your main language and call Julia. One reason to do that is frameworks (e.g. for the web or GUIs) want to call you (“the Hollywood principle”), and then Python may be better as the main language. PythonCall.jl and PyCall.jl support all Python libraries, i.e. at least non-framework libraries. Frameworks are libraries too, strictly speaking, just very unusual. I’m not sure they are supported (well), or at least then it may matter what your main language is. Also threading may be an issue, and have an influence what you choose (as your main language).

2 Likes
10

Yes, as I’m looking up candidate packages and jotting them down, a large number of them are on PyPI but not Anaconda or conda-forge. I suppose I shouldn’t be surprised, PyPI has 486k projects to Anaconda’s 7.5k packages and conda-forge’s 22.9k packages, a sizable “official repository” advantage. I’m not very happy about the switch because I’m used to conda, but since PythonCall/JuliaCall involves Pkg to manage the Julia dependencies, maybe tried-and-true Python-specific environments won’t be an issue. I read that conda environments aren’t so good at managing Julia anyways.

Yes it’s also appearing this way. I also am pretty certain that the project will juggle several large Python dependencies, while I don’t think I’ll have to go much beyond base Julia for the number crunching, so far just PrecompileTools to save some compiled methods. Easier to write than Numba at least.

11

CondaPkg can install pip packages (see the docs for more info).

1 Like