[ANN] CondaPkg.jl v0.2.6 - now with REPL integration!

(I don’t often post about new versions of packages, but I thought this was was pretty cool, so…)

I’ve just released v0.2.6 of CondaPkg.jl, the Pkg-like package for adding Conda dependencies to your Julia project.

Pkg REPL

The main cool new feature is that it now integrates into the Pkg REPL, so you can add and remove Conda packages interactively as easily as adding Julia packages. An example from the README:

julia> using CondaPkg
julia> # now press ] to enter the Pkg REPL
pkg> conda status                # see what we have installed
pkg> conda add python perl       # adds conda packages
pkg> conda add --pip build       # adds pip packages
pkg> conda rm perl               # removes conda packages
pkg> conda run python --version  # runs the given command in the conda environment

Status

The output from CondaPkg.status() (or pkg> conda status) has been revamped, and now includes the currently installed versions of packages. For example:

pkg> conda st
CondaPkg Status /root/.julia/environments/v1.7/CondaPkg.toml
Packages
  boto3 v1.20.45
  pyyaml v6.0
  python v3.10.2
  pydantic v1.9.0

Pip @ dependencies

We now support Pip’s direct reference syntax, such as foo @ http://example.com/foo.zip. As a special case, if the URL starts with a dot then it is interpreted as a path relative to the folder containing the CondaPkg.toml file. This means you can ship a Python package with your Julia package and have it installed with an entry like foo = "@ ./foo.zip" in CondaPkg.toml.

PythonCall

The reason this package exists in the first place is to support PythonCall.jl, which is essentially a total rewrite of PyCall. I’ll write more about it in another post one day, but crucially PythonCall uses the Python interpreter installed by CondaPkg. This means that each Julia project which uses PythonCall has its own independent set of Python dependencies managed by CondaPkg.

44 Likes

Amazing!

Looks fantastic!

How does this package relate to Conda.jl/PyCall.jl, if at all? Can I, for example, easily use CondaPkg.jl to manage my python dependencies and call them with PyCall? I love the idea of cleanly having a mini-conda environment specific to my active Julia environment.

Same question as @jondeuce here in terms of interplay with PyCall. For instance

using PyCall
using CondaPkg

py"""import gpy"""  # fails (no module called gpy), ok I've not installed it, fine

CondaPkg.add("gpy")
CondaPkg.withenv() do
  py"""
  import gpy
  """
end  # installs GPy but the pycall code fails (no module called gpy)

Surprisingly I would have expected this to work but it doesn’t: Edit see answer below

It feels like all this could work with the proper settings and that I just didn’t know what the proper settings are, maybe a demo with the above would be nice!

1 Like

The short answer is: not at all. The Python interpreter that PyCall uses is built in to the package at build-time, which means it cannot select a different interpreter based on which Julia project it is called from. Furthermore, it uses Conda.jl which also mainly manages a single global root Conda environment (to be fair, you can explicitly pass an environment to Conda.jl to use, but PyCall doesn’t use this functionality).

It’s because of these shortcomings that I wrote CondaPkg in the first place.

I also wrote PythonCall which is essentially a ground-up rewrite of PyCall. By default it uses Python from CondaPkg, which basically means that CondaPkg is the package manager for PythonCall.

In order for PyCall to use CondaPkg, it would first need to be modified to be able to select the Python interpreter at run-time, not build-time. I’m not sure how much work that would be - personally I’m invested in PythonCall instead.

14 Likes

The Python module is called GPy not gpy!

julia> CondaPkg.withenv() do
           run(`python -c 'import GPy; print("hello")'`)
       end
hello
Process(`python -c 'import GPy; print("hello")'`, ProcessExited(0))
1 Like

This is nice, I wasn’t aware of PythonCall and CondaPkg makes more sense to me now. Maybe a link to PyCall in the original post at the top would be useful to others who might not be aware of PythonCall?

3 Likes

Yea, I used to be a somewhat heavy user of PyCall.jl, but since discovering PythonCall.jl+CondaPkg.jl+MicroMamba.jl, I haven’t looked back. The fact that this fresh take on python interop uses project specific envs AND scratch spaces AND artifacts AND versatile show methods AND …

is just insane. I mean c’mon:

19 Likes

Good idea, done, thanks.

1 Like

Awww shucks :blush:

PythonCall.jl is Amazing.

Just there are fewer users of PythonCall.jl.

2 posts were split to a new topic: PyPlot for PythonCall

Thank you for that amazing package!
Today I was a bit surprised about the following: I have a package A depending on CondaPkg, and a package B that depends on package A.
What I observed now is that there is also a .CondaPkg environment added to package B. Is this intended or am I doing something wrong?

Assuming you activated package B this is expected - CondaPkg puts its environment into the topmost project in your load path which depends on CondaPkg.

Is there a way to add a pip package that is not in any channel, but only in a git repository? I am trying to reproduce something like pip install git+https://<path to github repo>. This works if the repo has a pyproject.toml file.

Sure, you do

pkg> conda pip_add '<package_name> @ git+https://.....'

or equivalently

julia> CondaPkg.add_pip("<package_name>", version="@ git+https://.....")

Thanks for your quick reply! Will this work if the url has an @ in it? The error that I am seeing is multiple direct references ('@ ...') given for pip package 'hello-python'

No, that error means that multiple conflicting @ ... versions have been defined for the same python package in different CondaPkg.toml files.

Indeed, got it to work after cleaning out the environment

1 Like