(I don’t often post about new versions of packages, but I thought this was was pretty cool, so…)
I’ve just released v0.2.6 of CondaPkg.jl, the Pkg-like package for adding Conda dependencies to your Julia project.
Pkg REPL
The main cool new feature is that it now integrates into the Pkg REPL, so you can add and remove Conda packages interactively as easily as adding Julia packages. An example from the README:
julia> using CondaPkg
julia> # now press ] to enter the Pkg REPL
pkg> conda status # see what we have installed
pkg> conda add python perl # adds conda packages
pkg> conda add --pip build # adds pip packages
pkg> conda rm perl # removes conda packages
pkg> conda run python --version # runs the given command in the conda environment
Status
The output from CondaPkg.status() (or pkg> conda status) has been revamped, and now includes the currently installed versions of packages. For example:
pkg> conda st
CondaPkg Status /root/.julia/environments/v1.7/CondaPkg.toml
Packages
boto3 v1.20.45
pyyaml v6.0
python v3.10.2
pydantic v1.9.0
Pip @ dependencies
We now support Pip’s direct reference syntax, such as foo @ http://example.com/foo.zip. As a special case, if the URL starts with a dot then it is interpreted as a path relative to the folder containing the CondaPkg.toml file. This means you can ship a Python package with your Julia package and have it installed with an entry like foo = "@ ./foo.zip" in CondaPkg.toml.
PythonCall
The reason this package exists in the first place is to support PythonCall.jl, which is essentially a total rewrite of PyCall. I’ll write more about it in another post one day, but crucially PythonCall uses the Python interpreter installed by CondaPkg. This means that each Julia project which uses PythonCall has its own independent set of Python dependencies managed by CondaPkg.
How does this package relate to Conda.jl/PyCall.jl, if at all? Can I, for example, easily use CondaPkg.jl to manage my python dependencies and call them with PyCall? I love the idea of cleanly having a mini-conda environment specific to my active Julia environment.
Same question as @jondeuce here in terms of interplay with PyCall. For instance
using PyCall
using CondaPkg
py"""import gpy""" # fails (no module called gpy), ok I've not installed it, fine
CondaPkg.add("gpy")
CondaPkg.withenv() do
py"""
import gpy
"""
end # installs GPy but the pycall code fails (no module called gpy)
Surprisingly I would have expected this to work but it doesn’t:Edit see answer below
It feels like all this could work with the proper settings and that I just didn’t know what the proper settings are, maybe a demo with the above would be nice!
The short answer is: not at all. The Python interpreter that PyCall uses is built in to the package at build-time, which means it cannot select a different interpreter based on which Julia project it is called from. Furthermore, it uses Conda.jl which also mainly manages a single global root Conda environment (to be fair, you can explicitly pass an environment to Conda.jl to use, but PyCall doesn’t use this functionality).
It’s because of these shortcomings that I wrote CondaPkg in the first place.
I also wrote PythonCall which is essentially a ground-up rewrite of PyCall. By default it uses Python from CondaPkg, which basically means that CondaPkg is the package manager for PythonCall.
In order for PyCall to use CondaPkg, it would first need to be modified to be able to select the Python interpreter at run-time, not build-time. I’m not sure how much work that would be - personally I’m invested in PythonCall instead.
This is nice, I wasn’t aware of PythonCall and CondaPkg makes more sense to me now. Maybe a link to PyCall in the original post at the top would be useful to others who might not be aware of PythonCall?
Yea, I used to be a somewhat heavy user of PyCall.jl, but since discovering PythonCall.jl+CondaPkg.jl+MicroMamba.jl, I haven’t looked back. The fact that this fresh take on python interop uses project specific envs AND scratch spaces AND artifacts AND versatile show methods AND …
Thank you for that amazing package!
Today I was a bit surprised about the following: I have a package A depending on CondaPkg, and a package B that depends on package A.
What I observed now is that there is also a .CondaPkg environment added to package B. Is this intended or am I doing something wrong?
Assuming you activated package B this is expected - CondaPkg puts its environment into the topmost project in your load path which depends on CondaPkg.
Is there a way to add a pip package that is not in any channel, but only in a git repository? I am trying to reproduce something like pip install git+https://<path to github repo>. This works if the repo has a pyproject.toml file.
Thanks for your quick reply! Will this work if the url has an @ in it? The error that I am seeing is multiple direct references ('@ ...') given for pip package 'hello-python'
When using CondaPkg.toml in PackageA, it seems that Python dependencies are installed when we do using PackageA rather than when we do add PackageA. Is this the expected behavior? Can you explain why this is preferred?
Also, if we are in v1.8 environment, are the dependencies installed under the conda environment under v1.8 or are they installed under PackageA’s conda environment?
