Looks fantastic!
How does this package relate to Conda.jl/PyCall.jl, if at all? Can I, for example, easily use CondaPkg.jl to manage my python dependencies and call them with PyCall? I love the idea of cleanly having a mini-conda environment specific to my active Julia environment.
Same question as @jondeuce here in terms of interplay with PyCall. For instance
using PyCall
using CondaPkg
py"""import gpy""" # fails (no module called gpy), ok I've not installed it, fine
CondaPkg.add("gpy")
CondaPkg.withenv() do
py"""
import gpy
"""
end # installs GPy but the pycall code fails (no module called gpy)
Surprisingly I would have expected this to work but it doesn’t: Edit see answer below
It feels like all this could work with the proper settings and that I just didn’t know what the proper settings are, maybe a demo with the above would be nice!
The short answer is: not at all. The Python interpreter that PyCall uses is built in to the package at build-time, which means it cannot select a different interpreter based on which Julia project it is called from. Furthermore, it uses Conda.jl which also mainly manages a single global root Conda environment (to be fair, you can explicitly pass an environment to Conda.jl to use, but PyCall doesn’t use this functionality).
It’s because of these shortcomings that I wrote CondaPkg in the first place.
I also wrote PythonCall which is essentially a ground-up rewrite of PyCall. By default it uses Python from CondaPkg, which basically means that CondaPkg is the package manager for PythonCall.
In order for PyCall to use CondaPkg, it would first need to be modified to be able to select the Python interpreter at run-time, not build-time. I’m not sure how much work that would be - personally I’m invested in PythonCall instead.
The Python module is called GPy not gpy!
julia> CondaPkg.withenv() do
run(`python -c 'import GPy; print("hello")'`)
end
hello
Process(`python -c 'import GPy; print("hello")'`, ProcessExited(0))
This is nice, I wasn’t aware of PythonCall and CondaPkg makes more sense to me now. Maybe a link to PyCall in the original post at the top would be useful to others who might not be aware of PythonCall?
Yea, I used to be a somewhat heavy user of PyCall.jl, but since discovering PythonCall.jl+CondaPkg.jl+MicroMamba.jl, I haven’t looked back. The fact that this fresh take on python interop uses project specific envs AND scratch spaces AND artifacts AND versatile show methods AND …
is just insane. I mean c’mon:
Good idea, done, thanks.
Awww shucks 
Thank you for that amazing package!
Today I was a bit surprised about the following: I have a package A depending on CondaPkg, and a package B that depends on package A.
What I observed now is that there is also a .CondaPkg environment added to package B. Is this intended or am I doing something wrong?
Assuming you activated package B this is expected - CondaPkg puts its environment into the topmost project in your load path which depends on CondaPkg.
Is there a way to add a pip package that is not in any channel, but only in a git repository? I am trying to reproduce something like pip install git+https://<path to github repo>. This works if the repo has a pyproject.toml file.
Sure, you do
pkg> conda pip_add '<package_name> @ git+https://.....'
or equivalently
julia> CondaPkg.add_pip("<package_name>", version="@ git+https://.....")
Thanks for your quick reply! Will this work if the url has an @ in it? The error that I am seeing is multiple direct references ('@ ...') given for pip package 'hello-python'
No, that error means that multiple conflicting @ ... versions have been defined for the same python package in different CondaPkg.toml files.
Indeed, got it to work after cleaning out the environment
When using CondaPkg.toml in PackageA, it seems that Python dependencies are installed when we do using PackageA rather than when we do add PackageA. Is this the expected behavior? Can you explain why this is preferred?
Also, if we are in v1.8 environment, are the dependencies installed under the conda environment under v1.8 or are they installed under PackageA’s conda environment?
Dependencies are installed when you load the package only out of necessity - the Julia package manager does not have suitable hooks to run code when packages are added to a project. You can do stuff in a build script (deps/build.jl) or at precompile-time, but this only happens once per version of a package - if the same version of a package is used in two different projects, it only gets built once, whereas CondaPkg would need to install packages twice (once into each project) because those projects may have a different set of Conda dependencies to satisfy.
I agree it would be nice to do the installation at add time, but as far as I can tell, Pkg would need to add some sort of add-time hook, e.g. a script similar to deps/build.jl that is run whenever the package is added to a project.
If this is an issue, you can always manually do Pkg.add("PackageA"); CondaPkg.resolve(); to immediately install your Conda packages. Or
pkg> add PackageA
pkg> conda resolve
As for where CondaPkg puts the environment, the rule is that whenever you start a fresh Julia session and load CondaPkg (e.g. indirectly through PackageA), it puts the Conda environment into the topmost project in your load path which depends on CondaPkg. Hence if you start Julia in the v1.8 environment, and you have PackageA installed there, then it depends on CondaPkg, so the Conda environment is put into your v1.8 environment. The Conda dependencies from any other PackageB or PackageC in your v1.8 environment are installed there too.
I currently have several .CondaPkg folders, two of them taking 1.5 GB each. I am using Julia 1.8.5 on Win 11.
How many of these folders should we normally expect?
Also, is it possible to select the directory path where this .CondaPkg resides?
Thank you.