[ANN] CondaPkg.jl v0.2.6 - now with REPL integration!

Package Announcements
,
23

Dependencies are installed when you load the package only out of necessity - the Julia package manager does not have suitable hooks to run code when packages are added to a project. You can do stuff in a build script (deps/build.jl) or at precompile-time, but this only happens once per version of a package - if the same version of a package is used in two different projects, it only gets built once, whereas CondaPkg would need to install packages twice (once into each project) because those projects may have a different set of Conda dependencies to satisfy.

I agree it would be nice to do the installation at add time, but as far as I can tell, Pkg would need to add some sort of add-time hook, e.g. a script similar to deps/build.jl that is run whenever the package is added to a project.

If this is an issue, you can always manually do Pkg.add("PackageA"); CondaPkg.resolve(); to immediately install your Conda packages. Or

pkg> add PackageA

pkg> conda resolve

As for where CondaPkg puts the environment, the rule is that whenever you start a fresh Julia session and load CondaPkg (e.g. indirectly through PackageA), it puts the Conda environment into the topmost project in your load path which depends on CondaPkg. Hence if you start Julia in the v1.8 environment, and you have PackageA installed there, then it depends on CondaPkg, so the Conda environment is put into your v1.8 environment. The Conda dependencies from any other PackageB or PackageC in your v1.8 environment are installed there too.

1 Like
24

I currently have several .CondaPkg folders, two of them taking 1.5 GB each. I am using Julia 1.8.5 on Win 11.

How many of these folders should we normally expect?

Also, is it possible to select the directory path where this .CondaPkg resides?

Thank you.

2 Likes
25

By default CondaPkg creates a new Conda environment for each Julia project. This is to keep Python dependencies separate across projects.

For the most part, Conda hard links installed files from the Conda root directory. This means that if two environments are using the same package at the same version, then the package is not duplicated.

This means that it is possible to have two Conda environments taking 1.5GB of disk space individually, but together they might only take 1.5GB total if they share packages.

There is a PR I need to review which adds the option to have a single central environment instead.

3 Likes
26

I’ve been using add `JULIA_CONDAPKG_ENV` for centralized environment by t-bltg · Pull Request #53 · cjdoris/CondaPkg.jl · GitHub for a month with multiple julia versions (at least 3 and each one being an official download and a custom build with patches, total 6) using a single shared conda env (1.6Gb), and it has been working as expected.
We still need to fix some ci failures in that PR and it also needs review.

3 Likes
27

@rafael.guerra, you can now use JULIA_CONDAPKG_ENV with the latest CondaPkg release for a single centralized conda env.

2 Likes
28

@t-bltg, thanks for this very useful feature, but please bear with me.

I’m on Win11 Julia 1.8.5 and I’m a bit lost, to say the least.

After installing PythonPlot in a project and calling Plots’ pythonplot() backend, a 1.5 GB .CondaPkg folder was created.

Then I added to the startup.jl file the following line pointing to the .CondaPkg folder (is there another way?):

ENV["JULIA_CONDAPKG_ENV"] = "C:\\Users\\jrafa\\OneDrive\\Julia_Code\\.CondaPkg"

Then, I restarted Julia, activated another Julia 1.8.5 project and added PythonPlot.

However, when I call Plots’s pythonplot() backend, it does not compile. It complains:

error    libmamba Expected environment not found at prefix: C:\Users\jrafa\OneDrive\Julia_Code\.CondaPkg
critical libmamba Aborting.

Any hint, please.

29

Setting JULIA_CONDAPKG_ENV in startup.jl is also what I do.

I would suggest backing up the existing .CondaPkg folder somewhere else on your system, and then remove it from [...]/.julia/environments/v1.8 where I assume it is present (I’m writing the path in a unix way, since I never used windows).

Then with the JULIA_CONDAPKG_ENV set, use:

julia> ENV["JULIA_CONDAPKG_ENV"]
"C:\\Users\\jrafa\\OneDrive\\Julia_Code\\shared_conda_env"  # I'd advise against using ` .CondaPkg` name in it
julia> using CondaPkg
pkg> conda add matplotlib
[...]
pkg> conda status
CondaPkg Status [...]\\.julia\\environments\\v1.8\\CondaPkg.toml
Packages
  matplotlib v3.6.2
julia> CondaPkg.envdir()
"C:\\Users\\jrafa\\OneDrive\\Julia_Code\\shared_conda_env"  # should point to `JULIA_CONDAPKG_ENV` ...

So now CondaPkg writes the specs / dependencies in a per-julia version .toml file, but installs them in a shared environment somewhere else.

3 Likes
30

@t-bltg, That’s great, it worked.
Now I only have one shared_conda_env directory (1.5 GB) which is shared by the different projects.

Thanks for the brilliant black magic.

NB:
Perhaps your detailed explanations should go to the link your provided JULIA_CONDAPKG_ENV

3 Likes
31

Agreed, add `JULIA_CONDAPKG_ENV` example by t-bltg · Pull Request #69 · JuliaPy/CondaPkg.jl · GitHub. Please comment in there if something is unclear and / or missing.

3 Likes