I have realized I am a little confused about one aspect of the behavior of Pkg.
In Python, local virtual environments are provided by venv. (Other tools also exist.)
You use Python to run venv, which is a globally accessible Python module. Creating a virtual environment in the current working directory will create a folder which contains any packages which are added when this local environment is active.
It works like this:
$ which pip3
/usr/bin/pip3 # can't remember exactly where it lives, but
# by default the package manager is the global instance
$ python3 -m venv .venv # the directory is named `.venv`
$ ./.venv/bin/activate
(.venv) $ # the environment is now active
(.venv) $ which pip3
./venv/bin/pip3 # the package manager is now the
# local package manager, rather than the global instance
(.venv) $ pip3 install pandas
...
# the folder `.venv` now contains the downloaded
# package `pandas`
With Julia and Pkg, the situation is somehow different. A Project.toml is created, but there is no local directory which stores data for downloaded packages.
As far as I am aware, there is also no way to “activate” the environment in your current shell, such that pkg> add X installs X in the local environment rather than defaulting to the global (system wide) one.
Just wondering if anyone can provide some helpful commentary on some of these points? In particular, how does this machinery work, without an equivalent of a (local) .venv folder?
And to answer your second question, the packages are installed in .julia/packages. Yes, this folder is shared by all environments, but which specific versions are loaded for a specific project is stored in the file Manifest.toml.
So if you start Julia in a project folder using either julia --project or start Julia in the global environment and than activate the local project, then Julia loads the exact package versions as remembered in the Manifest.toml file.
I’m not a Python so not sure what venv does user but are you asking why doesn’t Julia install the actual packages (i.e. their source code/compiled code/binaries) in a local folder? So that if you do:
you’d want to have two copies of DataFrames on your machine? Indeed that isn’t happening in Julia, rather Pkg will check whether the required version of a specific package is already installed and if so just use that.
Why would you want to duplicate that installation?
Probably you wouldn’t. Maybe the way Python’s venv works is a less sensible approach.
One advantage this does have is you can “build” your environment and ship everything in a Docker container relatively easily by just copying the parent folder. This will also copy the .venv subfolder. It’s only useful if you are deploying to a machine with the same CPU architecture. (Some packages such as numpy are architecture dependent iirc.)
In some sense, you might consider the Manifest.toml file that is created when you instantiate the environment to be the “equivalent” to the local .venv folder.
As the previous responses point out, the actual data for the installed packages will be in ~/.julia, in a way that reuses data if you have multiple environments that use the same version of the same dependency.
In Python, you get basically the same behavior if you manage environments with Poetry instead of the standard-library venv module. Many of the other “modern” Python packaging solutions have also switched to keeping the environment in a central location, but I’ve found Poetry in particular to be extremely close to how Pkg works in Julia.
To give my personal opinion - I don’t really like to hide machinery which changes how Julia behaves when it starts up. It becomes easy to forget this isn’t the default if you have to go help someone else or work in a different environment. (eg a server somewhere which you do not control)
I do quite like the alias suggestion, because this introduces a separate command.
Regardless, thanks for these suggestions, they are still interesting ideas to consider.
I’m missing what you’re saying that is possible in Python but not in Julia. If you instantiate a Julia environment in a docker image, all its content is automatically present in the depot (by default it’d be the .julia subdirectory of the home of the user, but if you want you can also set an environment variable to use some other directory), without having to manually copy files around. Also, if I remember correctly, venv creates symlinks to the python executable, if you copy that directory from the host machine to inside the container you may end up with broken links.
Julia optionally does function multiversioning, which means that if you set the JULIA_CPU_TARGET environment variable appropriately the compiler generates code for a function optimised for all the chosen microarchitectures and that can be saved in the package precompile cache, and the program will decide at runtime which version of the native code to pick based on the CPU available.