Understanding Pkg.add() and Environments

TLDR: When I add a package Julia installs the package and any dependencies (expected), then it says Precompiling project and appears to go through every package that I have ever used and spends a lot of time compiling everything.

Why does this happen and how can I quickly add a single package?

Full story: I want to check some math, so I open up the REPL and type using Unitful turns out I don’t have it on my personal laptop, so it asks if I want to add it and I say yes. It then downloads and installs several things, all of which I assume are either part of or needed for the Unitful.jl package, following that it updates Project.toml and Manifest.toml then it says Precompiling project and appears to go through every package or little thing that I have (apparently 299 even though I only have 41 packages according to the Project.toml), and takes quite a while to do so.
299 dependencies successfully precompiled in 2749 seconds. 327 already precompiled. (just got this output)
I was a bit confused about this so I started reading the packages docs, then I came here and saw this post on updating packages, following skimming that I realized I should learn more about environments and projects.
Looking over the Pkg.jl docs on Environments I think I learned the following- An environment is the space my current code or project is living in or drawing from, all available packages in that environment are in the Project.toml. Manifest.toml has their definitions and dependencies. I went to my environments folder and found I have 2, v1.9 and _pluto_boot… I’m guessing this means that all of my coding- firing up the REPL for a quick check, or extended coding and projects that I do in VScode out of different folders, are all operating out of the same global v1.9 environment.
Somewhere in my readings I saw that environments take up very little space, so why not make a lot? In my head, after having readingthat and the environments docs, this means every big project I have that I do in VSCode I should probably have an environment for each. How do I point a specific folder that I code in to a specific environment? Or do I somehow put that environment in the folder? I saw on the docs that a benefit of this is that the Project.toml can stay associated with one of my git repos, so that makes me think the environment and thus the Project.toml must be in the same folder I have the repo?
I learned about temporary environments, that seems like the fastest light-weight way to only work with what I need, so maybe I add to my startup file a check to see if I am in VSCode, and if not, to just start with a temp environment? Is that the Julian way of doing things?
The docs said “By default, any package that is added to a project or updated in a Pkg action will be automatically precompiled, along with its dependencies.” So when I added just 1 package, why did it precompile so many? Does it somehow think they are all dependencies? The sentence before that said “However, neither of these should be routinely required thanks to Pkg’s automatic precompilation.” So I think the environment always has everything precompiled, but when I add a package it does it all again?

Sorry for all of the questions, but I don’t fully understand Environments or projects and the proper way to work with them and would love to learn the proper way to code!
Thanks so much! Let me know if this is answered elsewhere or if there is a better way to learn or ask questions here!

1 Like

For each project, create an environment:

mkdir my_project
cd my_project
julia --project="."

and then add the packages you need for this project.

Further reading:

If you open a folder in VSCode it will ask you if you want to activate
the project that lives in this folder. Just say yes.


Thank you! When I open up the REPL for some quick work, does it always default to the full v1.9 (or whatever version I am on) environment? How can I / is it better to have it create a new environment even if I am just doing some quick checking or math?

For just adding and using a single package one time I do this.

(@v1.10) pkg> activate --temp
  Activating new project at `/tmp/jl_8DiyZ6`

(jl_8DiyZ6) pkg> add Unitful

If I might reuse the quick environment, I will create a shared one.

(@v1.10) pkg> activate --shared unitful
  Activating new project at `~/.julia/environments/unitful`

(@unitful) pkg> add Unitful

The next time I come back, the environment can be reactivated.

(@v1.10) pkg> activate --shared unitful
  Activating project at `~/.julia/environments/unitful`

(@unitful) pkg> st
Status `~/.julia/environments/unitful/Project.toml`
  [1986cc42] Unitful v1.19.0
1 Like

I cannot say to what extent this is “best practice,” but it works for me:

I set in my .bashrc this alias

alias jl='JULIA_LOAD_PATH="@:@stdlib:@dev" julia -q --project=.'

where @dev is a shared environment where I put packages I use frequently interactively (e.g. BenchmarkTools.jl, Arrow.jl, OnlineStats.jl )

and otherwise the --project=. makes a new project for each workspace I’m in and I can add more particular dependencies as needed

1 Like

I have a few packages in the global environment myself:

Status `~/.julia/environments/v1.10/Project.toml`
  [8b8fa57b] BayesOpt v0.1.0
  [e30172f5] Documenter v1.3.0
  [35a29f4d] DocumenterTools v0.1.19
  [16fef848] LiveServer v1.2.7
  [a562b63c] PkgHelpers v0.3.0 `https://github.com/ufechner7/PkgHelpers.jl.git#main`
  [295af30f] Revise v3.5.14
  [1e6cf692] TestEnv v1.101.1
  [3c954167] WeatherReport v0.21.1

Thats OK for me, but there should not be too many packages in the global environment because they can cause conflicts… A few should be fine…

1 Like

My workflow is to set this st the beginning of my “main” file in a project I work with:

using Pkg
Pkg.activate(@__DIR__) # activate an environment at the dir containing the file
#Pkg.instatiate() # This only if I move to another pc or give my oroject with someone else

[... rest of the code...]

You can also look on my online course


My workflow is similar to that of sylvaticus. Given that this is a recurring question, I am adding a very simple example here. One thing I would like to recommend is using the compat block in the Project.toml file. On this line, you can see that I added a constraint on the version of Distributions.jl. This will ensure that your code is reproducible across time by requiring you to opt into breaking changes. Another benefit is that local environments have fewer version conflicts because they have fewer dependencies.

1 Like

Thank you all so much! This is very helpful and I am learning a lot. I have the Julia extension in VSCode, the bottom left of VSCode says the Julia env is my global default. In the VSCode julia terminal I activated a new project, that I want to use from here on out in this folder. I was hopeful that would change the environment automatically, but it didn’t. Clicking on it and navigating to both the folder I am in and the Julia environments folder, I could not find the environment I just created in either?

If you open a folder that contains a Project.toml file in VSCode with the File menu it should ask you if you want to change the environment. Is that not sufficient?

I also use the Project Manager extension of VSCode with makes switching between projects even easier…