Pkg & activation

BLI · May 25, 2024, 3:04pm

OK – I’m a simple Pkg user… Here is my problem:

I have several Jupyter notebooks in the same directory.
In this directory, I have a Manifest.toml and a Project.toml file. I only want to use these local *.toml files in one of my Jupyter notebooks. Reason: one of the notebooks contains some ModelingToolkit code that requires an older version of ModelingToolkit. The other Jupyter notebooks depend on a newer version of MTK.
My understanding of Pkg.jl and the Environment feature is that…
– I need to invoke use of a local environment by the Pkg.activate(...) command, where, e.g., Pkg.activate(".") makes Julia look for the *.toml files in the current directory where the Jupyter notebooks are located.
– If I do not use the Pkg.activate(..) command, Julia will use the global environment.
Problem: I try to run a Jupyter notebook from this directory without invoking the Pkg.activate(...) command. So I expected Julia to use the global environment. However:
– Julia tells me that I need to instantiate the environment!! [I understand the need for that if Julia actually uses the local environment.]
– If I change the filenames (local Manifest.toml → _Manifest.toml; local Project.toml → _Project.toml), then Julia correctly uses the global .toml files.

Is my understanding of activation completely wrong? In other words, is my Point 3 above wrong??

mkitti · May 25, 2024, 3:53pm

Run Pkg.status() to figure iut which environment your Jupyter kernel is running in.

BLI · May 25, 2024, 4:18pm

OK – Pkg.status() tells me that I’m in the local directory!!

C:\Users\Bernt\OneDrive\...\Notebooks\Project.toml` (empty project)

This is so, even though I did not do Pkg.activate(".")!!

So this implies that (at least when I run Jupyter Notebooks in VSCode), command Pkg.activate(".") is not needed – it will always first search for the *.toml files locally!
If there is no *.toml files locally, it will go to the global environment and use the *.toml files there.

This also implies that I can not use my strategy of having multiple related notebooks in one directory where one of them uses a local environment! (Unless I explicitly specify the global environment in every notebook that do not use the local one?)

Perhaps I misunderstand the role of Pkg.activate()… I thought that was meant to specify what is the active environment, and if unspecified, the global environment is used. But instead, it seems to just be to activate the creation of local *.toml files??

mkitti · May 25, 2024, 4:56pm

You might want to check

The Jupyter kernel
The notebook

It might be better to explicitly activate an environment in the notebook for reproducibility.

My suspicion is that the environment you installed IJulia with us the environment that the kernel may activate, but it could be the local directory as well.

BLI · May 25, 2024, 6:01pm

OK… Here is what I have done for testing things:

I have created a new directory, C:\Users\Bernt\Slett\. [“Slett” means “Delete”…]
In this directory, I have created a test.jl file and a test.ipynb file using VSCode, both with the following code:

using Pkg
Pkg.status()

I also put a Project.toml file and a Manifest.toml file which a student created for me a while ago.

Now, the directory has the following content:

When running the code, both test.jl and test.ipynb [Julia release channel] respond with:

Status `C:\Users\Bernt\.julia\environments\v1.10\Project.toml`
  [336ed68f] CSV v0.10.14
  [a93c6f00] DataFrames v1.6.1
  [82cc6244] DataInterpolations v5.0.0
  [7073ff75] IJulia v1.24.2
  [b964fa9f] LaTeXStrings v1.3.1
  [23fbe1c1] Latexify v0.16.3
  [91a5bcdd] Plots v1.40.4

So far, so good: both codes ignore the local *.toml files.

I then clone the files as test_1.jl and test_1.ipynb, and in both of the new files, I add the commands:

Pkg.activate(".")
Pkg.status()

When I run the test_1.jl file, the response after these new lines is:

Status `C:\Users\Bernt\Project.toml` (empty project)

NB: OBSERVE that the files are in directory C:\Users\Bernt\Slett\. Probably because the VSCode was in directory C:\Users\Bernt by default… It is empty because there were no *.toml files in that directory.

When I run the test_1.ipynb file, the response after these new lines is:

Project JuliaProjectEnvironment v0.1.0
Status `C:\Users\Bernt\Slett\Project.toml`
→  [0c46a032] DifferentialEquations v7.13.0
   [b964fa9f] LaTeXStrings v1.3.1
→⌃ [961ee093] ModelingToolkit v8.75.0
   [91a5bcdd] Plots v1.40.4
   [37e2e46d] LinearAlgebra
Info Packages marked with → are not downloaded, use `instantiate` to download
Info Packages marked with ⌃ have new versions available and may be upgradable.
  Activating project at `c:\Users\Bernt\Slett`

This shows that:
a. The path of the Jupyter notebook is the same as the path where the notebook was located.
b. The notebook has found the content of the *.toml files created by the student.

I then go back to the files test.jl and test_1.ipynb. These do not contain the Pkg.activate() command.

Running the test.jl code, the response is:

Status `C:\Users\Bernt\.julia\environments\v1.10\Project.toml`
  [336ed68f] CSV v0.10.14
  [a93c6f00] DataFrames v1.6.1
  [82cc6244] DataInterpolations v5.0.0
  [7073ff75] IJulia v1.24.2
  [b964fa9f] LaTeXStrings v1.3.1
  [23fbe1c1] Latexify v0.16.3
  [91a5bcdd] Plots v1.40.4

No effect of the activation taking place in test_1.jl because it created empty *.toml files.

Finally, I re-run test.ipynb file (without an activation statement). Response:

Status `C:\Users\Bernt\.julia\environments\v1.10\Project.toml`
  [336ed68f] CSV v0.10.14
  [a93c6f00] DataFrames v1.6.1
  [82cc6244] DataInterpolations v5.0.0
  [7073ff75] IJulia v1.24.2
  [b964fa9f] LaTeXStrings v1.3.1
  [23fbe1c1] Latexify v0.16.3
  [91a5bcdd] Plots v1.40.4

Again, this is what I hoped would be the result: because I have not used the Pkg.activate(".") command, the global environment is used.

HOWEVER, in another Jupyter notebook, the notebook uses the local *.toml files even though I have not issued the Pkg.activate(".") command…

BLI · May 25, 2024, 8:29pm

OK – I did a little bit more testing…

I created a new file test_2.ipynb from test_1.ipynb by adding the line Pkg.instantiate().
I then run test.ipynb, and now, Pkg.status() responds with:

Project JuliaProjectEnvironment v0.1.0
Status `C:\Users\Bernt\...\Notebooks\Project.toml`
  [0c46a032] DifferentialEquations v7.13.0
  [b964fa9f] LaTeXStrings v1.3.1
⌃ [961ee093] ModelingToolkit v8.75.0
  [91a5bcdd] Plots v1.40.4
  [37e2e46d] LinearAlgebra
Info Packages marked with ⌃ have new versions available and may be upgradable.

What is noteworthy here is:
a. Now, after I used Pkg.activate(".") to activate the local directory, and then did Pkg.instantiate() to instantiate the versions of Julia + packages in the *.toml files I copied from directory C:\Users\Bernt\...\Notebooks\ into C:\Users\Bernt\Slett\, the *.toml files are taken from the directory of the Jupyter notebook.
b. The location of the file does, however, seem to point to the directory where I copied the *.toml files from.

I don’t find this very intuitive. I guess there is a “simple” explanation for why things are like this. Perhaps my failure to find this intuitive is that it works differently from what I was thinking it would do.

BLI · May 25, 2024, 9:40pm

OK – I may have figured out how to achieve what I need…

My “problem” is the following:

If I put *.toml files in the same directory as where I have several Jupyter Notebooks, and if I have activated this directory (Pkg.activate(".") statement in the Jupyter notebook) as an environment + have instantiated the packages listed in the *.toml file ==> Julia will use the *.toml files as default environment when I start the Jupyter notebooks. The consequence is that I can not easily use different environments for different related Notebooks.
Seems like I can get around this by instead creating the environment in a subdirectory of the directory of the notebooks, e.g., Pkg.activate("./MyEnv"). I can then instantiate the *.toml files there.
When I start up one of the notebooks, if I do not issue the activate command, Julia will not find *.toml files in the directory of the notebook (because they are in the subdirectory). Thus, the global environment will be used.
If there is a single notebook where I actually want to use the local environment, I simply do Pkg.activate("./MyEnv"), and now the local environment will be used.

Anyways, my “problem” stems from the fact that I did not realize that Juila will use a local environment if the *.toml files are located in the active directory.

OK – I hope this is correct… if so, it solves my problem.

mkitti · May 25, 2024, 9:53pm

See the documentation here:
https://julialang.github.io/IJulia.jl/stable/manual/installation/

danielwe · May 25, 2024, 10:40pm

With the default configuration, IJulia launches julia with --project=@., which activates any local environment that exists within the current folder or any of its parents. See Using IJulia · IJulia