Recommended way to install a Project/Manifest into the global environment

Tooling Package Management
1

If I have a project.toml and manifest.toml, is there a way I can add those packages and versions into the default environment? I know, I know… better to use project files but this is for building a reusable docker container for a jupyterhub environment.

If this isn’t possible with a combination of the CLI and using Pkg; etc. cleanly, is it recommended to do something shady like the following:

  • Put the project/manifest direectly inside of ~/.julia/environments/v1.10/ and overwrite if it exists
  • Start julia, activating the global environment and instantiate. Something like:
    julia --threads auto -e 'using Pkg; Pkg.instantiate();'

Or is that behavior undefined or unsafe?

2

If this a docker container, why not just overwrite ~/.julia/environments/v1.10?

You could also create another shared environment named whatever you would like then manipulate the environment variable JULIA_PROJECT to include it in the environment stack.

Further Reading

https://pkgdocs.julialang.org/v1/environments/#Shared-environments

https://docs.julialang.org/en/v1/manual/code-loading/#Environments-1

1 Like
3

Yes, that was my “shady” suggestion. Can I overwrite those files and then call the using Pkg; Pkg.instantiate(); from the commandline? Is that well-defined operation or a terrible idea? Normally you don’t mess around with those folders manually.

I don’t want to create new environments that require activation/etc. for the container if possible. At that point I am better just forcing a shared project file.

The main concern I have is that this makes it impossible to layer things effectively with docker. What if a downstream docker container wants to add packages/etc.? But maybe there is just no way around this issue.

4

Yes, I think that’s well defined, and a good engouh idea. At least that’s what I did in a similar situation, and I never encountered any issue with the produced Docker container.

The only downside is, as you noted, that only one layer can do this.

I think the other solution suggested by @mkitti above doesn’t require activation or anything complicated. IIUC, the suggestion would simply be along the lines of

# Dockerfile
COPY Project.toml  /home/user/.julia/environments/myenv/Project.toml
COPY Manifest.toml /home/user/.julia/environments/myenv/Manifest.toml
ENV JULIA_LOAD_PATH=":/home/user/.julia/environments/myenv"

One issue with this is that if someone down the line adds more environments to the stack, then each one will have been resolved in isolation, and they are not guaranteed to be compatible with eachother.


One last option would be to have a plain Julia script that uses the Pkg API to manipulate and resolve the environment:

# install_deps.jl
using Pkg
Pkg.add(["MyFirstDep", "MySecondDep"])

and

# Dockerfile
RUN julia --project=@v1.10 install_deps.jl

Several layers can then use the same method to add their own dependencies to the global environment, and dependencies resolution will always take all into account. The downside is that you only specify a list of packages (i.e. the contents of Project.toml), but don’t enforce specific versions for your dependencies (i.e. you lose the specific contents of Manifest.toml).

I’ve also used this technique in the past, and it worked well. But I switched to the first technique above because I only had one layer, and I wanted complete control over the environment.

1 Like