How to stack multiple environments/projects for a reproducibility repository?

General Usage
,
1

Is it possible - and if yes, how - to stack multiple environments for a reproducibility repository? That is, I would like to have two sets of Project.toml/Manifest.toml files living outside the DEPOT_PATH where I am able to independently (and in a reproducible manner!) control the package versions used.

Here is my use case (hopefully this makes it clearer):
To a reproducibility repository, I have added an environment A with a Project.toml/Manifest.toml pair that specifies two packages that are used for my workload, OrdinaryDiffEq.jl and Trixi.jl, including their dependencies. I would like to add a second environment B that only contains PackageCompiler.jl that allows one to compile a certain workflow using the packages from environment A.

Now, I could add PackageCompiler.jl directly to environment A, but I do not want to do this, since it might pull in additional dependencies that most people do not care about. Thus I would like to mimic the “standard” Julia setup where I have an active project (in this case, A) but where I am also able to load also packages from the default environment (which is in the DEPOT_PATH). The difference here is that I would like to put the “default environment” also in the reproducibility repository and have the users instantiate and use this independently.

2

You can append the PackageCompiler project to the installation instructions/make target for creating the sysimage. See for example https://github.com/fredrikekre/.dotfiles/blob/master/.julia/environments/nvim-lspconfig/Makefile#L52-#L53.

3

Ah, do you mean one should be modifying the LOAD_PATH by setting JULIA_LOAD_PATH appropriately? E.g., with a directory structure as

myroot/
  project_A/
    Manifest.toml
    Project.toml
  project_B/
    Manifest.toml
    Project.toml`

I would first instantiate both projects individually by running

cd myroot
julia --project=project_A -e 'using Pkg; Pkg.instantiate()'
julia --project=project_B -e 'using Pkg; Pkg.instantiate()'

and then I could stack the environments by setting JULIA_LOAD_PATH, e.g., like

JULIA_LOAD_PATH="$(pwd)/project_A:$(pwd)/project_B" julia -e '...'

where project_A would now be the primary environment, and project_B the secondary. At least that’s what I pieced together from looking at Code Loading · The Julia Language and Constants · The Julia Language.

EDIT1: Simplified the JULIA_LOAD_PATH command.
EDIT2: Remove @stdlib as suggested by @fredrikekre
EDIT3: Accept this answer incorporating the help from @fredrikekre to make it easier for future solution seekers.

4

Yes.

Also, don’t include @stdlib there, add necessary stdlibs to the project environments instead.

1 Like
5

Thanks a lot for your support!

I included your suggestion in my example above to have everything in one piece. Thanks!

6

For the specific case of PackageCompiler you don’t need to stack them though. You can use the project keyword argument instead. In this case you can start Julia in the PackageCompiler project, and then pass the real project to create_sysimg.

2 Likes