Shared and stacked environments

General Usage
, ,
1

A common case, you are working on your package, and need some additional packaged during the development only (for plotting, benchmarking, debugging etc.). You put them into your default environment, and soon the Manifest.toml file there lists dozens or hundreds of dependencies, with increasing chances of a conflict with the dependencies in your package.

I am thinking of a workflow where your default environment contains just a minimum of packages, maybe just one (see below). Then there are several shared environments for specific tasks - e.g. @Plots with the Plots.jl, @Data ( DataFrames.jl, CSV.jl, …), @Benchmarking, @Debug etc.

Then a package for a usage within REPL (and installed into the default environment), which would export the @using macro (any objections against the name?), doing roughly following:

  • If a package SomePackage is already in one of the environments in the current LOAD_PATH, then execute using SomePackage.
  • If a package AnotherOne is in a shared environment @AnotherEnv, then first push @AnotherEnv into the LOAD_PATH, then using AnotherOne.
  • If the package ToBeInstalled is not in any shared environment, ask user and install it into @ToBeInstalled environment.
  • For @using Pk1, Pk2, Pk3, find an optimal combination of environments, if some packages are present in multiple shared environments.

Does that all sound reasonably, of have I overlooked something important? Is that functionality or part of that already implemented in some package? (I am aware of similarly motivated Draft.jl .) Will it become obsolete after implementation of the incoming workspace feature of Pkg.jl?

I’m pinging some of those who already discussed relevant topics @lmiq @feanor12 @mkitti
Some former threads are listed below, but actually there were not many threads about it, suggesting these features are not commonly used:

Tentative package name ShareAdd or maybe SharAdd - is it ok?

In the following poll, the default Julia environment (i.e. v1.10) currently doesn’t count as shared and stacked.

  • I have no or only limited knowledge about shared and stacked environments
  • I am familiar with shared environments
  • I am familiar with stacked environments
  • I have used shared and/or stacked
  • I am using shared & stacked environments in combination
0 voters
1 Like
2

During the recent years, I found Julia temp environments and the autoinstall feature to be very nice.
Now, I only have few packages in the default env (stuff like Pluto and Revise), and for everything else I just type the package name:

julia> using DataManipulation
 │ Package DataManipulation not found, but a package named DataManipulation is available from a registry. 
 │ Install package?
 │   (@v1.10) pkg> add DataManipulation 
 └ Select environment:
 > 1: `/var/folders/2j/9vtd991d201dbkh9dnx3wvy00000gq/T/jl_tcW3N8` (/var/folders/2j/9vtd991d201dbkh9dnx3wvy00000gq/T/jl_tcW3N8)
   2: `~/.julia/environments/v1.10/Project.toml` (@v#.#)
   Resolving package versions...
<... Pkg output ...>

julia> # DataManipulation loaded

There are a few issues opened regarding making the autoinstall feature even more convenient (Autocomplete on `using` for uninstalled packages · Issue #51713 · JuliaLang/julia · GitHub, Autoinstall prompt should appear in more cases · Issue #53549 · JuliaLang/julia · GitHub) – but it’s already great!

Moreover, with BasicAutoloads.jl (and some setup) you get this for any package you want to use often:

julia> x = KeyedArray(rand(3, 4), x=[:a,:b,:c], y=1:4)
[ Info: Loading AxisKeys for KeyedArray ...
 │ Package AxisKeys not found, but a package named AxisKeys is available from a registry. 
 │ Install package?
 │   (@v1.10) pkg> add AxisKeys 
 └ Select environment:
 > 1: `/var/folders/2j/9vtd991d201dbkh9dnx3wvy00000gq/T/jl_NWdb44` (/var/folders/2j/9vtd991d201dbkh9dnx3wvy00000gq/T/jl_NWdb44)
   2: `~/.julia/environments/v1.10/Project.toml` (@v#.#)
   Resolving package versions...
<... Pkg output ...>

2-dimensional KeyedArray(NamedDimsArray(...)) with keys:
↓   x ∈ 3-element Vector{Symbol}
→   y ∈ 4-element UnitRange{Int64}
And data, 3×4 Matrix{Float64}:
        (1)          (2)          (3)         (4)
  (:a)    0.501477     0.460751     0.948424    0.841584
  (:b)    0.0307299    0.915012     0.150398    0.282693
  (:c)    0.3951       0.0445179    0.90074     0.452201

julia> # x is a KeyedArray now

Feels very nice and natural – just type what you want, install & import is done automatically.

Happy to hear if you are planning to further improve on this experience :slight_smile:

2 Likes
3

I like how Node.js has both ‘regular’ dependencies and then development dependencies. When you install a new package, you can simply include the --save-dev or -D flag. In the package.json file in a Node project, there’s a dependencies section and a devDependencies section. The packages in the devDependencies don’t get installed for someone cloning a repo and running npm install.

I wonder if it could be useful to have something similar in Julia. If you’re developing a package it may be nice to be able to add BenchmarkTools -D and then just have that package be listed as a development dependency…?

1 Like
4

For my just announced package ShareAdd.jl I am thinking of adding there another macro @usingtemp, with the usage like

@usingtemp Pk1, Pk2

which would

  1. If the current environment is a temporary one, then install package Pk1 and Pk2 there, and import them by using Pk1, Pk2 .
  2. If the current environment is a package under development, then switch to a temporary environment, dev there your package, goto 1.

However actually I do not see any advantage of using temporary environments over the workflow currently envisioned in ShareAdd.jl .

Also, for BasicAutoloads.jl - I didn’t try it, but I think it can be made to work with ShareAdd.jl like

 ["@benchmark"]           => :(@usingany BenchmarkTools)

What do you think?