It takes several minutes to add a single package

[lmiq]

@giordano

How’d the time be different if you had run the same command in the global environment?

As bad. The problem is the download of the registries. I will try the Pkg.offline option (which, by the way, is not documented here, apparently).

The Pkg.offline options is what really improves my experience here:

julia> Pkg.offline(true)

julia> @time Pkg.add("PDBTools")
   Resolving package versions...
    Updating `/tmp/jl_iorxwT/Project.toml`
  [e29189f1] + PDBTools v0.12.11
    Updating `/tmp/jl_iorxwT/Manifest.toml`
  [59287772] + Formatting v0.4.2
  [bac558e1] + OrderedCollections v1.4.1
  [e29189f1] + PDBTools v0.12.11
  [d96e819e] + Parameters v0.12.3
  [3a884ed6] + UnPack v1.0.2
  [2a0f44e3] + Base64
  [b77e0a4c] + InteractiveUtils
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [de0858da] + Printf
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  [8dfed614] + Test
  [4ec0a83e] + Unicode
  2.467132 seconds (3.48 M allocations: 205.214 MiB, 2.91% gc time, 6.54% compilation time)

@dilumaluthge

What if you do this at the beginning of your Julia session?

It becomes fast:

julia> import Pkg

julia> Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true
true

julia> @time Pkg.add("PDBTools")
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.6/Project.toml`
  No Changes to `~/.julia/environments/v1.6/Manifest.toml`
  5.910878 seconds (5.07 M allocations: 320.312 MiB, 1.37% gc time, 5.82% compilation time)

(@v1.6) pkg> activate --temp
  Activating new environment at `/tmp/jl_pMQIWm/Project.toml`

julia> @time Pkg.add("PDBTools")
   Resolving package versions...
    Updating `/tmp/jl_pMQIWm/Project.toml`
  [e29189f1] + PDBTools v0.12.11
    Updating `/tmp/jl_pMQIWm/Manifest.toml`
  [59287772] + Formatting v0.4.2
  [bac558e1] + OrderedCollections v1.4.1
  [e29189f1] + PDBTools v0.12.11
  [d96e819e] + Parameters v0.12.3
  [3a884ed6] + UnPack v1.0.2
  [2a0f44e3] + Base64
  [b77e0a4c] + InteractiveUtils
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [de0858da] + Printf
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  [8dfed614] + Test
  [4ec0a83e] + Unicode
  0.468187 seconds (807.31 k allocations: 67.419 MiB, 6.85% gc time, 52.59% compilation time)

julia> 

@GunnarFarneback

Julia 1.7 fares when it’s not unpacking the registry.

The problem is on fetching the registry there (downloading the Julia tarball took 3 seconds):

[giordano]

@Lilith

The essential difference is that you only have to add a package once in the global environment, whereas you need to add it in each new temporary environment you create that needs it.

I feel like you are conflating local environments with temporary environments. I suggest using temporary ones when you want to do a quick test with a new package that you may not use anymore. If you want to have something persistent, you create a local environment.

@lmiq

As bad.

So, I really don’t understand the point you’re trying to make about putting all packages in the global environment.

[lmiq]

@giordano

So, I really don’t understand the point you’re trying to make about putting all packages in the global environment.

Because once the package is in the global environment, I don’t need to install them again in the temporary environments to test things. That is why.

If I am developing let say package “A” in the environment of A, and I happen to need some other package there for testing something, I have to add it to that environment, or simply use it from the global environment.

As you can see, using it is “free”. Adding it to the local environment (now I know that I can improve that by using Pkg.offline(true) or the updated registry thing) may take several minutes.

[dilumaluthge]

@lmiq

The problem is on fetching the registry there (downloading the Julia tarball took 3 seconds):

From that screenshot, it looks like you are using the Git version of the General registry. In order to get the registry speedups in Julia 1.7 that @GunnarFarneback was talking about, you will need to be using the Pkg server version of the General registry.

In order to switch to the Pkg server version of the General registry, run the following commands:

julia> import Pkg

julia> Pkg.Registry.rm("General")

julia> delete!(ENV, "JULIA_PKG_SERVER")

julia> Pkg.Registry.add("General")

[giordano]

@lmiq

Because once the package is in the global environment, I don’t need to install them again in the temporary environments to test things. That is why.

Then create a local environment? As I said above, my point is to not clutter the global environment. Temporary environments are… well… temporary, I do not recommend them to put packages you know you want to use later on

[Lilith]

@giordano

I feel like you are conflating local environments with temporary environments. I suggest using temporary ones when you want to do a quick test with a new package that you may not use anymore. If you want to have something persistent, you create a local environment.

Perhaps I am misunderstanding “proper” use of local environments.

@giordano

Then create a local environment?

Let’s say I have 10 projects, that require an overlapping set of packages, say, each project has 3–5 packages it alone uses, 10–20 packages it shares with a couple other projects, and 5–10 packages that it shares with most projects. (Each of these direct requirements also has its own dependency tree.) When I initially create a package I don not know exactly what packages it will need, so I have to add them incrementally with ]add Xxx Yyy one or two at a time.

Are you recommending I put 5 packages in my global environment, create 10 local environments and install the remaining requisite 18–30 packages individually in each of the ten local environments?

[Lilith]

@dilumaluthge

julia> import Pkg

julia> Pkg.Registry.rm("General")

julia> delete!(ENV, "JULIA_PKG_SERVER")

julia> Pkg.Registry.add("General")

Thanks! (Does this have side effects folks should be aware of that span Julia sessions?)

[lmiq]

@giordano

Then create a local environment? As I said above, my point is to not clutter the global environment. Temporary environments are… well… temporary, I do not recommend them to put packages you know you want to use later on

Let us say that I am developing my package PDBTools. Then, I am working in the environment PDBTools pkg>. I happen to need to draw a plot, but Plots is not a dependency of my package at all.

If Plots is installed in the global environment, I just need to do using Plots. If not, what do I do?

@dilumaluthge

In order to switch to the Pkg server version of the General registry, run the following commands:

I am getting this error message, and it does not seem to be working (or is it?). In any case, it is taking a lot of time anyway:

[giordano]

@Lilith

Are you recommending I put 5 packages in my global environment, create 10 local environments and install the remaining requisite 18–30 packages individually in each of the ten local environments?

That could be a good middle ground, yes.

I tend to be more extreme and have only dev packages (BenchmarkTools, Revise, little else) in the global environment and then for all other tasks I use local environments. Local environments are useful for reproducibility: if I put all packages I need for an application in its local environment I can check in the manifest with Git and give it to others, who can simply instantiate the environment and have everything ready to use. If you rely on packages installed in your own global environment that’s not going to work.

[Lilith]

@giordano

@Lilith

Are you recommending I put 5 packages in my global environment, create 10 local environments and install the remaining requisite 18–30 packages individually in each of the ten local environments?

That could be a good middle ground, yes.

So the issue I see with that approach is that it takes some time to install a single package and when we multiply by 10*(18–30) we get 180–300 package installations. This is a lot of avoidable work for me because I only actually need to have a reproducible environment for two of those ten projects. With the global environment system I can install all 60 packages I have ever used in my global environment and then only go through manual installation in local environments 36–60 times for a total of 96–120 package installations rather than 180–300.

It’d be even nicer if the 36-60 manual instillations to create reproducible environments was automated, leaving me with only 60 in the global environment (3x–5x savings at the UI level)

I explain this alternative in detail above:

It takes several minutes to add a single package

Here’s a potential workflow: Remove “@v#.#” from Base.LOAD_PATH in startup.jl Typically always work in the global environment and add packages with ]add --preserve=all package. Update with ]up when I run into a package bug, want a new feature, or have a few minutes where it is okay to let Julia be unresponsive and am okay with a risk of having to deal with broken updates. When I want to share a minimum working environment, including whenever I do package development, make a new temporary or p…

[lmiq]

@Lilith

I explain this alternative in detail above:

From the discussion above, an alternative workflow can be:

1. Start a temporary environment.
2. Add every possible package you may wan to use to that temporary environment (i. e. download everything).
3. Add Pkg.offline(true) and perhaps Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true to startup.jl.

Than you can add packages quickly to new environments, temporary or not, all the time, and the main environment has no package whatsoever.

It still does not solve the fact that you cannot use those packages without installing them in each environment, but at least the interaction will be swift.

[Lilith]

@giordano

@Lilith

FWIW, here is a benchmark of adding packages available locally. Takes a 3.4s in my cluttered global environment and 0.13s in a fresh --temp environment.

Yes, the difference is between a cluttered environment (where the resolver must keep happy more packages together) and a non-cluttered one.

I’ve profiled, with the following results:
24% (~1s) resolve
23% (~1s) precompile
49% (~2s) handle interrupt (I did not trigger any keyboard interrupts during profiling)

Precompile shouldn’t be happening because there are no changes to the manifest, right?
And I have no idea what handle interrupt is (@Base/task.jl:411; (::Pkg.API.var"#212#239"{Bool, Vector{Task}, Pkg.API.var"#handle_interrupt#231"{Base.Event, ReentrantLo...).

In detail:

Overhead ╎ [+additional indent] Count File:Line; Function
=========================================================
     ╎26790  @Base/client.jl:485; _start()
     ╎ 26790  @Base/client.jl:302; exec_options(opts::Base.JLOptions)
     ╎  26790  @Base/client.jl:372; run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
     ╎   26790  @Base/essentials.jl:706; invokelatest
     ╎    26790  @Base/essentials.jl:708; #invokelatest#2
     ╎     26790  @Base/client.jl:387; (::Base.var"#874#876"{Bool, Bool, Bool})(REPL::Module)
     ╎    ╎ 26790  ...ackage_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:305; run_repl(repl::REPL.AbstractREPL, consumer::Any)
     ╎    ╎  26790  ...ackage_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:317; run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool)
     ╎    ╎   26790  ...ackage_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:185; start_repl_backend(backend::REPL.REPLBackend, consumer::Any)
     ╎    ╎    26790  ...ckage_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:200; repl_backend_loop(backend::REPL.REPLBackend)
     ╎    ╎     26790  ...ckage_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:139; eval_user_input(ast::Any, backend::REPL.REPLBackend)
     ╎    ╎    ╎ 26790  @Base/boot.jl:360; eval
     ╎    ╎    ╎  26790  REPL[37]:2; top-level scope
     ╎    ╎    ╎   26790  ...acos64/build/usr/share/julia/stdlib/v1.6/Profile/src/Profile.jl:28; macro expansion
     ╎    ╎    ╎    26789  @Base/timing.jl:210; macro expansion
     ╎    ╎    ╎     26789  ...ckage_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:74; add
     ╎    ╎    ╎    ╎ 26789  ...ckage_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:74; #add#22
     ╎    ╎    ╎    ╎  26789  ...kage_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:75; add
     ╎    ╎    ╎    ╎   26789  ...kage_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:75; #add#23
     ╎    ╎    ╎    ╎    26789  ...kage_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:77; add(pkgs::Vector{Pkg.Types.PackageSpec})
     ╎    ╎    ╎    ╎     14423  ...age_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:79; add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Iterators.Pairs{Union{},...
     ╎    ╎    ╎    ╎    ╎ 14423  ...ge_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:154; (::Pkg.API.var"#add##kw")(::NamedTuple{(:io,), Tuple{Base.TTY}}, ::typeof(Pkg.API.add), ctx...
     ╎    ╎    ╎    ╎    ╎  14416  ...ge_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:203; add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.Prese...
     ╎    ╎    ╎    ╎    ╎   14416  ...64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1223; add##kw
     ╎    ╎    ╎    ╎    ╎    12373  ...64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1231; add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Vector{Base.UU...
     ╎    ╎    ╎    ╎    ╎     12366  ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1216; _resolve
     ╎    ╎    ╎    ╎    ╎    ╎ 12366  ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1182; tiered_resolve(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
     ╎    ╎    ╎    ╎    ╎    ╎  11831  ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1210; targeted_resolve(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, preserve:...
     ╎    ╎    ╎    ╎    ╎    ╎   9618   ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:404; resolve_versions!(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
     ╎    ╎    ╎    ╎    ╎    ╎    4254   .../build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:534; deps_graph(ctx::Pkg.Types.Context, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{B...
     ╎    ╎    ╎    ╎     11944  ...age_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:80; add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Iterators.Pairs{Union{},...
     ╎    ╎    ╎    ╎    ╎ 11943  ...ge_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Pkg.jl:592; _auto_precompile(ctx::Pkg.Types.Context)
    1╎    ╎    ╎    ╎    ╎  11943  ...ge_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:920; (::Pkg.API.var"#precompile##kw")(::NamedTuple{(:internal_call,), Tuple{Bool}}, ::typeof(Pk...
     ╎    ╎    ╎    ╎    ╎   3737   ...e_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:921; precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, kwargs::Base.Iterat...
     ╎    ╎    ╎    ╎    ╎    3737   ...e_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1304; (::Pkg.API.var"#instantiate##kw")(::NamedTuple{(:allow_autoprecomp,), Tuple{Bool}}, ::ty...
     ╎    ╎    ╎    ╎    ╎     3669   ..._macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1335; instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::B...
     ╎    ╎    ╎    ╎    ╎    ╎ 3651   ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:131; is_instantiated(ctx::Pkg.Types.Context)
     ╎    ╎    ╎    ╎    ╎    ╎  3651   @Base/reducedim.jl:886; all
     ╎    ╎    ╎    ╎    ╎    ╎   3651   @Base/reducedim.jl:886; #all#698
     ╎    ╎    ╎    ╎    ╎    ╎    3651   @Base/reduce.jl:923; _all
     ╎    ╎    ╎    ╎    ╎    ╎     3651   .../build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:131; #13
     ╎    ╎    ╎    ╎    ╎    ╎    ╎ 3589   ...build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1786; is_package_downloaded(ctx::Pkg.Types.Context, pkg::Pkg.Types.PackageSpec)
     ╎    ╎    ╎    ╎    ╎    ╎    ╎  3589   ...build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:713; check_artifacts_downloaded
    5╎    ╎    ╎    ╎    ╎   4157   ...e_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:981; precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, kwargs::Base.Iterat...
     ╎    ╎    ╎    ╎    ╎    4143   ...e_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#in_deps#227")(pkgs::Vector{Base.PkgId}, deps::Vector{Base.PkgId}, dmap::D...
     ╎    ╎    ╎    ╎    ╎     4143   @Base/reducedim.jl:883; any
     ╎    ╎    ╎    ╎    ╎    ╎ 4143   @Base/reducedim.jl:883; #any#696
     ╎    ╎    ╎    ╎    ╎    ╎  4143   @Base/reduce.jl:876; _any(f::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgId}}}...
     ╎    ╎    ╎    ╎    ╎    ╎   4143   ..._macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgId}}})(dep...
     ╎    ╎    ╎    ╎    ╎    ╎    4061   ...macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#in_deps#227")(pkgs::Vector{Base.PkgId}, deps::Vector{Base.PkgId}, dma...
     ╎    ╎    ╎    ╎    ╎    ╎     4061   @Base/reducedim.jl:883; any
     ╎    ╎    ╎    ╎    ╎    ╎    ╎ 4061   @Base/reducedim.jl:883; #any#696
     ╎    ╎    ╎    ╎    ╎    ╎    ╎  4057   @Base/reduce.jl:876; _any(f::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgId...
     ╎    ╎    ╎    ╎    ╎    ╎    ╎   4057   ...acos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgId}}})(...
     ╎    ╎    ╎    ╎    ╎    ╎    ╎    3798   ...cos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#in_deps#227")(pkgs::Vector{Base.PkgId}, deps::Vector{Base.PkgId}, ...
     ╎    ╎    ╎    ╎    ╎    ╎    ╎     3798   @Base/reducedim.jl:883; any
     ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎ 3798   @Base/reducedim.jl:883; #any#696
     ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎  3789   @Base/reduce.jl:876; _any(f::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.Pk...
   12╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎   3789   ...os64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgId}}...
     ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    3474   ...s64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#in_deps#227")(pkgs::Vector{Base.PkgId}, deps::Vector{Base.PkgId...
     ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎     3474   @Base/reducedim.jl:883; any
     ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎ 3474   @Base/reducedim.jl:883; #any#696
     ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎  3469   @Base/reduce.jl:876; _any(f::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base...
    9╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎   3469   ...64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgI...
    1╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    3286   ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#in_deps#227")(pkgs::Vector{Base.PkgId}, deps::Vector{Base.Pk...
     ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎     3285   @Base/reducedim.jl:883; any
     ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎ 3285   @Base/reducedim.jl:883; #any#696
     ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎  3280   @Base/reduce.jl:876; _any(f::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{B...
   11╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎    ╎   3280   .../build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.P...
     ╎    ╎    ╎    ╎    ╎   3409   ...e_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1098; precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, kwargs::Base.Iterat...
     ╎    ╎    ╎    ╎    ╎    3138   @Base/loading.jl:290; locate_package(pkg::Base.PkgId)
   11╎25689  @Base/task.jl:411; (::Pkg.API.var"#212#239"{Bool, Vector{Task}, Pkg.API.var"#handle_interrupt#231"{Base.Event, ReentrantLo...
     ╎ 24550  ...er/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1126; macro expansion
    1╎  24542  ...er/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:904; _is_stale
    6╎   8281   @Base/loading.jl:1716; stale_cachefile(modpath::String, cachefile::String)
     ╎    6712   @Base/loading.jl:290; locate_package(pkg::Base.PkgId)
    4╎     5416   @Base/loading.jl:396; manifest_uuid_path(env::String, pkg::Base.PkgId)
    3╎   14504  @Base/loading.jl:1750; stale_cachefile(modpath::String, cachefile::String)
     ╎    13681  @Base/loading.jl:254; identify_package(where::Base.PkgId, name::String)
   65╎     12903  @Base/loading.jl:379; manifest_deps_get(env::String, where::Base.PkgId, name::String)
   45╎    ╎ 3048   @Base/loading.jl:503; explicit_manifest_deps_get(project_file::String, where::Base.UUID, name::String)
     ╎    ╎  3003   @Base/uuid.jl:82; UUID

[Lilith]

@lmiq

Start a temporary environment.

This should probably be a local environment so that you can keep it across Julia sessions.

@lmiq

Than you can add packages quickly to new environments

That depends on the environment, I’ve experienced between .1 and 8 seconds depending on the environment and the time of day (all without using any internet connection).

[lmiq]

@giordano

if I put all packages I need for an application in its local environment I can check in the manifest with Git and give it to others, who can simply instantiate the environment and have everything ready to use

I completely understand the importance and utility of this, but this is not how people use packages, nor never will be. No package user guide starts saying to the user to download a Manifest file and instantiate anything. Manuals say: install the package, start using it.

I think most people is coming in contact with environments not because of the features, but because they are a sort of annoying workaround for the fact that just using a lot of packages in the global environment starts to become problematic whenever a new package or update occurs.

I my humble opinion, making Julia deal well with a cluttered main environment would be a significant user experience improvement for the majority of users, and requires a more conservative approach to updates (update only if necessary, warn the user about the version available, etc.).

[dlakelan]

@lmiq

If what I experience in the cluster was my everyday experience, I would never think of using Julia for anything.

Does your cluster use glusterfs for storage? Also did you try Julia 1.7? A lot of work was put into speeding up package management on windows by not crawling directories. This also helps enormously for glusterfs and maybe also for other NFS based filesystems. I think you need to install 1.7 and rm the existing registry then add it back.

[lmiq]

@dlakelan

Does your cluster use glusterfs for storage? Also did you try Julia 1.7?

No, and yes (it seems to be something related to the access to git repositories, I don’t know). It is not my cluster. It is the cluster I have access to.

[giordano]

@lmiq

I completely understand the importance and utility of this, but this is not how people use packages, nor never will be. No package user guide starts saying to the user to download a Manifest file and instantiate anything. Manuals say: install the package, start using it.

I’m talking about applications I want to share with other people, or maybe reproduce myself on other systems, or even on the same machine later in time. I don’t think I said that’s what should be written in packages user guides?

@lmiq

I my humble opinion, making Julia deal well with a cluttered main environment would be a significant user experience improvement for the majority of users, and requires a more conservative approach to updates (update only if necessary, warn the user about the version available, etc.).

It’s a matter of fact packages do have compat bounds. The more packages you have in a single environment, the more constraints the resolver needs to satisfy, the more likely you’ll be to have some packages held back by some weird combinations. You’re happy with that?

[Lilith]

@lmiq

From the discussion above, an alternative workflow can be…

@Lilith

Here’s a potential workflow…

Yet another:

1. Create an environment called everything or workspace or all
2. Move all packages in global to all
3. Activate all in startup.jl
4. Work in all by default, adding new packages as you come across them
5. When necessary, create a local environment for reproducibility.

This also benefits from manual configuration of ]add (e.g. not automatically updating everything)

[Lilith]

@giordano

The more packages you have in a single environment, the more constraints the resolver needs to satisfy, the more likely you’ll be to have some packages held back by some weird combinations. You’re happy with that?

I actually don’t think these fundamental limitations are the issue here. In my use cases, I do not have problematic compat bounds in the packages I’m using, and the majority of my inconvenience comes from unnecessary pre-compilation, whatever makes ]add slow other than resolve, and the need to manually futz with environments.

[dlakelan]

@lmiq

No, and yes (it seems to be something related to the access to git repositories, I don’t know).

Doing the ] reg rm General then ]reg add General should dramatically speed things in 1.7 it won’t use git at all. This only affects your account not other users of the cluster.

I’ve got my home dir on a glusterfs directory and for me the registry updates were taking tens of minutes to an hour. Installing 1.7 and doing that dropped it to tens of seconds