It takes several minutes to add a single package

As bad. The problem is the download of the registries. I will try the Pkg.offline option (which, by the way, is not documented here, apparently).

The Pkg.offline options is what really improves my experience here:

julia> Pkg.offline(true)

julia> @time Pkg.add("PDBTools")
   Resolving package versions...
    Updating `/tmp/jl_iorxwT/Project.toml`
  [e29189f1] + PDBTools v0.12.11
    Updating `/tmp/jl_iorxwT/Manifest.toml`
  [59287772] + Formatting v0.4.2
  [bac558e1] + OrderedCollections v1.4.1
  [e29189f1] + PDBTools v0.12.11
  [d96e819e] + Parameters v0.12.3
  [3a884ed6] + UnPack v1.0.2
  [2a0f44e3] + Base64
  [b77e0a4c] + InteractiveUtils
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [de0858da] + Printf
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  [8dfed614] + Test
  [4ec0a83e] + Unicode
  2.467132 seconds (3.48 M allocations: 205.214 MiB, 2.91% gc time, 6.54% compilation time)

It becomes fast:

julia> import Pkg

julia> Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true
true

julia> @time Pkg.add("PDBTools")
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.6/Project.toml`
  No Changes to `~/.julia/environments/v1.6/Manifest.toml`
  5.910878 seconds (5.07 M allocations: 320.312 MiB, 1.37% gc time, 5.82% compilation time)

(@v1.6) pkg> activate --temp
  Activating new environment at `/tmp/jl_pMQIWm/Project.toml`

julia> @time Pkg.add("PDBTools")
   Resolving package versions...
    Updating `/tmp/jl_pMQIWm/Project.toml`
  [e29189f1] + PDBTools v0.12.11
    Updating `/tmp/jl_pMQIWm/Manifest.toml`
  [59287772] + Formatting v0.4.2
  [bac558e1] + OrderedCollections v1.4.1
  [e29189f1] + PDBTools v0.12.11
  [d96e819e] + Parameters v0.12.3
  [3a884ed6] + UnPack v1.0.2
  [2a0f44e3] + Base64
  [b77e0a4c] + InteractiveUtils
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [de0858da] + Printf
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  [8dfed614] + Test
  [4ec0a83e] + Unicode
  0.468187 seconds (807.31 k allocations: 67.419 MiB, 6.85% gc time, 52.59% compilation time)

julia> 

The problem is on fetching the registry there (downloading the Julia tarball took 3 seconds):

I feel like you are conflating local environments with temporary environments. I suggest using temporary ones when you want to do a quick test with a new package that you may not use anymore. If you want to have something persistent, you create a local environment.

So, I really don’t understand the point you’re trying to make about putting all packages in the global environment.

1 Like

Because once the package is in the global environment, I don’t need to install them again in the temporary environments to test things. That is why.

If I am developing let say package β€œA” in the environment of A, and I happen to need some other package there for testing something, I have to add it to that environment, or simply use it from the global environment.

As you can see, using it is β€œfree”. Adding it to the local environment (now I know that I can improve that by using Pkg.offline(true) or the updated registry thing) may take several minutes.

From that screenshot, it looks like you are using the Git version of the General registry. In order to get the registry speedups in Julia 1.7 that @GunnarFarneback was talking about, you will need to be using the Pkg server version of the General registry.

In order to switch to the Pkg server version of the General registry, run the following commands:

julia> import Pkg

julia> Pkg.Registry.rm("General")

julia> delete!(ENV, "JULIA_PKG_SERVER")

julia> Pkg.Registry.add("General")
6 Likes

Then create a local environment? As I said above, my point is to not clutter the global environment. Temporary environments are… well… temporary, I do not recommend them to put packages you know you want to use later on

1 Like

Perhaps I am misunderstanding β€œproper” use of local environments.

Let’s say I have 10 projects, that require an overlapping set of packages, say, each project has 3–5 packages it alone uses, 10–20 packages it shares with a couple other projects, and 5–10 packages that it shares with most projects. (Each of these direct requirements also has its own dependency tree.) When I initially create a package I don not know exactly what packages it will need, so I have to add them incrementally with ]add Xxx Yyy one or two at a time.

Are you recommending I put 5 packages in my global environment, create 10 local environments and install the remaining requisite 18–30 packages individually in each of the ten local environments?

Thanks! (Does this have side effects folks should be aware of that span Julia sessions?)

Let us say that I am developing my package PDBTools. Then, I am working in the environment PDBTools pkg>. I happen to need to draw a plot, but Plots is not a dependency of my package at all.

If Plots is installed in the global environment, I just need to do using Plots. If not, what do I do?

I am getting this error message, and it does not seem to be working (or is it?). In any case, it is taking a lot of time anyway:

That could be a good middle ground, yes.

I tend to be more extreme and have only dev packages (BenchmarkTools, Revise, little else) in the global environment and then for all other tasks I use local environments. Local environments are useful for reproducibility: if I put all packages I need for an application in its local environment I can check in the manifest with Git and give it to others, who can simply instantiate the environment and have everything ready to use. If you rely on packages installed in your own global environment that’s not going to work.

4 Likes

So the issue I see with that approach is that it takes some time to install a single package and when we multiply by 10*(18–30) we get 180–300 package installations. This is a lot of avoidable work for me because I only actually need to have a reproducible environment for two of those ten projects. With the global environment system I can install all 60 packages I have ever used in my global environment and then only go through manual installation in local environments 36–60 times for a total of 96–120 package installations rather than 180–300.

It’d be even nicer if the 36-60 manual instillations to create reproducible environments was automated, leaving me with only 60 in the global environment (3x–5x savings at the UI level)

I explain this alternative in detail above:

From the discussion above, an alternative workflow can be:

  1. Start a temporary environment.
  2. Add every possible package you may wan to use to that temporary environment (i. e. download everything).
  3. Add Pkg.offline(true) and perhaps Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true to startup.jl.

Than you can add packages quickly to new environments, temporary or not, all the time, and the main environment has no package whatsoever.

It still does not solve the fact that you cannot use those packages without installing them in each environment, but at least the interaction will be swift.

I’ve profiled, with the following results:
24% (~1s) resolve
23% (~1s) precompile
49% (~2s) handle interrupt (I did not trigger any keyboard interrupts during profiling)

Precompile shouldn’t be happening because there are no changes to the manifest, right?
And I have no idea what handle interrupt is (@Base/task.jl:411; (::Pkg.API.var"#212#239"{Bool, Vector{Task}, Pkg.API.var"#handle_interrupt#231"{Base.Event, ReentrantLo...).

In detail:

Overhead β•Ž [+additional indent] Count File:Line; Function
=========================================================
     β•Ž26790  @Base/client.jl:485; _start()
     β•Ž 26790  @Base/client.jl:302; exec_options(opts::Base.JLOptions)
     β•Ž  26790  @Base/client.jl:372; run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
     β•Ž   26790  @Base/essentials.jl:706; invokelatest
     β•Ž    26790  @Base/essentials.jl:708; #invokelatest#2
     β•Ž     26790  @Base/client.jl:387; (::Base.var"#874#876"{Bool, Bool, Bool})(REPL::Module)
     β•Ž    β•Ž 26790  ...ackage_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:305; run_repl(repl::REPL.AbstractREPL, consumer::Any)
     β•Ž    β•Ž  26790  ...ackage_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:317; run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool)
     β•Ž    β•Ž   26790  ...ackage_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:185; start_repl_backend(backend::REPL.REPLBackend, consumer::Any)
     β•Ž    β•Ž    26790  ...ckage_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:200; repl_backend_loop(backend::REPL.REPLBackend)
     β•Ž    β•Ž     26790  ...ckage_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:139; eval_user_input(ast::Any, backend::REPL.REPLBackend)
     β•Ž    β•Ž    β•Ž 26790  @Base/boot.jl:360; eval
     β•Ž    β•Ž    β•Ž  26790  REPL[37]:2; top-level scope
     β•Ž    β•Ž    β•Ž   26790  ...acos64/build/usr/share/julia/stdlib/v1.6/Profile/src/Profile.jl:28; macro expansion
     β•Ž    β•Ž    β•Ž    26789  @Base/timing.jl:210; macro expansion
     β•Ž    β•Ž    β•Ž     26789  ...ckage_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:74; add
     β•Ž    β•Ž    β•Ž    β•Ž 26789  ...ckage_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:74; #add#22
     β•Ž    β•Ž    β•Ž    β•Ž  26789  ...kage_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:75; add
     β•Ž    β•Ž    β•Ž    β•Ž   26789  ...kage_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:75; #add#23
     β•Ž    β•Ž    β•Ž    β•Ž    26789  ...kage_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:77; add(pkgs::Vector{Pkg.Types.PackageSpec})
     β•Ž    β•Ž    β•Ž    β•Ž     14423  ...age_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:79; add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Iterators.Pairs{Union{},...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž 14423  ...ge_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:154; (::Pkg.API.var"#add##kw")(::NamedTuple{(:io,), Tuple{Base.TTY}}, ::typeof(Pkg.API.add), ctx...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž  14416  ...ge_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:203; add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.Prese...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž   14416  ...64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1223; add##kw
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    12373  ...64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1231; add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Vector{Base.UU...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž     12366  ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1216; _resolve
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž 12366  ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1182; tiered_resolve(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž  11831  ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1210; targeted_resolve(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, preserve:...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž   9618   ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:404; resolve_versions!(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    4254   .../build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:534; deps_graph(ctx::Pkg.Types.Context, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{B...
     β•Ž    β•Ž    β•Ž    β•Ž     11944  ...age_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:80; add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Iterators.Pairs{Union{},...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž 11943  ...ge_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Pkg.jl:592; _auto_precompile(ctx::Pkg.Types.Context)
    1β•Ž    β•Ž    β•Ž    β•Ž    β•Ž  11943  ...ge_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:920; (::Pkg.API.var"#precompile##kw")(::NamedTuple{(:internal_call,), Tuple{Bool}}, ::typeof(Pk...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž   3737   ...e_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:921; precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, kwargs::Base.Iterat...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    3737   ...e_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1304; (::Pkg.API.var"#instantiate##kw")(::NamedTuple{(:allow_autoprecomp,), Tuple{Bool}}, ::ty...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž     3669   ..._macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1335; instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::B...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž 3651   ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:131; is_instantiated(ctx::Pkg.Types.Context)
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž  3651   @Base/reducedim.jl:886; all
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž   3651   @Base/reducedim.jl:886; #all#698
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    3651   @Base/reduce.jl:923; _all
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž     3651   .../build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:131; #13
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž 3589   ...build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1786; is_package_downloaded(ctx::Pkg.Types.Context, pkg::Pkg.Types.PackageSpec)
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž  3589   ...build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:713; check_artifacts_downloaded
    5β•Ž    β•Ž    β•Ž    β•Ž    β•Ž   4157   ...e_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:981; precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, kwargs::Base.Iterat...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    4143   ...e_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#in_deps#227")(pkgs::Vector{Base.PkgId}, deps::Vector{Base.PkgId}, dmap::D...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž     4143   @Base/reducedim.jl:883; any
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž 4143   @Base/reducedim.jl:883; #any#696
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž  4143   @Base/reduce.jl:876; _any(f::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgId}}}...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž   4143   ..._macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgId}}})(dep...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    4061   ...macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#in_deps#227")(pkgs::Vector{Base.PkgId}, deps::Vector{Base.PkgId}, dma...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž     4061   @Base/reducedim.jl:883; any
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž 4061   @Base/reducedim.jl:883; #any#696
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž  4057   @Base/reduce.jl:876; _any(f::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgId...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž   4057   ...acos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgId}}})(...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    3798   ...cos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#in_deps#227")(pkgs::Vector{Base.PkgId}, deps::Vector{Base.PkgId}, ...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž     3798   @Base/reducedim.jl:883; any
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž 3798   @Base/reducedim.jl:883; #any#696
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž  3789   @Base/reduce.jl:876; _any(f::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.Pk...
   12β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž   3789   ...os64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgId}}...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    3474   ...s64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#in_deps#227")(pkgs::Vector{Base.PkgId}, deps::Vector{Base.PkgId...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž     3474   @Base/reducedim.jl:883; any
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž 3474   @Base/reducedim.jl:883; #any#696
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž  3469   @Base/reduce.jl:876; _any(f::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base...
    9β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž   3469   ...64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.PkgI...
    1β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    3286   ...4/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#in_deps#227")(pkgs::Vector{Base.PkgId}, deps::Vector{Base.Pk...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž     3285   @Base/reducedim.jl:883; any
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž 3285   @Base/reducedim.jl:883; #any#696
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž  3280   @Base/reduce.jl:876; _any(f::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{B...
   11β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    β•Ž   3280   .../build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:978; (::Pkg.API.var"#204#228"{Vector{Base.PkgId}, Dict{Base.PkgId, Vector{Base.P...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž   3409   ...e_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1098; precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, kwargs::Base.Iterat...
     β•Ž    β•Ž    β•Ž    β•Ž    β•Ž    3138   @Base/loading.jl:290; locate_package(pkg::Base.PkgId)
   11β•Ž25689  @Base/task.jl:411; (::Pkg.API.var"#212#239"{Bool, Vector{Task}, Pkg.API.var"#handle_interrupt#231"{Base.Event, ReentrantLo...
     β•Ž 24550  ...er/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1126; macro expansion
    1β•Ž  24542  ...er/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:904; _is_stale
    6β•Ž   8281   @Base/loading.jl:1716; stale_cachefile(modpath::String, cachefile::String)
     β•Ž    6712   @Base/loading.jl:290; locate_package(pkg::Base.PkgId)
    4β•Ž     5416   @Base/loading.jl:396; manifest_uuid_path(env::String, pkg::Base.PkgId)
    3β•Ž   14504  @Base/loading.jl:1750; stale_cachefile(modpath::String, cachefile::String)
     β•Ž    13681  @Base/loading.jl:254; identify_package(where::Base.PkgId, name::String)
   65β•Ž     12903  @Base/loading.jl:379; manifest_deps_get(env::String, where::Base.PkgId, name::String)
   45β•Ž    β•Ž 3048   @Base/loading.jl:503; explicit_manifest_deps_get(project_file::String, where::Base.UUID, name::String)
     β•Ž    β•Ž  3003   @Base/uuid.jl:82; UUID

This should probably be a local environment so that you can keep it across Julia sessions.

That depends on the environment, I’ve experienced between .1 and 8 seconds depending on the environment and the time of day (all without using any internet connection).

I completely understand the importance and utility of this, but this is not how people use packages, nor never will be. No package user guide starts saying to the user to download a Manifest file and instantiate anything. Manuals say: install the package, start using it.

I think most people is coming in contact with environments not because of the features, but because they are a sort of annoying workaround for the fact that just using a lot of packages in the global environment starts to become problematic whenever a new package or update occurs.

I my humble opinion, making Julia deal well with a cluttered main environment would be a significant user experience improvement for the majority of users, and requires a more conservative approach to updates (update only if necessary, warn the user about the version available, etc.).

3 Likes

Does your cluster use glusterfs for storage? Also did you try Julia 1.7? A lot of work was put into speeding up package management on windows by not crawling directories. This also helps enormously for glusterfs and maybe also for other NFS based filesystems. I think you need to install 1.7 and rm the existing registry then add it back.

No, and yes (it seems to be something related to the access to git repositories, I don’t know). It is not my cluster. It is the cluster I have access to.

I’m talking about applications I want to share with other people, or maybe reproduce myself on other systems, or even on the same machine later in time. I don’t think I said that’s what should be written in packages user guides?

It’s a matter of fact packages do have compat bounds. The more packages you have in a single environment, the more constraints the resolver needs to satisfy, the more likely you’ll be to have some packages held back by some weird combinations. You’re happy with that?

1 Like

Yet another:

  1. Create an environment called everything or workspace or all
  2. Move all packages in global to all
  3. Activate all in startup.jl
  4. Work in all by default, adding new packages as you come across them
  5. When necessary, create a local environment for reproducibility.

This also benefits from manual configuration of ]add (e.g. not automatically updating everything)

1 Like

I actually don’t think these fundamental limitations are the issue here. In my use cases, I do not have problematic compat bounds in the packages I’m using, and the majority of my inconvenience comes from unnecessary pre-compilation, whatever makes ]add slow other than resolve, and the need to manually futz with environments.

Doing the ] reg rm General then ]reg add General should dramatically speed things in 1.7 it won’t use git at all. This only affects your account not other users of the cluster.

I’ve got my home dir on a glusterfs directory and for me the registry updates were taking tens of minutes to an hour. Installing 1.7 and doing that dropped it to tens of seconds