First Pluto notebook launches are slower on Julia 1.9 beta 3

Internals & Design
, ,
158

That means that if I add a package to a large project, it will often install new low-level dependencies and trigger a lot of precompilation, despite UPDATED_REGISTRY_THIS_SESSION .

I think part of the problem here is this:

pkg> add defaults to preserve=PRESERVE_TIERED, which is a sensible strategy IMO that tries all of the above strategies from the top down until the package can be added.

| Value             | Description                                                                         |
|:------------------|:------------------------------------------------------------------------------------|
| `PRESERVE_ALL`    | Preserve the state of all existing dependencies (including recursive dependencies)  |
| `PRESERVE_DIRECT` | Preserve the state of all existing direct dependencies                              |
| `PRESERVE_SEMVER` | Preserve semver-compatible versions of direct dependencies                          |
| `PRESERVE_NONE`   | Do not attempt to preserve any version information                                  |
| `PRESERVE_TIERED` | Use the tier which will preserve the most version information (this is the default) |

That should usually work well, but one exception is that Pkg currently does not know how to handle build numbers on packages, and will always update if a new build is available.

JLL’s use build numbers, and so can punch through this preserve behavior causing updates that are usually quite low in the dep tree.

Take here where the registry is checked out before and after a new jll build is available.

ImageMagick_jll v6.9.11+3 ⇒ v6.9.11+4 happens even though it’s completely unrelated to Example.

shell> cd ~/.julia/registries/General
/home/ian/.julia/registries/General

shell> git checkout ae8ec3b695efb04ddec4371b97477779b6c62549
HEAD is now at ae8ec3b695 New version: OpenAPI v0.1.7 (#77093)

(@v1.10) pkg> activate --temp
  Activating new project at `/tmp/jl_bgg4u9`

(jl_bgg4u9) pkg> add ImageMagick_jll@6.9.11
    Updating registry at `~/.julia/registries/General`
┌ Error: Some registries failed to update:
│     — `~/.julia/registries/General` — registry detached
└ @ Pkg.Registry ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.10/Pkg/src/Registry/Registry.jl:476
   Resolving package versions...
   Installed ImageMagick_jll ─ v6.9.11+3
  Downloaded artifact: ImageMagick
    Updating `/tmp/jl_bgg4u9/Project.toml`
⌃ [c73af94c] + ImageMagick_jll v6.9.11+3
    Updating `/tmp/jl_bgg4u9/Manifest.toml`
  [692b3bcd] + JLLWrappers v1.4.1
  [21216c6a] + Preferences v1.3.0
  [61579ee1] + Ghostscript_jll v9.55.0+3
⌃ [c73af94c] + ImageMagick_jll v6.9.11+3
  [aacddb02] + JpegTurbo_jll v2.1.2+0
  [88015f11] + LERC_jll v3.0.0+1
  [89763e89] + Libtiff_jll v4.4.0+0
  [d3a379c0] + LittleCMS_jll v2.12.0+0
  [643b3616] + OpenJpeg_jll v2.4.0+0
  [3161d3a3] + Zstd_jll v1.5.2+0
  [b53b4c65] + libpng_jll v1.6.38+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.10.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.10.11
  [83775a58] + Zlib_jll v1.2.13+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
        Info Packages marked with ⌃ have new versions available and may be upgradable.
Precompiling environment...
  1 dependency successfully precompiled in 1 seconds. 13 already precompiled.

(jl_bgg4u9) pkg> add Example
   Resolving package versions...
    Updating `/tmp/jl_bgg4u9/Project.toml`
  [7876af07] + Example v0.5.3
    Updating `/tmp/jl_bgg4u9/Manifest.toml`
  [7876af07] + Example v0.5.3

(jl_bgg4u9) pkg> rm Example
    Updating `/tmp/jl_bgg4u9/Project.toml`
  [7876af07] - Example v0.5.3
    Updating `/tmp/jl_bgg4u9/Manifest.toml`
  [7876af07] - Example v0.5.3

shell> git checkout 3dd229c080a140cf67146ceb3ed6dba8add22ddf
Previous HEAD position was ae8ec3b695 New version: OpenAPI v0.1.7 (#77093)
HEAD is now at 3dd229c080 New version: ImageMagick_jll v6.9.11+4 (#77094)

(jl_bgg4u9) pkg> add Example
   Resolving package versions...
   Installed ImageMagick_jll ─ v6.9.11+4
  Downloaded artifact: ImageMagick
    Updating `/tmp/jl_bgg4u9/Project.toml`
  [7876af07] + Example v0.5.3
⌃ [c73af94c] ↑ ImageMagick_jll v6.9.11+3 ⇒ v6.9.11+4
    Updating `/tmp/jl_bgg4u9/Manifest.toml`
  [7876af07] + Example v0.5.3
⌃ [c73af94c] ↑ ImageMagick_jll v6.9.11+3 ⇒ v6.9.11+4
        Info Packages marked with ⌃ have new versions available and may be upgradable.
Precompiling environment...
  1 dependency successfully precompiled in 1 seconds. 14 already precompiled.

Issue tracked at Feature request: support version numbers with build metadata · Issue #1568 · JuliaLang/Pkg.jl · GitHub

So I think a reasonable strategy here is:

  • Make pkg> add no longer update the registry automatically
  • Make Pkg aware of build numbers, so it can know not to update them appropriately
  • Add a pkg> update --noreg mode to do an env update without updating the registry
21 Likes