Although I think the goal of making manual environment management relatively easy is a good idea, Julia is a power-user tool and power users will generally want to understand what the tool does. I absolutely agree with more information about how to use environments @mbaz
It didn’t use to do this in old versions of Julia and there was a lot of confusion about people installing packages and getting old versions because they hadn’t manually updated their registry for a significant time.
This is not really true, i.e.:
(jl_LnOYYu) pkg> add Plots@1.22
Resolving package versions...
Installed Plots ─ v1.22.7
Updating `/tmp/jl_LnOYYu/Project.toml`
[91a5bcdd] ↓ Plots v1.23.3 ⇒ v1.22.7
Updating `/tmp/jl_LnOYYu/Manifest.toml`
[28b8d3ca] ↓ GR v0.62.0 ⇒ v0.61.0
[91a5bcdd] ↓ Plots v1.23.3 ⇒ v1.22.7
Precompiling project...
1 dependency successfully precompiled in 23 seconds (120 already precompiled)
(jl_LnOYYu) pkg> add Crayons
Resolving package versions...
Updating `/tmp/jl_LnOYYu/Project.toml`
[a8cc5b0e] + Crayons v4.0.4
Updating `/tmp/jl_LnOYYu/Manifest.toml`
[a8cc5b0e] + Crayons v4.0.4
Here I intentionally added an old version of Plots. Upon adding Crayons, you can see that Plots did not update.
2 Likes
Isn’t that just because you did both operations in the same Julia section? One in which the registry is already updated?
I’m talking about the statement that all packages get upgraded.
1 Like
I would think that this is mostly a matter of communication: a two step update then install / upgrade process is used in other places as well. And of course, one could think of graytones, i.e. Julia could automatically update the registry “every once in a while”. Compare this to how ] gc currently works.
2 Likes
Doesn’t seem like it, here I added Plots@1.22 and then restarted julia. Adding Crayons didn’t update anything else, but doing up after did update Plots (among some other packages).
(@v1.7) pkg> add Crayons
Updating registry at `~/.julia/registries/General.toml`
Resolving package versions...
Updating `~/.julia/environments/v1.7/Project.toml`
[a8cc5b0e] + Crayons v4.0.4
No Changes to `~/.julia/environments/v1.7/Manifest.toml`
Precompiling project...
Progress [> ] 0/1
◐ Plots
^C Interrupted: Exiting precompilation...
(@v1.7) pkg> st
Status `~/.julia/environments/v1.7/Project.toml`
[6e4b80f9] BenchmarkTools v1.2.0
[a8cc5b0e] Crayons v4.0.4
[31c24e10] Distributions v0.25.23
[587475ba] Flux v0.12.8
[e9467ef8] GLMakie v0.4.7
[7073ff75] IJulia v1.23.2
[5fb14364] OhMyREPL v0.5.10
[9b87118b] PackageCompiler v1.7.6
[14b8a8f1] PkgTemplates v0.7.23
[91a5bcdd] Plots v1.22.7
[c3e4b0f8] Pluto v0.17.0
[295af30f] Revise v3.1.20
(@v1.7) pkg> up
Updating registry at `~/.julia/registries/General.toml`
Installed PackageCompiler ─ v2.0.0
Installed Pluto ─────────── v0.17.1
Updating `~/.julia/environments/v1.7/Project.toml`
[9b87118b] ↑ PackageCompiler v1.7.6 ⇒ v2.0.0
[91a5bcdd] ↑ Plots v1.22.7 ⇒ v1.23.3
[c3e4b0f8] ↑ Pluto v0.17.0 ⇒ v0.17.1
Updating `~/.julia/environments/v1.7/Manifest.toml`
[28b8d3ca] ↑ GR v0.61.0 ⇒ v0.62.0
[9b87118b] ↑ PackageCompiler v1.7.6 ⇒ v2.0.0
[91a5bcdd] ↑ Plots v1.22.7 ⇒ v1.23.3
[c3e4b0f8] ↑ Pluto v0.17.0 ⇒ v0.17.1
Precompiling project...
2 dependencies successfully precompiled in 10 seconds (252 already precompiled)
(@v1.7) pkg>
1 Like
Thanks. Well, that makes thing more confusing. Am I imagining stuff? Or there are other situation in which adding a package may trigger the update of other (non-compat related) packages? Some situation related to the expiration of a registry, or something?
The intention is at least to try to add the package while keeping everything fixed and only if that fail allow other packages to also upgrade. If it doesn’t work like that, it is a bug. More explicitly, add --preserve=all Foo should never work if a add Foo has to upgrade other packages.
2 Likes
all other packages, or some subset of packages with a dependency tree taken into consideration?
All direct dependencies, if that fails, also indirect dependencies.
I think this is where things get confusing. For example, I have added Plots@1.22.0, then added an old version of a package of mine (which does not have Plots as a dependency). Then I updated that package, and that triggered the update of Plots:
(uptest) pkg> add ComplexMixtures@0.5.4
Resolving package versions...
Installed Plots ─ v1.23.4
Installed GR ──── v0.62.1
Updating `~/Drive/Work/JuliaPlay/uptest/Project.toml`
[6f35c628] ↑ ComplexMixtures v0.4.8 ⇒ v0.5.4
[e29189f1] ↑ PDBTools v0.11.4 ⇒ v0.12.11
[91a5bcdd] ↑ Plots v1.22.7 ⇒ v1.23.4
Updating `~/Drive/Work/JuliaPlay/uptest/Manifest.toml`
[6f35c628] ↑ ComplexMixtures v0.4.8 ⇒ v0.5.4
[28b8d3ca] ↑ GR v0.61.0 ⇒ v0.62.1
[e29189f1] ↑ PDBTools v0.11.4 ⇒ v0.12.11
[91a5bcdd] ↑ Plots v1.22.7 ⇒ v1.23.4
[860ef19b] + StableRNGs v1.0.0
[1cfade01] + UnicodeFun v0.4.1
Building GR → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/30f2b340c2fff8410d89bfcdc9c0a6dd661ac5f7/build.log`
Precompiling project...
2 dependencies successfully precompiled in 31 seconds (138 already precompiled)
(uptest) pkg>
But in a fresh environment the same didn’t do it:
(uptest2) pkg> add ComplexMixtures@0.5.4
Resolving package versions...
Updating `~/Drive/Work/JuliaPlay/uptest2/Project.toml`
[6f35c628] ↑ ComplexMixtures v0.4.8 ⇒ v0.5.4
Updating `~/Drive/Work/JuliaPlay/uptest2/Manifest.toml`
[6f35c628] ↑ ComplexMixtures v0.4.8 ⇒ v0.5.4
[e29189f1] ↑ PDBTools v0.11.4 ⇒ v0.12.11
[860ef19b] + StableRNGs v1.0.0
Not for me:
(jl_zMmory) pkg> st
Status `/tmp/jl_zMmory/Project.toml`
[91a5bcdd] Plots v1.22.0
(jl_zMmory) pkg> add ComplexMixtures@0.5.4
Resolving package versions...
Updating `/tmp/jl_zMmory/Project.toml`
[6f35c628] + ComplexMixtures v0.5.4
Updating `/tmp/jl_zMmory/Manifest.toml`
[b99e7846] + BinaryProvider v0.5.10
[46823bd8] + Chemfiles v0.9.3
[6f35c628] + ComplexMixtures v0.5.4
[c58ffaec] + FortranFiles v0.6.0
[0f8b85d8] + JSON3 v1.9.2
[e29189f1] + PDBTools v0.12.11
[d96e819e] + Parameters v0.12.3
[92933f4c] + ProgressMeter v1.7.1
[860ef19b] + StableRNGs v1.0.0
[856f2bd8] + StructTypes v1.8.1
[b189fb0b] + ThreadPools v1.2.1
[3a884ed6] + UnPack v1.0.2
[78a364fa] + Chemfiles_jll v0.9.3+1
[9fa8497b] + Future
1 Like
Ok, this is reproducible:
-
add
Plots@1.22.0 -
PDBTools@0.11.0 -
add
ComplexMixtures@0.5.4
The last version of ComplexMixtures requires PDBTools@0.12, thus, when I deliberately add this last version, it triggers the update of PDBTools. (that is expected).
However, it also triggers the update of Plots (that is not expected):
(uptest3) pkg> activate uptest4
Activating new environment at `~/Drive/Work/JuliaPlay/uptest4/Project.toml`
(uptest4) pkg>
(uptest4) pkg> add Plots@1.22.0
Resolving package versions...
Updating `~/Drive/Work/JuliaPlay/uptest4/Project.toml`
[91a5bcdd] + Plots v1.22.0
Updating `~/Drive/Work/JuliaPlay/uptest4/Manifest.toml`
[79e6a3ab] + Adapt v3.3.1
[d360d2e6] + ChainRulesCore v1.11.1
[35d6a980] + ColorSchemes v3.15.0
[3da002f7] + ColorTypes v0.11.0
[5ae59095] + Colors v0.12.8
[34da2185] + Compat v3.40.0
[d38c429a] + Contour v0.5.7
[9a962f9c] + DataAPI v1.9.0
[864edb3b] + DataStructures v0.18.10
[e2d170a0] + DataValueInterfaces v1.0.0
[ffbed154] + DocStringExtensions v0.8.6
[c87230d0] + FFMPEG v0.4.1
[53c48c17] + FixedPointNumbers v0.8.4
[59287772] + Formatting v0.4.2
[28b8d3ca] + GR v0.59.0
[5c1252a2] + GeometryBasics v0.4.1
[42e2da0e] + Grisu v1.0.2
[cd3eb016] + HTTP v0.9.16
[83e8ac13] + IniFile v0.5.0
[3587e190] + InverseFunctions v0.1.1
[92d709cd] + IrrationalConstants v0.1.1
[c8e1da08] + IterTools v1.3.0
[82899510] + IteratorInterfaceExtensions v1.0.0
[692b3bcd] + JLLWrappers v1.3.0
[682c06a0] + JSON v0.21.2
[b964fa9f] + LaTeXStrings v1.2.1
[23fbe1c1] + Latexify v0.15.9
[2ab3a3ac] + LogExpFunctions v0.3.4
[1914dd2f] + MacroTools v0.5.9
[739be429] + MbedTLS v1.0.3
[442fdcdd] + Measures v0.3.1
[e1d29d7a] + Missings v1.0.2
[77ba4419] + NaNMath v0.3.5
[bac558e1] + OrderedCollections v1.4.1
[69de0a69] + Parsers v2.1.2
[ccf2f8ad] + PlotThemes v2.0.1
[995b91a9] + PlotUtils v1.0.15
[91a5bcdd] + Plots v1.22.0
[21216c6a] + Preferences v1.2.2
[3cdcf5f2] + RecipesBase v1.1.2
[01d81517] + RecipesPipeline v0.4.1
[189a3867] + Reexport v1.2.2
[ae029012] + Requires v1.1.3
[6c6a2e73] + Scratch v1.1.0
[992d4aef] + Showoff v1.0.3
[a2af1166] + SortingAlgorithms v1.0.1
[90137ffa] + StaticArrays v1.2.13
[82ae8749] + StatsAPI v1.0.0
[2913bbd2] + StatsBase v0.33.12
[09ab397b] + StructArrays v0.6.3
[3783bdb8] + TableTraits v1.0.1
[bd369af6] + Tables v1.6.0
[5c2747f8] + URIs v1.3.0
[6e34b625] + Bzip2_jll v1.0.8+0
[83423d85] + Cairo_jll v1.16.1+0
[5ae413db] + EarCut_jll v2.2.3+0
[2e619515] + Expat_jll v2.2.10+0
[b22a6f82] + FFMPEG_jll v4.4.0+0
[a3f928ae] + Fontconfig_jll v2.13.93+0
[d7e528f0] + FreeType2_jll v2.10.4+0
[559328eb] + FriBidi_jll v1.0.10+0
[0656b61e] + GLFW_jll v3.3.5+1
[d2c73de3] + GR_jll v0.62.0+0
[78b55507] + Gettext_jll v0.21.0+0
[7746bdde] + Glib_jll v2.68.3+0
[3b182d85] + Graphite2_jll v1.3.14+0
[2e76f6c2] + HarfBuzz_jll v2.8.1+0
[aacddb02] + JpegTurbo_jll v2.1.0+0
[c1c5ebd0] + LAME_jll v3.100.1+0
[dd4b983a] + LZO_jll v2.10.1+0
[e9f186c6] + Libffi_jll v3.2.2+0
[d4300ac3] + Libgcrypt_jll v1.8.7+0
[7e76a0d4] + Libglvnd_jll v1.3.0+3
[7add5ba3] + Libgpg_error_jll v1.42.0+0
[94ce4f54] + Libiconv_jll v1.16.1+1
[4b2f31a3] + Libmount_jll v2.35.0+0
[89763e89] + Libtiff_jll v4.3.0+0
[38a345b3] + Libuuid_jll v2.36.0+0
[e7412a2a] + Ogg_jll v1.3.5+0
[458c3c95] + OpenSSL_jll v1.1.10+0
[91d4177d] + Opus_jll v1.3.2+0
[2f80f16e] + PCRE_jll v8.44.0+0
[30392449] + Pixman_jll v0.40.1+0
[ea2cea3b] + Qt5Base_jll v5.15.3+0
[a2964d1f] + Wayland_jll v1.19.0+0
[2381bf8a] + Wayland_protocols_jll v1.18.0+4
[02c8fc9c] + XML2_jll v2.9.12+0
[aed1982a] + XSLT_jll v1.1.34+0
[4f6342f7] + Xorg_libX11_jll v1.6.9+4
[0c0b7dd1] + Xorg_libXau_jll v1.0.9+4
[935fb764] + Xorg_libXcursor_jll v1.2.0+4
[a3789734] + Xorg_libXdmcp_jll v1.1.3+4
[1082639a] + Xorg_libXext_jll v1.3.4+4
[d091e8ba] + Xorg_libXfixes_jll v5.0.3+4
[a51aa0fd] + Xorg_libXi_jll v1.7.10+4
[d1454406] + Xorg_libXinerama_jll v1.1.4+4
[ec84b674] + Xorg_libXrandr_jll v1.5.2+4
[ea2f1a96] + Xorg_libXrender_jll v0.9.10+4
[14d82f49] + Xorg_libpthread_stubs_jll v0.1.0+3
[c7cfdc94] + Xorg_libxcb_jll v1.13.0+3
[cc61e674] + Xorg_libxkbfile_jll v1.1.0+4
[12413925] + Xorg_xcb_util_image_jll v0.4.0+1
[2def613f] + Xorg_xcb_util_jll v0.4.0+1
[975044d2] + Xorg_xcb_util_keysyms_jll v0.4.0+1
[0d47668e] + Xorg_xcb_util_renderutil_jll v0.3.9+1
[c22f9ab0] + Xorg_xcb_util_wm_jll v0.4.1+1
[35661453] + Xorg_xkbcomp_jll v1.4.2+4
[33bec58e] + Xorg_xkeyboard_config_jll v2.27.0+4
[c5fb5394] + Xorg_xtrans_jll v1.4.0+3
[3161d3a3] + Zstd_jll v1.5.0+0
[0ac62f75] + libass_jll v0.15.1+0
[f638f0a6] + libfdk_aac_jll v2.0.2+0
[b53b4c65] + libpng_jll v1.6.38+0
[f27f6e37] + libvorbis_jll v1.3.7+0
[1270edf5] + x264_jll v2021.5.5+0
[dfaa095f] + x265_jll v3.5.0+0
[d8fb68d0] + xkbcommon_jll v0.9.1+5
[0dad84c5] + ArgTools
[56f22d72] + Artifacts
[2a0f44e3] + Base64
[ade2ca70] + Dates
[8bb1440f] + DelimitedFiles
[8ba89e20] + Distributed
[f43a241f] + Downloads
[b77e0a4c] + InteractiveUtils
[b27032c2] + LibCURL
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[a63ad114] + Mmap
[ca575930] + NetworkOptions
[44cfe95a] + Pkg
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA
[9e88b42a] + Serialization
[1a1011a3] + SharedArrays
[6462fe0b] + Sockets
[2f01184e] + SparseArrays
[10745b16] + Statistics
[fa267f1f] + TOML
[a4e569a6] + Tar
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
[e66e0078] + CompilerSupportLibraries_jll
[deac9b47] + LibCURL_jll
[29816b5a] + LibSSH2_jll
[c8ffd9c3] + MbedTLS_jll
[14a3606d] + MozillaCACerts_jll
[83775a58] + Zlib_jll
[8e850ede] + nghttp2_jll
[3f19e933] + p7zip_jll
(uptest4) pkg> add PDBTools@0.11.0
Resolving package versions...
Updating `~/Drive/Work/JuliaPlay/uptest4/Project.toml`
[e29189f1] + PDBTools v0.11.0
Updating `~/Drive/Work/JuliaPlay/uptest4/Manifest.toml`
[e29189f1] + PDBTools v0.11.0
(uptest4) pkg> add ComplexMixtures@0.5.4
Resolving package versions...
Updating `~/Drive/Work/JuliaPlay/uptest4/Project.toml`
[6f35c628] + ComplexMixtures v0.5.4
[e29189f1] ↑ PDBTools v0.11.0 ⇒ v0.12.11
[91a5bcdd] ↑ Plots v1.22.0 ⇒ v1.23.4
Updating `~/Drive/Work/JuliaPlay/uptest4/Manifest.toml`
[b99e7846] + BinaryProvider v0.5.10
[46823bd8] + Chemfiles v0.9.3
[6f35c628] + ComplexMixtures v0.5.4
[c58ffaec] + FortranFiles v0.6.0
[28b8d3ca] ↑ GR v0.59.0 ⇒ v0.62.1
[0f8b85d8] + JSON3 v1.9.2
[e29189f1] ↑ PDBTools v0.11.0 ⇒ v0.12.11
[d96e819e] + Parameters v0.12.3
[91a5bcdd] ↑ Plots v1.22.0 ⇒ v1.23.4
[92933f4c] + ProgressMeter v1.7.1
[860ef19b] + StableRNGs v1.0.0
[856f2bd8] + StructTypes v1.8.1
[b189fb0b] + ThreadPools v1.2.1
[3a884ed6] + UnPack v1.0.2
[1cfade01] + UnicodeFun v0.4.1
[78a364fa] + Chemfiles_jll v0.9.3+1
[9fa8497b] + Future
(uptest4) pkg>
1 Like
It is not easy to figure out the “smallest possible change” to the environment one can do to add the package. Currently, the strategy works as written in It takes several minutes to add a single package - #90 by kristoffer.carlsson.
1 Like
I can see that this is not easy. But Plots here is not a direct or indirect dependency of any of the packages there. I can add the latest version of ComplexMixtures and after that add Plots@1.22.0 without triggering any downgrade.
You need to recursively look through all versions of the package and all compatible versions of all the dependencies etc. However, I am sure it is possible to improve the current algorithm, any such improvement is gracefully accepted.
2 Likes
Does it have a fallback that finally just updates everything? (it seems to be the case in this example)
Yes, that’s covered in It takes several minutes to add a single package - #90 by kristoffer.carlsson “if that fails, also indirect dependencies.”
1 Like
Ok, I won’t bother you all anymore with this, because I have nothing to contribute in terms in actual code development on that front. But just to register that in that particular case Plots@1.22.0 does not require anything that was installed or upgraded when I installed ComplexMixtures@0.5.4, and as far I as I can see this is demonstrated by the fact that I can install Plots@1.22.0 after installing that package without any conflict. This is the kind of thing that is causing, possibly, the perception that adding anything to a more or less cluttered environment just triggers updating all other packages. If anyone with the talent, skills, and time, has anything to say and do about this, that would be great contribution to the user experience, in my opinion.
Thank you all for the kind feedback. I learnt anyway many tricks here on how to improve my workflow.
MWE:
activate uptest01
add Plots@1.22.0
add PDBTools@0.11.0
add ComplexMixtures@0.5.4 # this triggers the update of everything
Result:
Plots is updated, and:
(uptest01) pkg> st
Status `~/Drive/Work/JuliaPlay/uptest01/Project.toml`
[6f35c628] ComplexMixtures v0.5.4
[e29189f1] PDBTools v0.12.11
[91a5bcdd] Plots v1.23.4
If I change the order in which the packages are added:
activate uptest02
add PDBTools@0.11.0
add ComplexMixtures@0.5.4 # updates PDBTools
add Plots@1.22.0 # works fine
(uptest02) pkg> st
Status `~/Drive/Work/JuliaPlay/uptest03/Project.toml`
[6f35c628] ComplexMixtures v0.5.4
[e29189f1] PDBTools v0.12.11
[91a5bcdd] Plots v1.22.0
Thus Plots@1.22.0 can live with the other package versions, and from a user perspective it is not clear why it was updated the first time.
3 Likes
This may be true in theory, but because there are so many package interdependencies in Julia (which is a good thing!), in practice it seems extremely common for adding a package to trigger updates of dozens of other packages with no obvious (to the user) relationship.
6 Likes