How does this impact packages that support testing of older Julia versions which did not have this segmentation of standard libraries? At first glance, it almost seems like I have to create a separate Project.toml for these older Julia versions (pre 1.10) that I test my packages on and another toml for these Julia versions after 1.9.
What was the thought process behind this separation of standard libraries from the main Julia distribution?
The 1st question is more coming from me as a package developer and wondering what I should do to accommodate these changes going forward. The 2nd was more just curiosity and what the perceived benefits are about it.
About 2): IIUC there are essentially no real benefits of having a library in the base sysimage. If the runtime itself uses some functionality from a library, then this library needs to be part of the base sysimage. Separating libraries from the sysimage also decouples the library from the release process of Julia itself, so you don’t need to wait for a new release to get basic bugfixes. Another nice bonus is improved startup time, since you don’t have to load libraries you don’t use.
There should be no effect on older versions of Julia. Julia 1.6 implies Statistics 1.6 and vice-versa.
Stdlibs will still be distributed along-side Julia, some were part of the system image primarily for latency reasons. E.g. to lower the startup time of Julia REPL and Pkg were both baked into the system image.
With package images the latency cost of not having them in the system image is acceptable, this means we are more moving to a “if you use it you have to pay for it”. E.g. if I run Julia as a webserver and I never need REPL and Pkg, why am I required to load them by default. Julia will startup faster (as long as you don’t load the REPL) and use less memory.
Lastly “standard libraries is where code goes to die”, the development process for standard libraries was clunky and unlike any other Julia packages and their release cycle bound to the Julia release cycle.
This has lead to the (social) development that rather than contributing and improving standard libraries, folks (including me) have rather preferred working on new packages that extend standard libraries. Why wait 3-6 months for your bug fix or feature to be available if you could release an package version in 15 minutes…
Making standard libraries upgradeable gives us the freedom to put out quicker releases for them, including bug-fixes and respond to CVEs more flexibly.
Ah! Thanks for explaining @vchuravy – this allays all my concerns. In fact, I think this is pretty fantastic! So going forward, if I install a fresh Julia installation (>1.9), would I not only want to do something like install the Julia version I want but also figure out what packages I want?
I’d imagine wanting the following packages in my environment:
And install it somehow before first opening Julia? Like julia -e 'Pkg.add(["REPL.jl", "Pkg.jl", ...) and then run julia from the shell like normal?
Note that the stdlibs are still shipped with Julia so you shouldn’t need to add them to their environment it to get a REPL working, but yes you could update the version of Downloads.jl being used in your Project.
This change is meant to be non-breaking so as Julia works today it should work on 1.11. The one difference is that you may notice us compiling newly cached versions of standard libraries
Julia has two concepts that control the behavior of package loading, LOAD_PATH and DEPOT_PATH. Both function as a sort of stack, where we start our search with first entry and if we are unsuccessful we proceed to the next entry:
The default depot path on your system contains: First the mutable user depot, then the local/share/julia and lastly the global share/julia. The local/share/julia is by default empty and the share/julia contains all the things that we distributed along-side Julia.
base/ base.cache cert.pem compiled/ julia-config.jl* stdlib/ test/
In stdlib/v1.10/ is the source code for all of the stdlibs (corresponding to the LOAD_PATH entry @stdlib) and in compiled/v1.10 are the corresponding cache files.
Already early one not all stdlibs were baked into the system image, but rather we load them on demand.
@dilumaluthge has suggested using "<0.0.1, 1" to support old Julia versions, like in this PR. And that SHA in particular needs "<0.0.1, 0.7, 1".
It seems there was a Pkg bug that has since been fixed which requires this setting, because during testing at different parts in thinks the stdlib versions as v0.0.0 at times, so the package needs to be compatible with v0.0.0 to be loadable.
From my experimenting it seems Julia 1.0 to 1.3 needs this to be able to run tests at all, while some later Julia versions need this (and all their dependencies to have this) only if the test environment is re-resolved by e.g. Pkg.add during the tests.
It’s not clear to me which versions are affected by which version of the bug since there is no issue filed. Although @kristoffer.carlsson said on Slack that the bug is fixed, so that is good at least.