How Does Compat Requirement Changes for General Registry Affect Package Maintainers?

General Usage
,
1

Hi folks!

This is a post about the recent announcement:

I was curious about a few things here:

  1. How does this impact packages that support testing of older Julia versions which did not have this segmentation of standard libraries? At first glance, it almost seems like I have to create a separate Project.toml for these older Julia versions (pre 1.10) that I test my packages on and another toml for these Julia versions after 1.9.
  2. What was the thought process behind this separation of standard libraries from the main Julia distribution?

The 1st question is more coming from me as a package developer and wondering what I should do to accommodate these changes going forward. The 2nd was more just curiosity and what the perceived benefits are about it.

Thanks!

~ tcp :deciduous_tree:

6 Likes
2

Also, just saw this post: Question about new stdlib compact requirements which seems to be asking some similar questions! Maybe we could combine discussion here @longemen3000?

2 Likes
3

Yes! I will close mine

1 Like
4

About 2): IIUC there are essentially no real benefits of having a library in the base sysimage. If the runtime itself uses some functionality from a library, then this library needs to be part of the base sysimage. Separating libraries from the sysimage also decouples the library from the release process of Julia itself, so you donā€™t need to wait for a new release to get basic bugfixes. Another nice bonus is improved startup time, since you donā€™t have to load libraries you donā€™t use.

5 Likes
5

There should be no effect on older versions of Julia. Julia 1.6 implies Statistics 1.6 and vice-versa.

Stdlibs will still be distributed along-side Julia, some were part of the system image primarily for latency reasons. E.g. to lower the startup time of Julia REPL and Pkg were both baked into the system image.

With package images the latency cost of not having them in the system image is acceptable, this means we are more moving to a ā€œif you use it you have to pay for itā€. E.g. if I run Julia as a webserver and I never need REPL and Pkg, why am I required to load them by default. Julia will startup faster (as long as you donā€™t load the REPL) and use less memory.

Lastly ā€œstandard libraries is where code goes to dieā€, the development process for standard libraries was clunky and unlike any other Julia packages and their release cycle bound to the Julia release cycle.

This has lead to the (social) development that rather than contributing and improving standard libraries, folks (including me) have rather preferred working on new packages that extend standard libraries. Why wait 3-6 months for your bug fix or feature to be available if you could release an package version in 15 minutesā€¦

Making standard libraries upgradeable gives us the freedom to put out quicker releases for them, including bug-fixes and respond to CVEs more flexibly.

11 Likes
6

Over the next week or so we will be PRs like Add compat for Distributed by vchuravy Ā· Pull Request #93409 Ā· JuliaRegistries/General Ā· GitHub that add compat information on historical basis.

1 Like
7

This sounds dumb, but does this mean that the versions of the standard libraries will, in time, diverge from the current julia version?

1 Like
8

Not a dumb question at all. Short answer yes.
The goal is for stdlibs to ā€œjustā€ be normal Julia libraries,
with the oy difference that a initial version is shipped with a distribution of Julia.

3 Likes
9

Ah! Thanks for explaining @vchuravy ā€“ this allays all my concerns. In fact, I think this is pretty fantastic! So going forward, if I install a fresh Julia installation (>1.9), would I not only want to do something like install the Julia version I want but also figure out what packages I want?

Iā€™d imagine wanting the following packages in my environment:

  1. Julia Base
  2. Julia REPL
  3. Julia Banner
  4. Shell Mode
  5. Pkg Mode

And install it somehow before first opening Julia? Like julia -e 'Pkg.add(["REPL.jl", "Pkg.jl", ...) and then run julia from the shell like normal?

10

Note that the stdlibs are still shipped with Julia so you shouldnā€™t need to add them to their environment it to get a REPL working, but yes you could update the version of Downloads.jl being used in your Project.

This change is meant to be non-breaking so as Julia works today it should work on 1.11. The one difference is that you may notice us compiling newly cached versions of standard libraries :wink:

5 Likes
11

Ah gotcha gotcha. Thanks for explaining!

I am however letting my mind run wild a little bit with fun ideas for Julia REPL ā€œdistrosā€ that one could build. Exciting times!

12

Hi,
Thanks for taking the time to explain all of this.
There is still one point I donā€™t really understand.
How can I stdlib be shipped with julia but not be in the sysimage?

13

We actually have been doing so for a while.

Julia has two concepts that control the behavior of package loading, LOAD_PATH and DEPOT_PATH. Both function as a sort of stack, where we start our search with first entry and if we are unsuccessful we proceed to the next entry:

julia> Base.DEPOT_PATH
3-element Vector{String}:
 "/home/vchuravy/.julia"
 "/home/vchuravy/.julia/juliaup/j" ā‹Æ 20 bytes ā‹Æ "x64.linux.gnu/local/share/julia"
 "/home/vchuravy/.julia/juliaup/julia-1.10.0-beta3+0.x64.linux.gnu/share/julia"

and

julia> Base.LOAD_PATH
3-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"

The default depot path on your system contains: First the mutable user depot, then the local/share/julia and lastly the global share/julia. The local/share/julia is by default empty and the share/julia contains all the things that we distributed along-side Julia.

share/julia> ls
base/  base.cache  cert.pem  compiled/  julia-config.jl*  stdlib/  test/

In stdlib/v1.10/ is the source code for all of the stdlibs (corresponding to the LOAD_PATH entry @stdlib) and in compiled/v1.10 are the corresponding cache files.

Already early one not all stdlibs were baked into the system image, but rather we load them on demand.

Hope that helps!

6 Likes
14

Thank you so much!
That was quite an AHA moment for me!
Much clearer indeed

15

Iā€™m getting a CI failure for the Richardson.jl package after adding a version requirement for LinearAlgebra:

   Testing Richardson
 Resolving package versions...
ERROR: LoadError: empty intersection between LinearAlgebra@0.0.0 and project compatibility 1.0.0-1

Apparently the package manager for Julia 1.0 thinks that LinearAlgebra is version 0.0.0.

Does this only work with Julia 1.6 or later?

1 Like
16

Thatā€™s the oldest version I testedā€¦ It may be that this was fixed later.

17

I am getting the same problem with FiniteDifferences.jl,

I do not think that Julia 1.0 supports compat entries for stdlibs.

18

@dilumaluthge has suggested using "<0.0.1, 1" to support old Julia versions, like in this PR. And that SHA in particular needs "<0.0.1, 0.7, 1".

It seems there was a Pkg bug that has since been fixed which requires this setting, because during testing at different parts in thinks the stdlib versions as v0.0.0 at times, so the package needs to be compatible with v0.0.0 to be loadable.

From my experimenting it seems Julia 1.0 to 1.3 needs this to be able to run tests at all, while some later Julia versions need this (and all their dependencies to have this) only if the test environment is re-resolved by e.g. Pkg.add during the tests.

Itā€™s not clear to me which versions are affected by which version of the bug since there is no issue filed. Although @kristoffer.carlsson said on Slack that the bug is fixed, so that is good at least.

7 Likes