Forward compatibility and stability of Julia vs. Packages

General Usage
41

You’ve just given three good reasons why packages should not use Julia internals. All I’m saying is that if we claim that the ecosystem follows semantic versioning, then we should strongly encourage the ecosystem to actually follow semantic versioning. I’m still surprised that this is a controversial idea.

8 Likes
42

Well, if 1.9 is technically not released yet, and so isn’t “fair game” to look for packages breaking, I looked at a few popular packages under Julia 1.8.

I took the General registry as 2021-06-01 (just ~1 year before 1.8) and tried to install and precompile — not even test — a few of the most popular Julia packages with Julia 1.8.
Lots didn’t work! Examples include CUDA, FiniteDiff, all of SciML, LoopVectorization, and even IntervalSets.

In my experience, this is the rule, not an exception — that lots of code (eg older package versions) transitively breaks when updating Julia. And this kind of breakage (doesn’t precompile and load) makes it impossible to do anything with those packages, not even a subset of their functionality.

2 Likes
43

This isn’t just about Julia internals. Your example:

[compat]
julia = "=1.6.7"

would also translate to packages using internals of other packages, right? Then x.y would be wrong vs. x.y.z? But I’m guessing we have a problem there too (common for the packages that actually do actually use internals? I.e. no problem in most packages, unless a few such packages are commonly used as direct or indirect dependencies…).

Doing that should be uncontroversial, and NOT stop Pkg and Julia/PkgEval(?), when its about packages and not Julia, as is.

No? Currently yes, it would prevent, since you couldn’t install the packages for testing purposes. Should Pkg have some back-door to allow as if e.g. 1.6 stated despite 1.6.7 being in compat? Even if for just testing purposes, or just PkgEval? A counterpoint, if you have the back-door, people would start using it… But it would be great to have it even then, since then people would walk into the trouble with their eyes open, be asking for it. What is sensible for Julia PkgEval, if not many more users.

We want to “overhead” for users to discourage use of internals?

My plan would also work for them? I discovered test actually run with non-default opt (why I got unexpected (parallel) precompilation), to enable --check-bounds=yes so there’s a precedent to do some things differently for tests.

44

Example? I don’t know of any case that’s not working on v1.9. There are transitory periods where you cannot use an unreleased version of Julia because an autodiff package has not updated, and since a lot of SciML depends on autodiff packages you can be blocked from using nightly until that is fixed. However, I’d like to see what your test case is here since CI shows that “all of SciML” generally passes for any released Julia version?

1 Like
45

Not sure if you read my message in full, because: I didn’t test on 1.9, and didn’t use the current state of packages. Both on purpose.

6 Likes
46

Since DataFrames.jl was mentioned several times. The policy for this package is:

Make sure to provide a release of DataFrames.jl that works on all promised released versions of Julia. Currently we promise that DataFrames.jl will work on Julia 1.6 or newer.

Note that if someone develops a project the specification of project environment is Julia version + package versions. So if someone updates Julia version of the project then also updating of package versions might be required.

7 Likes
47

On the topic of stability, I think the biggest issue right now is that we don’t have tooling for lower bounds. Semvar upper bounds are “fine”. There’s always going to be edge cases on the definition of breaking and public vs private etc. but I think that’s mostly handled other than the few edge cases. It’s lower bounds that’s the issue. When people dig up an old version, the version resolution part is scary is the fact that lower bounds are much less likely to be maintained.

So the most common way to break a good package is to set one of its dependencies far back, let it resolve, and try to use it. If any lower bound wasn’t correctly bumped, you can get a failure. These aren’t too difficult to fix, but IMO it means that we do need more tooling. I’m not sure what the best solution here is. I would for example be okay with something that auto-bumps all lower bounds to the version that was used on CI, or if we have enough compute actually test each package moving each dependency down to the lower bound. If we had this kind of testing then I think our ecosystem would be 10000% more stable.

4 Likes
48

Totally agree! This is an orthogonal issue, and I tried to start a discussion on that some time ago on slack. Not much extra compute is needed to run these checks rarely, from time to time — and even this would catch lots of issues. What’s missing is infrastructure + buyin from packages.

I tried playing with this kind of compat testing for my packages, but buyin is critical — doesn’t make sense to do that for a small subset. See oldnew_compat.yml · GitHub for a github action example that tests for either oldest/newest versions of each dep, or for all versions from oldest to newest.

49

Yes, this!
Newer Julia versions often don’t work with old code => updating packages is potentially needed whenever updating Julia => updating Julia cannot be less painful/more comfortable than updating a package.

2 Likes
50

As a demonstration that the issue is lower bounds and not necessarily upper bounds, I created a test case of DataFrames resolution. I added packages at the minimums, but that wouldn’t resolve because some of the minimum package versions require an older Julia version than what’s allowed (oops), and then after slowly bumping I found a version that would install:

(@v1.9) pkg> add DataFrames Reexport@0.2.0 Missings TableTraits SortingAlgorithms@0.3.0 ShiftedArrays@1.0 IteratorInterfaceExtensions@0.1.1 DataAPI@1.14.0 InlineStrings@1.3.0 Unitful@1.0
   Resolving package versions...
   Installed InlineStrings ───── v1.3.0
   Installed SortingAlgorithms ─ v0.3.0
   Installed Unitful ─────────── v1.0.0
    Updating `~/.julia/environments/v1.9/Project.toml`
  [9a962f9c] + DataAPI v1.14.0
⌃ [842dd82b] + InlineStrings v1.3.0
⌃ [a2af1166] ↓ SortingAlgorithms v0.3.2 ⇒ v0.3.0
⌃ [1986cc42] + Unitful v1.0.0
    Updating `~/.julia/environments/v1.9/Manifest.toml`
  [187b0558] + ConstructionBase v1.5.1
⌃ [842dd82b] ↓ InlineStrings v1.4.0 ⇒ v1.3.0
⌃ [a2af1166] ↓ SortingAlgorithms v0.3.2 ⇒ v0.3.0
⌃ [1986cc42] + Unitful v1.0.0
        Info Packages marked with ⌃ have new versions available and may be upgradable.
Precompiling environment...
  ✗ SortingAlgorithms
  ✗ DataFrames
  2 dependencies successfully precompiled in 14 seconds. 36 already precompiled.
  2 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages
  1 dependency had warnings during precompilation:
┌ Unitful [1986cc42-f94f-5a68-af5c-568840ba703d]
│  WARNING: could not import FastMath.libm into Unitful

And boom that failed to precompile. This is not to single out @bkamins of course because you can find many examples of this around. For example, digging through Discourse search I found

where the issue was that someone added a non-updated Trebuchet.jl to their environment that burned it. This is then the interaction of two things:

  1. Packages that haven’t updated in a long time are bad for the environment.
  2. Lower bounds not auto-updating in any way means that a package that makes an environment generally old may install, but may be using a combination that has never been tested before and fails.

Of course the “simple” fix is to just remove the package that is old and bad. In that case, Trebuchet.jl either needed to be removed from one’s environment or updated, and in that case we did both and the world went back to normal. But, it does point to the fact that unmaintained repos are a “poison” that is hard to deal with.

So over time one of the things we have done with SciML, especially through 2022, is gobbling up some of these periphery libraries to keep them up-to-date and maintained because the real problems aren’t DifferentialEquations.jl but things like Trebuchet.jl or MomentClosure.jl that people install one time to follow a blog post and then never realize that their version environment is pretty messed up.

What can we do about this?

  1. The aforementioned tooling on lower bounds. If lower bounds bumped correctly, you’d either get an older version of DifferentialEquations.jl that worked, or it would say it just cannot install. Honestly I’m fine with either, as a package resolution issue is a nice clear sign to the user that they have some mess to clean up and its their fault not something with the packages.
  2. Some way of pointing out “old packages”. Even if it’s just a heuristic. The most common issue has an answer of “Yo, PackageX.jl last had a release during the JuliaCon v1.0 release event. You should expect that as long as this on your system that pretty much all other packages will not be good”. I’d prefer a very bold warning than the current silence, since this is the issue almost all of the time.
  3. It would be nice to single out any package that isn’t allowing a recent version of a dependency. The new v1.8 package manager ^ is very helpful, but I’d like to know if there’s anything in my environment that has an unmerged CompatHelper PR. Anything that’s like that either should have an issue discussing why it’s not up to date, or is unmaintained. Either way, it’s a nice signal to the red flags in the ecosystem.
  4. Periodic upper bounding of past releases. I would be interested at around the next LTS (v1.10?) to just make everything in current SciML not be allowed on v1.10, and just release new versions. Why? Over the last 5 years or so there have been some missed bounds. And what then tends to happen is that the cases where you may have missed a bound one time are the cases which have a loose bound. And then because that case has a loose bound, the package manager goes “wow this version is compatible with everything! Very nice!” and is very ready to pick that version over and over whenever you have one of the unmaintained packages in the system. So as a clean up measure, we need some way to clean those versions out so the package manager is in a cleaner slate to better resolve again.

Anyways, hopefully that distills what the core issues are. But in general this is pointing to missing lower bounds updates being some of the biggest issues, not upper bounds. We can debate the merits of SemVar, but that’s really small potatoes and is only exposing this other issue by making an unmaintained package cause resolution to the past, which then fails to work well because of untested lower bounds.

3 Likes
51

My worry here is that I think CompatHelper is not enough. First because after some time it stops running, and second because not every package uses it.

IMHO an automated program that ran tests of packages with new dependency versions and suggested PRs is needed, independently of the CI setup of the packages.

Unmaintained repos will be more and more frequent, particularly in those cases where the author expects that nothing should be breaking the usage of the package given he/she has being careful in not using internals of other packages and provided appropriate compat entries. We need an external tool running tests to figure out these issues.

1 Like
52

I’m a strong opponent of preventing access to internals entirely. We don’t have tools for checking whether something is internal or not is exactly what the discussion on github linked above is about:

My position on this and on why we need that should be clear from that (I’m Seelengrab on Github… I really should get on changing that name).

There are release candidates, which are absolutely fair game to test stuff with. That’s exactly their purpose - catching potential breakage before a version is released.

This is attacking the symptom, not the cause. A failure due to “lower bounds” in an updated version can have one of three causes:

  1. The package originally specified “too friendly” upper bounds, spanning more than one major (=breaking) version. The failure is fixable by setting a correct upper bound. The package may not install at all anymore, but that’s a MUCH better outcome than it just outright crashing with a MethodError or something like that.
  2. A dependency of the package made a breaking change, but didn’t bump the major version.
  3. The package relied on an internal object that was fair game to remove.

None of these is fixed by just running CI on old versions. You end up playing whack-a-mole and running after accidentally breaking changes. It’s better to plan for this from the start, by being able to programmatically declare what actually is API and then never removing tests that test the declared API (unless you make a breaking change - then anything goes). Having such a capability would instantly solve 2 & 3, because it’s obvious during development when you accidentally make a breaking change, as well as when you rely on an internal of a dependency.

The cause is us not having a way of defining what the API surface of a given thing actually is, in a way that is accessible programmatically. So let’s fix that.

5 Likes
53

Indeed, but it is a pure SortingAlgorithms.jl issue (i.e. this is not caused by a combination of packages issue):

(test) pkg> st
Status `C:\WORK\dev\test\Project.toml`
⌃ [a2af1166] SortingAlgorithms v0.3.0
Info Packages marked with ⌃ have new versions available and may be upgradable.

julia> versioninfo()
Julia Version 1.9.0-rc2
Commit 72aec423c2 (2023-04-01 10:41 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 12 × 12th Gen Intel(R) Core(TM) i7-1250U
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, alderlake)
  Threads: 1 on 12 virtual cores

julia> using SortingAlgorithms
[ Info: Precompiling SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c]
ERROR: LoadError: UndefVarError: `Float` not defined
2 Likes
54

I think it’s worth taking a step back and noting that there are four different flavors of compatibility being discussed in this thread.

  • Holding packages constant and changing Julia versions forwards (Julia’s forwards compatibility)
    • This is where we started — and what @aplavin wants to have work, with that concrete case of SortingAlgorithms removing its prior dependency on internals. And this is where @CameronBieganek would put upper bounds to prevent the use of some old packages on newer Julias by rule. This is where we have PkgEval to help test new Julia releases and ensure the package ecosystem works with the new Julia version (yes, potentially by requiring new package versions).
  • Holding packages constant and changing Julia versions backwards (Julia’s backwards compatibility)
  • Holding one package constant and changing another package forwards
    • These are the typical semver upper bounds that get put in [compat] — and these are indeed becoming more common thanks to CompatHelper.jl and this is the most common case in which it bumps things up. It’s much much less common for packages to do the equivalent of Julia’s PkgEval when releasing new versions, and this is where @Elrod is saying it can be more dicey than that first bullet.
  • Holding one package constant and changing another package backwards
    • This is the case that @ChrisRackauckas describes with old Trebuchets and pinned DataFrames ecosystem. Just as with Julia’s backwards compatibility it’s not guaranteed to work by the terms of semver but unlike Julia folks are much less likely to think about the package versions that introduce new features and bound them because it’s not often tested. And it’s much harder to test (typically folks have a matrix of Julia versions in CI, this would require a matrix of package dependency versions). CompatHelper will help you get started here, but it won’t identify that some new code relied upon newer dependency than you started with.
14 Likes
55

Thanks for describing all these scenarios in details!

I think the last point,

is among the easiest to fix — at least it’s definitely possible, and a potential solution is readily apparent:

Meanwhile, the first point

is not clear if actionable at all. Maybe, the best what can be done is informing users that existing code/packages often breaks with newer Julia versions, they aren’t expected to work with exact same code as earlier ones.
And, of course, try to depend on internals less — but this happens by necessity, I don’t think package authors deliberately dig into Julia internals.

is not a solution, because this doesn’t keep old code but replaces it with updated one.

56

Packages ARE expected to work with newer Julia, so I’m not sure if/how we would want to inform. Yes, that’s only if internals aren’t used, and while such packages may be infrequent, in practice it could affect many packages, or influential ones, because of dependency on such packages. So does that change the picture? Ideally no, since PkgEval should catch those and internals that would have caused breakage aren’t changed for final release (or RC). In some cases they are changed however… and considered allowed, by SemVer. Then some (few?) packages need to be updated that rely on those internals, but what does it mean for the dependants of those? If I understand, then they may not need updating (however it makes the Manifest idea broken, and thus e.g. Pluto, right…?).

57

This is where I mildly disagree. New versions of compilers can and do expose downstream software bugs — bugs that require addressing for it to be used in the new version — and this can happen in any language ecosystem and can even happen if the downstream software isn’t explicitly “mucking about” in the compiler internals. A classic example is C’s undefined behaviors and how the newer versions of optimizing compilers are getting more and more aggressive at exploiting such cases. It can be easy to accidentally fall into undefined behavior that happens to work ok until a new version decides to punish you for it. Personally I find that to be far too punitive and user-hostile — just because the compiler team can say RTFM when downstream code breaks doesn’t mean that they don’t share in any of the responsibility to help prevent such things from happening.

The polar opposite approach to compatibility (never break anything in downstream code) leads you to never changing anything — as it’s possible for folks to depend upon buggy compiler behaviors and correct for them in their code in a manner that would break upon the original issue’s fix.

That’s precisely why I really appreciate all the work that goes into PkgEval. There’s a shared responsibility here. SemVer isn’t a legal contract; it’s a best-effort sort of way of communicating among ourselves about how software changes.

6 Likes
58

Note that I never promoted

approach. What I propose is better informing users, that despite PkgEval (which is a great project!) newer Julia versions regularly break old code (eg old packages) in practice. It’s not an objectively bad thing, it’s just something to keep in mind.

Speaking from my own experience getting familiar with Julia, I really thought that this kind of breakage should be extremely rare or minimal, given how PkgEval is taken seriously. While in practice, lots of popular packages require updates to get them working on newer Julia versions. It’s nice to let potential users know about this (somewhere).

And looking at this very thread, somehow I even have troubles convincing (some) others, that this breakage actually happens (:

3 Likes
59

Sorry, didn’t mean for those quotes to put words in anyone’s mouth here (I was thinking of Linux and Rich Hickey when I wrote them; Rich’s talk is quite good as usual and worth a watch) — I’ve removed them.

I think there are two or three things getting lost in the fray here:

  • Julia updates don’t break typical user and package code — and we have pretty high confidence in this from running PkgEval.
  • A number of packages do the extraordinary and are themselves quite atypical. Things like GPU compilation, differentiation, compiler-adjacent optimizations like LoopVectorization, etc, etc. And these extraordinary things often sit towards the bottom of the dependency trees. The number here is pretty small, but a good number of upstream packages do depend upon them, so if they break it has a larger splash radius and appears worse than it is. For example, the oldest version of IntervalSets (v0.2.1 from 2018) does work on Julia v1.8 and passes all its functional* tests if allowed to use newer versions of its dependencies.
  • These packages — the latest versions of them — aren’t broken when a new version of Julia is released. But, yes, older versions might be.
6 Likes
60

Breakage in software (even with use of internals, it seems) can be avoided.

What does (forward) compatibility/“breakage” in software mean, and dependency on “internals” (not official API)?

Breakage can only mean (unintended bugs or) deleting stuff (Rick Hickey has a very good talk on this, and why SemVer not needed). Note, changing stuff, e.g. names, is basically deleting then adding [new name].

And it can only mean deleting from an API, or internals (neither is good), something you can call, or deleting (or accessing/mutating) a member of some struct or class.

If it’s a simple name change, maybe you had a typo, then it seems wanted, in theory you can keep both names, deprecate the older. When do you really need to drop the deprecated name? Never really, Rick argues, at least in open source code (if you want to respect your users). Before publishing, registering for us (or for non-public code [of a company]), you can justify it.

Another “need” is accessing a member of a class, or struct. Julia has no syntax to allow for aliases of member names (but it would be plausible in a language). I’ll concentrate on a different part of it, should you be accessing it somehow, or even know the name?

Since I mentioned encapsulation before (the proposed right way according to Parnas’ principles, having internals you don’t need, or shouldn’t know, or change directly), there’s an alternative:

How do you achieve encapsulation with Clojure? Clojure - Frequently Asked Questions

Because of its focus on immutable data, there is generally not a high value placed on data encapsulation. Because data is immutable, there is no need to worry about someone else modifying a value. Likewise, because Clojure data is designed to be manipulated directly, there is significant value in providing direct access to data, rather than wrapping it in APIs.

So APIs, “internals”, and mutability are considered harmful(?).

I’ll make this concrete with two examples (this one then my own):

Was it “internals”? I didn’t look to carefully at this, it seems like a red herring if it was internals, the real reason is the “name” change.

To make this concrete, why (did @kristoffer.carlsson) delete aliases?

It meant another package had to make the corresponding change:

For my own example, rand() used to mean MersenneTwister, and I proposed a new (faster) RNG, and thankfully it was implemented in 1.7. The “API”, the name, is the same, but the stream of numbers isn’t (nor was it promized). It would have been very annoying to add rand2() for it, and make it an opt-in. The older RNG is still available, but an opt-out (and might go away in 2.0, possibly sooner if no one cares, I’m not sure many actually use it since all got migrated to the new one, and still it would be somewhat wrong to drop it).

[The new RNG was proposed because it’s faster, not because the old one was insecure, it could have been, then it would have been (even more) justified to do the change.]