As a demonstration that the issue is lower bounds and not necessarily upper bounds, I created a test case of DataFrames resolution. I added packages at the minimums, but that wouldn’t resolve because some of the minimum package versions require an older Julia version than what’s allowed (oops), and then after slowly bumping I found a version that would install:
(@v1.9) pkg> add DataFrames Reexport@0.2.0 Missings TableTraits SortingAlgorithms@0.3.0 ShiftedArrays@1.0 IteratorInterfaceExtensions@0.1.1 DataAPI@1.14.0 InlineStrings@1.3.0 Unitful@1.0
Resolving package versions...
Installed InlineStrings ───── v1.3.0
Installed SortingAlgorithms ─ v0.3.0
Installed Unitful ─────────── v1.0.0
Updating `~/.julia/environments/v1.9/Project.toml`
[9a962f9c] + DataAPI v1.14.0
⌃ [842dd82b] + InlineStrings v1.3.0
⌃ [a2af1166] ↓ SortingAlgorithms v0.3.2 ⇒ v0.3.0
⌃ [1986cc42] + Unitful v1.0.0
Updating `~/.julia/environments/v1.9/Manifest.toml`
[187b0558] + ConstructionBase v1.5.1
⌃ [842dd82b] ↓ InlineStrings v1.4.0 ⇒ v1.3.0
⌃ [a2af1166] ↓ SortingAlgorithms v0.3.2 ⇒ v0.3.0
⌃ [1986cc42] + Unitful v1.0.0
Info Packages marked with ⌃ have new versions available and may be upgradable.
Precompiling environment...
✗ SortingAlgorithms
✗ DataFrames
2 dependencies successfully precompiled in 14 seconds. 36 already precompiled.
2 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages
1 dependency had warnings during precompilation:
┌ Unitful [1986cc42-f94f-5a68-af5c-568840ba703d]
│ WARNING: could not import FastMath.libm into Unitful
And boom that failed to precompile. This is not to single out @bkamins of course because you can find many examples of this around. For example, digging through Discourse search I found
where the issue was that someone added a non-updated Trebuchet.jl to their environment that burned it. This is then the interaction of two things:
- Packages that haven’t updated in a long time are bad for the environment.
- Lower bounds not auto-updating in any way means that a package that makes an environment generally old may install, but may be using a combination that has never been tested before and fails.
Of course the “simple” fix is to just remove the package that is old and bad. In that case, Trebuchet.jl either needed to be removed from one’s environment or updated, and in that case we did both and the world went back to normal. But, it does point to the fact that unmaintained repos are a “poison” that is hard to deal with.
So over time one of the things we have done with SciML, especially through 2022, is gobbling up some of these periphery libraries to keep them up-to-date and maintained because the real problems aren’t DifferentialEquations.jl but things like Trebuchet.jl or MomentClosure.jl that people install one time to follow a blog post and then never realize that their version environment is pretty messed up.
What can we do about this?
- The aforementioned tooling on lower bounds. If lower bounds bumped correctly, you’d either get an older version of DifferentialEquations.jl that worked, or it would say it just cannot install. Honestly I’m fine with either, as a package resolution issue is a nice clear sign to the user that they have some mess to clean up and its their fault not something with the packages.
- Some way of pointing out “old packages”. Even if it’s just a heuristic. The most common issue has an answer of “Yo, PackageX.jl last had a release during the JuliaCon v1.0 release event. You should expect that as long as this on your system that pretty much all other packages will not be good”. I’d prefer a very bold warning than the current silence, since this is the issue almost all of the time.
- It would be nice to single out any package that isn’t allowing a recent version of a dependency. The new v1.8 package manager
^is very helpful, but I’d like to know if there’s anything in my environment that has an unmerged CompatHelper PR. Anything that’s like that either should have an issue discussing why it’s not up to date, or is unmaintained. Either way, it’s a nice signal to the red flags in the ecosystem. - Periodic upper bounding of past releases. I would be interested at around the next LTS (v1.10?) to just make everything in current SciML not be allowed on v1.10, and just release new versions. Why? Over the last 5 years or so there have been some missed bounds. And what then tends to happen is that the cases where you may have missed a bound one time are the cases which have a loose bound. And then because that case has a loose bound, the package manager goes “wow this version is compatible with everything! Very nice!” and is very ready to pick that version over and over whenever you have one of the unmaintained packages in the system. So as a clean up measure, we need some way to clean those versions out so the package manager is in a cleaner slate to better resolve again.
Anyways, hopefully that distills what the core issues are. But in general this is pointing to missing lower bounds updates being some of the biggest issues, not upper bounds. We can debate the merits of SemVar, but that’s really small potatoes and is only exposing this other issue by making an unmaintained package cause resolution to the past, which then fails to work well because of untested lower bounds.