A call to all Julia package authors to add Downgrade CI with scripts to copy-paste! This is an article about how to improve testing of package version compatibility and new systems we have adopted in SciML that I would like to see throughout the rest of the ecosystem. For full details see the article:
slap that into .github/workflows/Downgrade.yml and you’ll get the new CI group running.
Call to Action: Help Out?
Downgrade CI is actually really hard to get working the first time. Lower bounds can be old. If you have a little bit of time, help out by adding downgrade CI to a package you love. Note that it will take a bit of sweat. You most likely will need to bump some lower bounds, so this may take an hour or so. My recommendation is to be conservative and bump packages as high as possible because otherwise it’s simply too difficult to find the right granularity.
Good luck, I hope by this time next year everyone will have CI running that checks minimum versions and we should see a lot less version compatibility issues!
PSA to the PSA: the cjdoris/julia-downgrade-compat-action action is being upstreamed to the julia-actions repo right now. Mentioning it here for folks that have enabled strict access control for actions and whitelists.
I tried this on my package, and got a surpising issue:
ERROR: LoadError: Unsatisfiable requirements detected for package Random [9a3f8284]:
Random [9a3f8284] log:
├─possible versions are: 1.10.0 or uninstalled (package in sysimage!)
└─restricted to versions 1.6 by Copulas [ae264745] — no versions left
└─Copulas [ae264745] log:
├─possible versions are: 0.1.20 or uninstalled
└─Copulas [ae264745] is fixed to version 0.1.20
I’m not 100% I understand your action. I tried to use it on the repo here. However, the lower versions that fail seem to be lower than what the compat allows. So I don’t see how increasing the upper bound makes them pass.
EDIT: it might be because I’m adding the package master branch. I will try with “.”
This seems to lead to issues in resolution when my package supports versions 0.1, 0.2 of a dependency B, but another dependency C only supports version 0.2 of B. Forcing version 0.1 will lead to errors in package resolution, but this seems harmless otherwise, as version 0.1 would not be normally installable? Ideally, the compat bounds should be kept updated, but it’s a bit painful to do this every time the bounds of a common dependency are updated upstream.
I don’t think this is your responsibility. Your compatibility bounds are perfectly reasonable. It is the job of the package manager to resolve a compatible set of dependencies. If the downgrade CI pins every package at the minimum version, but it’s not actually an installable environment, then that’s a problem with the downgrade CI.
Yes, this is the problem I am currently having. Pinning all my dependencies to their lower bounds ends up in a contradictory environment, but each of these bounds are actually achievable taken separately so I do not want to change them. The issue is that now I cannot get a “all-lower-bound” environement to test my functionalities in…
Maybe the compat pinning action could do something here and try to “resolve as low as possible” the environment instead of “resolving as high as possible” like Pkg is doing normally.
oldnew_compat.yml · GitHub does basically that, finding the lowest resolvable compat separately for each direct dependency. The actual implementation is in the CompatHelperLocal.jl package, around these lines. Also can test low compat locally as well, with test_compats_combinations function (undocumented).
The global “resolve as low as possible” flag would be nice to have in Pkg! Using it in testing would miss lowerbounds that are only achievable separately but not at once, though. Still better than no testing at all!
I fear that choosing lower bounds such that all packages can be pinned there and it still resolvable might be too strict, as in packages with many dependencies this might force one to pick quite high bounds and therefore limit compatibility with other packages, even though in practice only a fraction of the dependencies might be held back.
At the moment it seems that pinning only individual packages as @aplavin’s action does is the better choice. This gives you lower bounds that are more reasonable, even though holding back multiple packages might still lead to failures.
Why the OP’s recipe is a good default (in my opinion):
Life is too short to aim to make your package super compatible with old versions of its dependencies.
CI is too expensive to be testing lots of combinations of old versions of dependencies.
If it works for SciML, it probably works for you too.
There can be exponentially many minimal configurations of dependencies. aplavin’s recipe does not exhaust these, although it probably does catch most incompatibilities in practice.
The OP’s recipe forces there to be a single minimal configuration chosen, and therefore is exhaustive.
Cases where choosing one single minimal configuration causes someone’s project to become unresolvable is highly unlikely. It can basically only happen in cases like where PkgA depends on DepA v1 and DepB v2 but PkgB depends on DepA v2 and DepB v1. The ideal solution there is to make PRs to PkgA and PkgB to be compatible with DepA v2 and DepB v2.
Identifying which lower bounds are at fault (aplavin’s recipe does this) is cool. But it’s usually pretty easy to work it out reading the test logs from the OP’s recipe. And you’ll need to do that anyway in order to work out what the lower bound should be unless you’re going to bisect your way there or just set it to the latest version.
The general recommendation from Julia core devs is to just make your package compatible with the latest version of Julia, and then maybe apply minimal effort to see how far back it happens to be compatible. The OP’s recipe is the logical extension of this advice to compat bounds for packages.
Tried for 2 hours before giving up and posting here. Gone through at least 12 comits of different bounds, each time going to other packages repo and finding out using git blame when the lower bound changed… And it is still not working.
I think our packages have in common that they both depend on Distributions.jl and StatsBase.jl. I think Distributions.jl only bumped the compat of StatsBase.jl to 0.34 by v0.25.88. So putting anything lower than 0.25.88 for Distributions.jl and 0.34 for StatsBase.jl won’t work (IIRC).