PSA: Add Downgrade CI to Better Check Version Compatibility

Community
, , ,
1

A call to all Julia package authors to add Downgrade CI with scripts to copy-paste! This is an article about how to improve testing of package version compatibility and new systems we have adopted in SciML that I would like to see throughout the rest of the ecosystem. For full details see the article:

I want to start testing my lower bounds!

Good, just steal this script SciMLBase.jl/.github/workflows/Downgrade.yml at v2.24.0 · SciML/SciMLBase.jl · GitHub

name: Downgrade
on:
  pull_request:
    branches:
      - master
    paths-ignore:
      - 'docs/**'
  push:
    branches:
      - master
    paths-ignore:
      - 'docs/**'
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        version: ['1']
    steps:
      - uses: actions/checkout@v4
      - uses: julia-actions/setup-julia@v1
        with:
          version: ${{ matrix.version }}
      - uses: cjdoris/julia-downgrade-compat-action@v1
        with:
          skip: Pkg,TOML
      - uses: julia-actions/julia-buildpkg@v1
      - uses: julia-actions/julia-runtest@v1

slap that into .github/workflows/Downgrade.yml and you’ll get the new CI group running.

Call to Action: Help Out?

Downgrade CI is actually really hard to get working the first time. Lower bounds can be old. If you have a little bit of time, help out by adding downgrade CI to a package you love. Note that it will take a bit of sweat. You most likely will need to bump some lower bounds, so this may take an hour or so. My recommendation is to be conservative and bump packages as high as possible because otherwise it’s simply too difficult to find the right granularity.

Good luck, I hope by this time next year everyone will have CI running that checks minimum versions and we should see a lot less version compatibility issues!

26 Likes
2

Glad to finally see someone trying to popularize this kind of testing!

Multiple approaches for it were suggested in another thread last year.
The one I put there (oldnew_compat.yml · GitHub) gives nicer reports:

  • doesn’t even try testing unresolvable combinations, no point in doing that
  • indicates which specific dependency downgrade breaks the tests and how:
    image
    image548×1028 76.4 KB

    each GH job can be clicked on.
  • but is more expensive to run, does 2n tests – oldest/newest of each dep

So it can be applied to basically any package without fiddling with compats beforehand.

2 Likes
3

I think adding this to PkgTemplates.jl would also be a good idea? If someone wants to do a PR, be my guest, otherwise I can do it

2 Likes
4

Great, I would love to mass install this on many JuliaDynamics packages. Anyone have a simple script that does a “mass add of a given file” to a list of repos?

5

PSA to the PSA: the cjdoris/julia-downgrade-compat-action action is being upstreamed to the julia-actions repo right now. Mentioning it here for folks that have enabled strict access control for actions and whitelists.

1 Like
6

George, I belive you are asking for MassInstallAction.jl GitHub - julia-actions/MassInstallAction.jl: Quickly and easily install a GitHub Action on all repositories in a GitHub organization - it has been used in the past to create PRs to add or update actions for pretty much the entire Julia ecosystem.

2 Likes
7

I tried this on my package, and got a surpising issue:

ERROR: LoadError: Unsatisfiable requirements detected for package Random [9a3f8284]:
 Random [9a3f8284] log:
 ├─possible versions are: 1.10.0 or uninstalled (package in sysimage!)
 └─restricted to versions 1.6 by Copulas [ae264745] — no versions left
   └─Copulas [ae264745] log:
     ├─possible versions are: 0.1.20 or uninstalled
     └─Copulas [ae264745] is fixed to version 0.1.20

that you can look at there : Add downgrade CI · lrnv/Copulas.jl@6b48904 · GitHub

Looks like the proposed workflow is running on julia 1.10, but requirements for packages in the sysimage (such as Random) are 1.6 and cannot be matched.

8

I’m not 100% I understand your action. I tried to use it on the repo here. However, the lower versions that fail seem to be lower than what the compat allows. So I don’t see how increasing the upper bound makes them pass.

EDIT: it might be because I’m adding the package master branch. I will try with “.”

9

Which specific version is that? For example, the tested Distributions 0.24.15 is clearly allowed by Project.toml.

The issue can be caused by that package doing fancy stuff during testing, in AbstractGPs.jl/test/runtests.jl at cf311bd28fa27ba4bfa6c9d042a694d327e4fd10 · JuliaGaussianProcesses/AbstractGPs.jl · GitHub. I don’t understand why that’s needed instead of regular ]test.

10

Actually it now works as intended. I was checking out the master branch in the step, instead of the local branch.

11

You can skip certain packages with the skip keyword of the action, see OP.

12

Ty. However I still have troubles finding the right set of compatibles lower-bounds. Is there a way I can ask Pkg to tell me the lowest set a compatible dependencies ?

1 Like
13

This seems to lead to issues in resolution when my package supports versions 0.1, 0.2 of a dependency B, but another dependency C only supports version 0.2 of B. Forcing version 0.1 will lead to errors in package resolution, but this seems harmless otherwise, as version 0.1 would not be normally installable? Ideally, the compat bounds should be kept updated, but it’s a bit painful to do this every time the bounds of a common dependency are updated upstream.

1 Like
14

I don’t think this is your responsibility. Your compatibility bounds are perfectly reasonable. It is the job of the package manager to resolve a compatible set of dependencies. If the downgrade CI pins every package at the minimum version, but it’s not actually an installable environment, then that’s a problem with the downgrade CI.

1 Like
15

Yes, this is the problem I am currently having. Pinning all my dependencies to their lower bounds ends up in a contradictory environment, but each of these bounds are actually achievable taken separately so I do not want to change them. The issue is that now I cannot get a “all-lower-bound” environement to test my functionalities in…

Maybe the compat pinning action could do something here and try to “resolve as low as possible” the environment instead of “resolving as high as possible” like Pkg is doing normally.

1 Like
16

oldnew_compat.yml · GitHub does basically that, finding the lowest resolvable compat separately for each direct dependency. The actual implementation is in the CompatHelperLocal.jl package, around these lines. Also can test low compat locally as well, with test_compats_combinations function (undocumented).

The global “resolve as low as possible” flag would be nice to have in Pkg! Using it in testing would miss lowerbounds that are only achievable separately but not at once, though. Still better than no testing at all!

1 Like
17

I fear that choosing lower bounds such that all packages can be pinned there and it still resolvable might be too strict, as in packages with many dependencies this might force one to pick quite high bounds and therefore limit compatibility with other packages, even though in practice only a fraction of the dependencies might be held back.

At the moment it seems that pinning only individual packages as @aplavin’s action does is the better choice. This gives you lower bounds that are more reasonable, even though holding back multiple packages might still lead to failures.

1 Like
18

Why the OP’s recipe is a good default (in my opinion):

  • Life is too short to aim to make your package super compatible with old versions of its dependencies.
  • CI is too expensive to be testing lots of combinations of old versions of dependencies.
  • If it works for SciML, it probably works for you too.
  • There can be exponentially many minimal configurations of dependencies. aplavin’s recipe does not exhaust these, although it probably does catch most incompatibilities in practice.
  • The OP’s recipe forces there to be a single minimal configuration chosen, and therefore is exhaustive.
  • Cases where choosing one single minimal configuration causes someone’s project to become unresolvable is highly unlikely. It can basically only happen in cases like where PkgA depends on DepA v1 and DepB v2 but PkgB depends on DepA v2 and DepB v1. The ideal solution there is to make PRs to PkgA and PkgB to be compatible with DepA v2 and DepB v2.
  • Identifying which lower bounds are at fault (aplavin’s recipe does this) is cool. But it’s usually pretty easy to work it out reading the test logs from the OP’s recipe. And you’ll need to do that anyway in order to work out what the lower bound should be unless you’re going to bisect your way there or just set it to the latest version.
  • The general recommendation from Julia core devs is to just make your package compatible with the latest version of Julia, and then maybe apply minimal effort to see how far back it happens to be compatible. The OP’s recipe is the logical extension of this advice to compat bounds for packages.
5 Likes
19

100% for me for the moment.

Tried for 2 hours before giving up and posting here. Gone through at least 12 comits of different bounds, each time going to other packages repo and finding out using git blame when the lower bound changed… And it is still not working.

If it is so easy, please be my guest there [CI] Add downgrade checks by lrnv · Pull Request #138 · lrnv/Copulas.jl · GitHub :slight_smile:

3 Likes
20

I think our packages have in common that they both depend on Distributions.jl and StatsBase.jl. I think Distributions.jl only bumped the compat of StatsBase.jl to 0.34 by v0.25.88. So putting anything lower than 0.25.88 for Distributions.jl and 0.34 for StatsBase.jl won’t work (IIRC).

1 Like