Chaos compatibility testing?

[Sukera]

@ChrisRackauckas

Downstream CI is essentially a fallback for missing test coverage or interface specification to catch the core users misalignment with the test suite. That quick fix for coverage is why it’s part of the status quo.

Yes, I understand that. I’m saying that downstream CI doesn’t catch the case with PkgF & PkgA in the same environment, because PkgC testing all possible combinations of downstream packages is not really feasible.

@ChrisRackauckas

In the cases you’re describing, either PkgB or PkgD would have to see failures in their downstream CI or they would work together.

I don’t understand this. In the situation with PkgF, there isn’t necessarily a package where the combination of PkgA-F are all in the same environment; it can just as well be an unnamed project on someones PC. How could downstream CI of either PkgB or PkgD catch this case, when they have no knowledge of PkgF even existing for their (reverse) dependency chain? Are you saying that PkgC should install all its reverse dependencies into the same environment at once for downstream CI…?

@ChrisRackauckas

This at least makes sure latest is always working (unless they both happen to merge and tag within the same half hour).

Yes, but again, the latest versions of PkgA & PkgF in isolation working with the most recent versions of PkgB/PkgD and PkgG respectively doesn’t really tell you anything about the combination of PkgA & PkgF in the same environment also working, unless you assume that newer versions always have strictly fewer bugs (which I think is unrealistic) and that every package has a perfect track record when it comes to bumping their lower bounds (which would again require perfect test coverage of PkgC to do reliably).

[MilesCranmer]

Thanks everyone. I actually didn’t know downgrade CI existed and it looks great, will definitely try it out. Indeed it would not be an exhaustive solution because other packages in an environment might pin a dependency to a specific version, so I would still like a way to test all possible compat bounds for the entire dependency tree (randomly, of course – testing all of them exhaustively would be practically impossible).

But to first order downgrade CI looks like it would get 90% of the way there so I will try that!