Chaos compatibility testing?

It sounds like you’re asking for compatibility fuzz testing. I don’t know of a CI script for that but it sounds reasonable. However, I’d suggest using a form of Downgrade CI to more directly do this rather than random sampling, see:

2 Likes