I had a similar problem with DifferentiationInterface.jl, where I need to test compatibility with a dozen different autodiff packages and pairs thereof. I solved it by creating several test “groups”, defined by an environment variable.
This environment variable is set in CI (possibly using entries from the workflow matrix), and then queried in runtests.jl. You can take a look at the following files: Test.yml, runtests.jl.
Also related: