What we do in DataFrames.jl (and plan to as I think it is a good practice):
- Run tests against Julia nightly to catch issues in Julia Base (we detect them indeed, this is not just a theoretical reason)
- Set up CI not to report failure on Julia nightly as failure (i.e. it is run, we see the result, but it is not considered
)