If breakages due to using internals are known unknowns, we also have to account for unknown unknowns. Aside from @joa-quim’s examples, there are many instances of packages making relatively benign assumptions about functionality which breaks with a minor Julia update. "failed to start primary task" with Julia 1.9 and nthreads(:interactive) > 0 · Issue #21 · JuliaFolds/FoldsThreads.jl · GitHub is a recent example that comes to mind. Packages which are somewhat sensitive to inference quality (e.g. StaticArrays), subtyping, inlining etc are another.
Now, does that mean everyone needs to test on nightly? No, I think we’re in agreement there. But testing alphas/betas/release candidates when they come out (which to your point, would have a much smaller impact and fewer false positives to look through)? I think packages which are at risk for breakage due to one or more of the aforementioned reasons should do that. My question was whether doing so is even possible at the moment, because my understanding is that GHA will happily run jobs with 1.9rc well after 1.9 stable is released and the yaml format doesn’t provide a lot of flexibility to say “only run this job if there’s a newer pre-release than the current stable release”.
Switching gears, another concern that was touched on above is that the mechanism for checking for bugs in the language (PkgEval) has its own “boy who cried wolf” problem. If the last line of defense is not extremely reliable or frequently kicks in too late in the release/dev cycle, the pragmatic solution is to have a defense in depth approach. That has generally taken the shape of packages running CI on nightly. It would be great to have a more sustainable alternative to this and I’d love to hear how PkgEval could be improved to avoid some of the issues noted in this thread, but my (perhaps incorrect) impression is that there are no silver bullets there.