Automation to ensure green CI on master

[jar1]

Relevant from another thread

@tim.holy

We don’t (deliberately) merge PRs that cause new failures. Most current causes of failures probably passed CI when they were initially submitted, but intermittent issues or hardware changes or subtle interactions with other changes might cause them to (sometimes) fail; if you have enough flaky tests and failure of just one is enough to call the CI run a failure, then CI failures are common.

I’m not saying this is good, but chasing down the causes tends to be hard work and not that many people make this a core mission. If you want to find examples of PRs fixing flaky tests, a subset (probably only a minority) can be found with the “ci” label: Pull requests · JuliaLang/julia · GitHub

Sometimes, though, things get through due to inadequate test coverage and only discovered later.