Automation to ensure green CI on master

[tim.holy]

And at least at one point, the second-most common was OS-specific failures. For example, my understanding is that certain aspects of OSX made the FileWatching stdlib likely to have rare failures; these failures are not observed on other platforms. There’s not much we can do when the underlying OS is supposed to pass us events, but doesn’t. IIRC, didn’t you (or someone) just have to relax certain tests on OSX?

[jar1]

Relevant from another thread

@tim.holy

We don’t (deliberately) merge PRs that cause new failures. Most current causes of failures probably passed CI when they were initially submitted, but intermittent issues or hardware changes or subtle interactions with other changes might cause them to (sometimes) fail; if you have enough flaky tests and failure of just one is enough to call the CI run a failure, then CI failures are common.

I’m not saying this is good, but chasing down the causes tends to be hard work and not that many people make this a core mission. If you want to find examples of PRs fixing flaky tests, a subset (probably only a minority) can be found with the “ci” label: Pull requests · JuliaLang/julia · GitHub

Sometimes, though, things get through due to inadequate test coverage and only discovered later.

[jar1]

Proposal 1

In any PR, whenever a test fails for reasons irrelevant to the PR, do not merge before filing an issue labeled test failure referencing the test and the CI failure. Then anybody so inclined can go through the list of issues and decide whether it’s a bad test, bad code-under-test, or no-action-needed CI platform problem.

Thoughts, before I propose it on GitHub?

[gbaraldi]

I think that puts a burden into the PR maker that’s unnecessary. If you see a CI failure that you think is odd open an issue.