I noticed that the continuous integration builds are frequently failing on GitHub: master builds · julia-master. I flipped through the last 20 pages there and saw a significant percentage of failed builds.
Further, even in the builds that pass, it seems there are failed tests. For example, in the “coverage” box for this build, I see failed tests: julia-master #12619.
What’s going on here? I have no professional experience in software development, but my amateur understanding is that failing tests ought to make the build fail, and failed builds should be rare. Probably there’s a good reason things are they way they are, I’m just curious what that is.
(Also, are questions like this better for the Slack?)
For the Coverage job specifically, it is known that some tests are expected to fail when code coverage emission is enabled. So we allow the Coverage job to pass even if some of the tests don’t pass. This is perfectly fine; the purpose of the Coverage job is not to determine whether or not the tests passed, but rather to determine the code coverage.
2 Likes
On the other hand, the purpose of the Test group is indeed to know whether or not our tests are passing, so failures of jobs in the Test group is bad. As you noticed, we have several non-deterministic failures that occur in jobs within the Test group. We’re working on fixing those, and eventually we’d like Buildkite to be consistently green on master, but we’re not there yet.
If anyone is interested in helping out, please join us in the #ci-dev channel on Slack.
3 Likes