I broke this question out so as to not clutter this thread. In there, @Lilith mentions how new folk can help with Julia’s CI. I have no experience with CI, but would like to learn.
The steps detailed, in my poor understanding, started with locating a failing test on this page … i.e. look for a red x. I picked one near the top of the page at that instant of time, which winds up being 883c19b. When I click on the red x, I get a drop-down window with a few fails …
I scroll around a little bit and I find a test that fails on line 302 of binaryplatforms.jl. I then did a cursory search in the issues page for things mentioning detect_libgfortran_version and found nothing.
Would I then open up an issue suggesting that this test be skipped or broken?
EDIT: is this even the right place to post this question?
Our CI has several steps, with failures possible at many points. Commit 883c19b had failures in
in the “Build”, "Test, and “Allow Fail” groups. Failures anywhere except for the “Allow Fail” group cause CI to fail and the little red x to appear next to the commit. Failures in the “Allow Fail” group, as the name implies, are acceptable and even expected.
Build failures are tricky—you can’t just skip a step in the build and expect the next step to work. Allow fail failures are fine to have linger around and should typically be fixed by fixing the underlying bug that causes the tests to fail (in this case, the problem is presumably that detect_libgfortran_version fails to detect libgfortran, and the solution is to fix that so it works on that platform).
You found a failure in an “allow fail” group, look for one in the “test” group. The “Test” failures are what we can and should skip to avoid CI failing.
You would open an issue saying that the underlying issue should be fixed (the long term solution is to fix the bugs) and open a pull request skipping the test (we want CI to be green even in the presence of known bugs)
