# Automation to ensure green CI on master

**URL:** <https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616>\
**Category:** Internals & Design\
**Created:** [August 8, 2023, 8:50pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616 "2023-08-08T20:50:57Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [August 8, 2023, 8:50pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/1 "2023-08-08T20:50:57Z")

</div>

CI on master is sometimes failing, which makes it harder for new contributors to make PRs because they don’t have a good signal about success/failure. Some projects like Rust ensure that the [master branch of rust-lang/rust is always in a valid state](https://forge.rust-lang.org/infra/docs/rustc-ci.html).

Are devs opposed to such a system or has nobody got around to setting up the automation or is there a budget problem?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [August 8, 2023, 8:58pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/2 "2023-08-08T20:58:33Z")

</div>

The problem with this is that our test suite is slightly non-deterministic so there are times where someone merges a PR that was green by chance that then ends up having a bug that is detected in say 10% of CI runs which would then block a lot of other PRs from merging while we identify the PR that broke things and fix it.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [August 8, 2023, 9:00pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/3 "2023-08-08T21:00:24Z")

</div>

How (if at all) does rust manage to evade spurious failures? Do we know what kinds of non-determinism in our testsuite often leads to these failures?

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [August 8, 2023, 10:07pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/4 "2023-08-08T22:07:02Z")

</div>

Is there a good way to identify tests with nondeterministic outcomes?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 8, 2023, 11:21pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/5 "2023-08-08T23:21:26Z")

</div>

> [@Sukera](#):
>
> How (if at all) does rust manage to evade spurious failures?

example of commits to main branch that failed CI

> <https://github.com/rust-lang/rust/commit/20c25d6c31204331787332df49987c8ea9c7c08e>

> <https://github.com/rust-lang/rust/commit/09c71a55470b4750f29d3d8c3afe971a5713ec21>

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [August 8, 2023, 11:50pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/6 "2023-08-08T23:50:32Z")

</div>

From talking with some rustaceans I think that’s not really in master. My link above shows Rust has a system of trying multiple commits at once to save time, and in this case one of the commits in a group failed, but the final “rollup” merge commit for that group was successful [Auto merge of #114565 - matthiaskrgr:rollup-p7cjs3m, r=matthiaskrgr · rust-lang/rust@72c6b8d · GitHub](https://github.com/rust-lang/rust/commit/72c6b8d36fa27207fb66b2d8386ad959aa10e6bb)

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [August 9, 2023, 12:11am UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/7 "2023-08-09T00:11:33Z")

</div>

> [@Sukera](#):
>
> How (if at all) does rust manage to evade spurious failures?

I think a good philosophy is described in this issue from a (non-rust but very good) async library [https://github.com/python-trio/trio/issues/200](https://github.com/python-trio/trio/issues/200) , namely applying heavy effort to track down and eliminate each flaky test (and how to do it).

---

<div class="post-metadata">

**Author:** ![e3c6](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@e3c6](https://discourse.julialang.org/u/e3c6)\
**Post date:** [August 9, 2023, 12:31am UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/8 "2023-08-09T00:31:57Z")

</div>

> [@Oscar\_Smith](#):
>
> our test suite is slightly non-deterministic

What aspect of the CI tests are non-deterministic?

Would it be worthwhile to make all tests deterministic?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [August 9, 2023, 2:12am UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/9 "2023-08-09T02:12:22Z")

</div>

There are a number of non-deterministic aspects. Some are intentional (e.g. math tests using random numbers to increase code coverage over time), some are more inherent/annoying (e.g. file state, internet connection issues). The second set are definitely good to remove but the first possibly should stay.

---

<div class="post-metadata">

**Author:** ![viralbshah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viralbshah/32/54_2.png) [@viralbshah](https://discourse.julialang.org/u/viralbshah)\
**Post date:** [August 9, 2023, 2:16am UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/10 "2023-08-09T02:16:39Z")

</div>

One could imagine two sets of tests - with the fully deterministic set never allowed to fail, with stringent things like CI having to pass before merge is allowed.

---

<div class="post-metadata">

**Author:** ![woclass](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/woclass/32/212699_2.png) [@woclass](https://discourse.julialang.org/u/woclass)\
**Post date:** [August 9, 2023, 3:06am UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/11 "2023-08-09T03:06:16Z")

</div>

> [@viralbshah](#):
>
> One could imagine two sets of tests

There are two ways to divide those test sets:

- split into two phases of execution
- split into two separate CI tasks

If divided into two execution phases: simpler to implement, perhaps just modify `test/runtests.jl`.  
The deterministic test is executed first, and if it fails, the whole test fails. After passing the deterministic test, continue executing the other tests.  
If the CI platform does not support returning multiple states, you may need to manually check the test results of the latter phase.

If you choose to split tests: the number of tests to be run doubles, but you can get a better view of the status of each type of test run.

---

<div class="post-metadata">

**Author:** ![e3c6](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@e3c6](https://discourse.julialang.org/u/e3c6)\
**Post date:** [August 9, 2023, 1:25pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/12 "2023-08-09T13:25:07Z")

</div>

> [@woclass](#):
>
> the number of tests to be run doubles

Not sure what you mean?

---

<div class="post-metadata">

**Author:** ![e3c6](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@e3c6](https://discourse.julialang.org/u/e3c6)\
**Post date:** [August 9, 2023, 1:31pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/13 "2023-08-09T13:31:19Z")

</div>

> [@Oscar\_Smith](#):
>
> math tests using random numbers to increase code coverage over time

What is rust stance on non-deterministic tests? A very superficial search seems to indicate they don’t have them but I couldn’t find any reliable source.

---

<div class="post-metadata">

**Author:** ![sbuercklin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sbuercklin/32/15728_2.png) [@sbuercklin](https://discourse.julialang.org/u/sbuercklin)\
**Post date:** [August 9, 2023, 1:47pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/14 "2023-08-09T13:47:19Z")

</div>

> (e.g. math tests using random numbers to increase code coverage over time)

How does using an RNG which can lead to random failures improve test coverage if the first solution to failures is to run the test suite again and hope the RNG plays nicely this time around?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [August 9, 2023, 1:58pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/15 "2023-08-09T13:58:47Z")

</div>

theoretically at least we only rerun tests after looking to see what failed and if it’s potentially real

---

<div class="post-metadata">

**Author:** ![rfourquet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rfourquet/32/3610_2.png) [@rfourquet](https://discourse.julialang.org/u/rfourquet)\
**Post date:** [August 9, 2023, 2:00pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/16 "2023-08-09T14:00:24Z")

</div>

> [@sbuercklin](#):
>
> How does using an RNG which can lead to random failures improve test coverage if the first solution to failures is to run the test suite again and hope the RNG plays nicely this time around?

Hopefully in this case the preferred solution is to use the failure as a bug report and try to fix it. This class of failure is generally easy to reproduce by setting the RNG seed.

---

<div class="post-metadata">

**Author:** ![sbuercklin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sbuercklin/32/15728_2.png) [@sbuercklin](https://discourse.julialang.org/u/sbuercklin)\
**Post date:** [August 9, 2023, 2:14pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/17 "2023-08-09T14:14:31Z")

</div>

Is there a list of tests which are RNG-unstable?

---

<div class="post-metadata">

**Author:** ![gbaraldi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gbaraldi/32/22101_2.png) [@gbaraldi](https://discourse.julialang.org/u/gbaraldi)\
**Post date:** [August 9, 2023, 2:21pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/18 "2023-08-09T14:21:27Z")

</div>

The rng tests aren’t the ones that cause failures, by far the most common failure is network tests.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [August 9, 2023, 2:27pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/19 "2023-08-09T14:27:01Z")

</div>

Would it help if all network tests had their own ci job so one could immediately see that failures are unrelated, and one could run only that one again manually?

---

<div class="post-metadata">

**Author:** ![Krastanov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/krastanov/32/6817_2.png) [@Krastanov](https://discourse.julialang.org/u/Krastanov)\
**Post date:** [August 9, 2023, 2:43pm UTC](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616/20 "2023-08-09T14:43:33Z")

</div>

This discussion is great and I am learning a lot from it. Some of the questions and suggestions are quite pertinent/insightful. **I would urge the folks spearheading this questioning to document what is being learnt and to even make pull requests with their suggestions (even if the pull request is draft and incomplete) in order to keep the ball rolling** – otherwise we will just have a long thread that becomes outdated. The core devs have not done this not because they do not agree it is valuable but because their TODO lists of valuable work is 10x longer than what they have the time to do.

[Next page](https://discourse.julialang.org/t/automation-to-ensure-green-ci-on-master/102616.md?page=2)