# Package test (based on shuffle) ok locally and on GitHub CI but failing on PkgEval

**URL:** <https://discourse.julialang.org/t/package-test-based-on-shuffle-ok-locally-and-on-github-ci-but-failing-on-pkgeval/107334>\
**Category:** Package Management\
**Created:** [December 8, 2023, 3:27pm UTC](https://discourse.julialang.org/t/package-test-based-on-shuffle-ok-locally-and-on-github-ci-but-failing-on-pkgeval/107334 "2023-12-08T15:27:18Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [December 8, 2023, 3:27pm UTC](https://discourse.julialang.org/t/package-test-based-on-shuffle-ok-locally-and-on-github-ci-but-failing-on-pkgeval/107334/1 "2023-12-08T15:27:19Z")

</div>

In my package I have a [`partition`](https://github.com/sylvaticus/BetaML.jl/blob/8ec5cf9b0dd52cca0116d25aa5ff593127a48621/src/Utils/Processing.jl#L487) function that randomly partitions a set of arrays into different shares, keeping the order across the various arrays.

I [test it](https://github.com/sylvaticus/BetaML.jl/blob/8ec5cf9b0dd52cca0116d25aa5ff593127a48621/test/Utils_tests.jl#L486C1-L488C55) using on [`StableRNGs.jl`](https://github.com/JuliaRandom/StableRNGs.jl) and I have the test working both locally and on GitHub CI, but when I check the [PkgEval report](https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2023-12/05/BetaML.primary.log) I have:

```julia
** Testing partition()...
Test Failed at /home/pkgeval/.julia/packages/BetaML/r7gn5/test/Utils_tests.jl:458
  Expression: out == [[31 1 51 61; 32 2 52 62], [11 41; 12 42]]
   Evaluated: AbstractMatrix{Int64}[[41 51 11 1; 42 52 12 2], [31 61; 32 62]] == [[31 1 51 61; 32 2 52 62], [11 41; 12 42]]

```

What could make it ending up with a different random stream ?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [December 8, 2023, 3:41pm UTC](https://discourse.julialang.org/t/package-test-based-on-shuffle-ok-locally-and-on-github-ci-but-failing-on-pkgeval/107334/2 "2023-12-08T15:41:07Z")

</div>

You call `Random.shuffle` who’s implementation changed ([unify `shuffle` and `randperm` by ctarn · Pull Request #50318 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/50318)).

---

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [December 8, 2023, 4:01pm UTC](https://discourse.julialang.org/t/package-test-based-on-shuffle-ok-locally-and-on-github-ci-but-failing-on-pkgeval/107334/3 "2023-12-08T16:01:10Z")

</div>

Thanks… this is going to hurt a lot on my tests… by using StableRNG they did assume a certain output…

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [December 8, 2023, 4:10pm UTC](https://discourse.julialang.org/t/package-test-based-on-shuffle-ok-locally-and-on-github-ci-but-failing-on-pkgeval/107334/4 "2023-12-08T16:10:12Z")

</div>

In general, tests like the one you are doing aren’t good because they aren’t really testing anything. Why is `[[31 1 51 61; 32 2 52 62], [11 41; 12 42]]` a good result? All this test is doing is making sure that the function runs without errors, and that no one has changed any of the implementation (rather than that the implementation remains correct). Instead, I would recommend just testing that the no error is thrown (and not checking the result), or testing the statistical properties of the result. To do this, you would (for example) generate 1000 partitions of a length 3 list and test that the number of times you get each partition is approximately the expected number.

---

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [December 8, 2023, 4:33pm UTC](https://discourse.julialang.org/t/package-test-based-on-shuffle-ok-locally-and-on-github-ci-but-failing-on-pkgeval/107334/5 "2023-12-08T16:33:45Z")

</div>

Well, the point is that that specific output has some structural characteristics (e.g. picking up whole columns of the data - I am partitioning on the second dimension, getting a complete partition,…) so it is well more than just check it doesn’t error, and conditional to relying that the random part was deterministic, allowed me to summarize my checks on these characteristics (that I manually validated) with a simple equal. Without this guarantee instead I need to explicitly check each characteristic in the test code… sure it is more appropriate … but longer 😉

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [December 8, 2023, 4:45pm UTC](https://discourse.julialang.org/t/package-test-based-on-shuffle-ok-locally-and-on-github-ci-but-failing-on-pkgeval/107334/6 "2023-12-08T16:45:46Z")

</div>

It’s a little longer, but it also makes it a lot easier to improve things later on since you can change implementations without your tests breaking (which means that you won’t accidentally change your tests in a bad way). Specifically for `partition`, the natural things to test are that it returns a `Vector{Vector{T}}`, and that concatenating and sorting the result yields an identical result to the sorted original list. This test will be ~3 lines rather than 1 line, but it is actually testing a lot more.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [December 8, 2023, 11:35pm UTC](https://discourse.julialang.org/t/package-test-based-on-shuffle-ok-locally-and-on-github-ci-but-failing-on-pkgeval/107334/7 "2023-12-08T23:35:12Z")

</div>

> [@sylvaticus](#):
>
> What could make it ending up with a different random stream ?

This is a bug in StableRNG’s, as it is [documented](https://github.com/JuliaRandom/StableRNGs.jl#stablerngs) that `shuffle` is considered stable. [We did not vendor the Random implementation when adding that guarantee](https://github.com/JuliaRandom/StableRNGs.jl/issues/10#issuecomment-810951669), but need to do so now as the upstream implementation has changed. It is fixed in [Add `shuffle` implementation to preserve stability by ericphanson · Pull Request #17 · JuliaRandom/StableRNGs.jl · GitHub](https://github.com/JuliaRandom/StableRNGs.jl/pull/17), but needs a JuliaRandom maintainer to take a look.