Add Random.seed!(0) with block to standard library

Internals & Design
1

Useful in writing deterministic tests

@testset "SkewT fit_mle" begin
  Random.seed!(0) do
    x = rand(SkewT(1.0, 2.0, 5.0, 0.5), 10_000);
    (; μ, σ, ν, λ) = fit_mle(SkewT, x)
    @test [μ, σ, ν, λ] ≈ [1.0, 2.0, 5.0, 0.5] rtol=0.05
  end
end
2

That’s already in Julia v1.12 for testsets: www.julialang.org/blog/2025/10/julia-1.12-highlights.md at b2f7dd210cc400cf29a2bb5525644f995286c947 · JuliaLang/www.julialang.org · GitHub

3

If I understand correctly - the doc you mentioned say if test is non deterministic, it would print the random number state if test failed, to reproduce it.

Wile this change makes the test deterministic.

1 Like
4

You missed the bit where you can pass a custom RNG to @testset

Normally, it’d require several attempts to reproduce a similar failure, but now the RNG is printed to screen and you can reproduce the run in a new session by setting the rng option of @testset to the value printed in the failed test:

julia> using Test, Random

julia> @testset rng=Random.Xoshiro(0xd02e9404e1026b37, 0xca5ae9c15acf6752, 0x976a327d42433534, 0xb5b1305af1734f3a, 0x1c2aa037d6e7d5c7) begin
           @test rand() > 0.001
       end;
test set: Test Failed at REPL[2]:2
  Expression: rand() > 0.001
   Evaluated: 0.00036328334842516963 > 0.001

Stacktrace:
 [1] top-level scope
   @ REPL[2]:2
 [2] macro expansion
   @ ~/.julia/juliaup/julia-1.12.0.x64.linux.gnu/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined]
 [3] macro expansion
   @ REPL[2]:2 [inlined]
 [4] macro expansion
   @ ~/.julia/juliaup/julia-1.12.0.x64.linux.gnu/share/julia/stdlib/v1.12/Test/src/Test.jl:680 [inlined]
Test Summary: | Fail  Total  Time
test set      |    1      1  1.4s
RNG of the outermost testset: Xoshiro(0xd02e9404e1026b37, 0xca5ae9c15acf6752, 0x976a327d42433534, 0xb5b1305af1734f3a, 0x1c2aa037d6e7d5c7)
ERROR: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.
5

Also note that hardcoding an RNG like that is not a good practice, and should be reserved for reproducing an existing failure. As the RNG docs say:

Relying on a specific seed or generated stream of numbers during unit testing is thus discouraged - consider testing properties of the methods in question instead.

Hardcoding certain results can and does hide issues in your code, and is not guaranteed to be stable.

Shameless plug, consider using property based testing libraries like Supposition.jl for this.

2 Likes
6

Yes, the way I envisioned the rng option for @testset was only to reproduce observed failures which only depend on the global RNG, because that happens a lot and reproducing failures observed on remote machines can be frustratingly hard (especially because @testset does know the RNG state but doesn’t tell you :enraged_face:). But if someone for some reason wants to have fully deterministic runs they should consider passing the RNG object explicitly to rand and friends (which is also faster than accessing the global RNG, BTW), or even better test statistical properties of distributions rather than exact numbers, as you also suggested.

1 Like
7

Ok, feel free to close the issue.

P.S. I think the tests should be deterministic, predictable and reproducible.