Useful in writing deterministic tests
@testset "SkewT fit_mle" begin
Random.seed!(0) do
x = rand(SkewT(1.0, 2.0, 5.0, 0.5), 10_000);
(; μ, σ, ν, λ) = fit_mle(SkewT, x)
@test [μ, σ, ν, λ] ≈ [1.0, 2.0, 5.0, 0.5] rtol=0.05
end
end
Useful in writing deterministic tests
@testset "SkewT fit_mle" begin
Random.seed!(0) do
x = rand(SkewT(1.0, 2.0, 5.0, 0.5), 10_000);
(; μ, σ, ν, λ) = fit_mle(SkewT, x)
@test [μ, σ, ν, λ] ≈ [1.0, 2.0, 5.0, 0.5] rtol=0.05
end
end
That’s already in Julia v1.12 for testsets: www.julialang.org/blog/2025/10/julia-1.12-highlights.md at b2f7dd210cc400cf29a2bb5525644f995286c947 · JuliaLang/www.julialang.org · GitHub
If I understand correctly - the doc you mentioned say if test is non deterministic, it would print the random number state if test failed, to reproduce it.
Wile this change makes the test deterministic.
You missed the bit where you can pass a custom RNG to @testset
Normally, it’d require several attempts to reproduce a similar failure, but now the RNG is printed to screen and you can reproduce the run in a new session by setting the
rng
option of@testset
to the value printed in the failed test:julia> using Test, Random julia> @testset rng=Random.Xoshiro(0xd02e9404e1026b37, 0xca5ae9c15acf6752, 0x976a327d42433534, 0xb5b1305af1734f3a, 0x1c2aa037d6e7d5c7) begin @test rand() > 0.001 end; test set: Test Failed at REPL[2]:2 Expression: rand() > 0.001 Evaluated: 0.00036328334842516963 > 0.001 Stacktrace: [1] top-level scope @ REPL[2]:2 [2] macro expansion @ ~/.julia/juliaup/julia-1.12.0.x64.linux.gnu/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined] [3] macro expansion @ REPL[2]:2 [inlined] [4] macro expansion @ ~/.julia/juliaup/julia-1.12.0.x64.linux.gnu/share/julia/stdlib/v1.12/Test/src/Test.jl:680 [inlined] Test Summary: | Fail Total Time test set | 1 1 1.4s RNG of the outermost testset: Xoshiro(0xd02e9404e1026b37, 0xca5ae9c15acf6752, 0x976a327d42433534, 0xb5b1305af1734f3a, 0x1c2aa037d6e7d5c7) ERROR: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.
Also note that hardcoding an RNG like that is not a good practice, and should be reserved for reproducing an existing failure. As the RNG docs say:
Relying on a specific seed or generated stream of numbers during unit testing is thus discouraged - consider testing properties of the methods in question instead.
Hardcoding certain results can and does hide issues in your code, and is not guaranteed to be stable.
Shameless plug, consider using property based testing libraries like Supposition.jl for this.
Yes, the way I envisioned the rng
option for @testset
was only to reproduce observed failures which only depend on the global RNG, because that happens a lot and reproducing failures observed on remote machines can be frustratingly hard (especially because @testset
does know the RNG state but doesn’t tell you ). But if someone for some reason wants to have fully deterministic runs they should consider passing the RNG object explicitly to
rand
and friends (which is also faster than accessing the global RNG, BTW), or even better test statistical properties of distributions rather than exact numbers, as you also suggested.
Ok, feel free to close the issue.
P.S. I think the tests should be deterministic, predictable and reproducible.