# Unit tests for packages that use random number generation that are robust to version changes

**URL:** <https://discourse.julialang.org/t/unit-tests-for-packages-that-use-random-number-generation-that-are-robust-to-version-changes/14477>\
**Category:** General Usage\
**Created:** [September 3, 2018, 12:06pm UTC](https://discourse.julialang.org/t/unit-tests-for-packages-that-use-random-number-generation-that-are-robust-to-version-changes/14477 "2018-09-03T12:06:53Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![colintbowers](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/colintbowers/32/8033_2.png) [@colintbowers](https://discourse.julialang.org/u/colintbowers)\
**Post date:** [September 3, 2018, 12:06pm UTC](https://discourse.julialang.org/t/unit-tests-for-packages-that-use-random-number-generation-that-are-robust-to-version-changes/14477/1 "2018-09-03T12:06:53Z")

</div>

Hi all,

I just got caught out by the change in `rand` and `MersenneTwister` between v0.6 and v0.7 (my own fault - I didn’t read the release notes carefully enough).

However, this raised a deeper issue for me. If one has a package that is inherently dependent on random number generation (in my case this is my DependentBootstrap package), what is the best way to write unit tests to go in runtests.jl?

Until now, I used a call to `srand(1234)` (`Random.seed!(1234)` in v0.7) to get consistent runs with known results for testing purposes, but this is what caught me out in the upgrade. Is there a smarter way to deal with this? Or is the best solution simply to use `seed!` and to read the release notes carefully and upgrade your unit-tests if the behaviour of random number generation changes between upgrades?

Cheers,

Colin

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [September 3, 2018, 12:10pm UTC](https://discourse.julialang.org/t/unit-tests-for-packages-that-use-random-number-generation-that-are-robust-to-version-changes/14477/2 "2018-09-03T12:10:26Z")

</div>

The latter is best - as far as I know, RNG is not guaranteed to give the same results between versions. It would be impossible to fix bugs affecting it. Always make sure though that there are changes to RNG announced when your expected result changes, before fixing your unit test - otherwise, a different result may be a bug.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 3, 2018, 12:43pm UTC](https://discourse.julialang.org/t/unit-tests-for-packages-that-use-random-number-generation-that-are-robust-to-version-changes/14477/3 "2018-09-03T12:43:58Z")

</div>

> [@colintbowers](#):
>
> Or is the best solution simply to use `seed!` and to read the release notes carefully and upgrade your unit-tests if the behaviour of random number generation changes between upgrades?

I did this when updating DynamicHMC.jl.

In the ideal case, I construct my stochastic tests to have minimal (say 10^{-4} or similar) probability of [Type I errors](https://en.wikipedia.org/wiki/Type_I_and_type_II_errors#Type_I_error), yet still retain power, but sometimes there are trade-offs. I think that using a fixed random seed and going through the tests every time the RNG changes has benefits, I actually caught a few misspecified tests that way.

Also, keep in mind that even if you set the RNG seed, results may be subtly different over long calculations, with different CPUs, compilation and optimization settings.

Finally, I think that `@testset` resets the random seed after it is done for the global RNG, so it is very useful.

---

<div class="post-metadata">

**Author:** ![colintbowers](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/colintbowers/32/8033_2.png) [@colintbowers](https://discourse.julialang.org/u/colintbowers)\
**Post date:** [September 3, 2018, 10:41pm UTC](https://discourse.julialang.org/t/unit-tests-for-packages-that-use-random-number-generation-that-are-robust-to-version-changes/14477/4 "2018-09-03T22:41:06Z")

</div>

Thanks for responding. Based on what you and Tamas have said, I think I’ll keep using RNG and just be a bit more careful in the future.

---

<div class="post-metadata">

**Author:** ![colintbowers](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/colintbowers/32/8033_2.png) [@colintbowers](https://discourse.julialang.org/u/colintbowers)\
**Post date:** [September 3, 2018, 10:44pm UTC](https://discourse.julialang.org/t/unit-tests-for-packages-that-use-random-number-generation-that-are-robust-to-version-changes/14477/5 "2018-09-03T22:44:20Z")

</div>

Thanks for responding.

That’s quite a neat idea to construct the tests such that you can say with high certainty what interval the answer will lie on. I’ll have a think about it, although it may prove more work than using RNG and just being careful. Point taken about different CPU’s.