# Nightly build CI failing because rand(UnitRange) changed (when to care?)

**URL:** <https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857>\
**Category:** New to Julia\
**Created:** [November 9, 2020, 5:35pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857 "2020-11-09T17:35:28Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 9, 2020, 5:35pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/1 "2020-11-09T17:35:28Z")

</div>

The tests of a package of mine are failing with the nightly build of Julia, and I have found out that the reason is that the behavior of the random number generator changed when it is called with a range.

I have two questions:

1. Should I care about nightly builds failing at all? Is it likely that the behavior will return back to what it was previously so that I should wait for the final release of 1.6 to even bother with that?

2. Is there any option to guarantee among different releases that the exact same random number generator is used (I understand that having the same behavior in general is not guaranteed, but there could be an option to be used just for testing).

Thank you.

The output of `rand` in different versions is shown bellow:

Nightly:

```julia-repl
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.6.0-DEV.1451 (2020-11-09)
 _/ |\ __'_|_|_|\__'_| | Commit d562a97f2b (0 days old master)
|__/ |

julia> import Random

julia> Random.seed!(1);

julia> rand()
0.23603334566204692

julia> rand(1:10)
6

```

1.5.2:

```julia-repl
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.5.2 (2020-09-23)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

julia> import Random

julia> Random.seed!(1);

julia> rand()
0.23603334566204692

julia> rand(1:10)
10

```

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [November 9, 2020, 5:44pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/2 "2020-11-09T17:44:27Z")

</div>

Use StableRNGs.jl. This package was written with great care, specifically to solve this sort of problem.

You can find the announcement of this package here on discourse and I’ve used it a lot in my applications.

---

<div class="post-metadata">

**Author:** ![rfourquet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rfourquet/32/3610_2.png) [@rfourquet](https://discourse.julialang.org/u/rfourquet)\
**Post date:** [November 9, 2020, 5:49pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/3 "2020-11-09T17:49:20Z")

</div>

> [@lmiq](#):
>
> Is it likely that the behavior will return back to what it was previously

It is very unlikely! It might “return back” if a bug is found in the change leading that affected the random streams, and no fix is found fast enough.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [November 9, 2020, 5:59pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/4 "2020-11-09T17:59:37Z")

</div>

> [@lmiq](#):
>
> Is it likely that the behavior will return back to what it was previously so that I should wait for the final release of 1.6 to even bother with that?

The exact sequence of pseudo-random numbers produced from a given seed is explicitly _not_ guaranteed to be reproducible across Julia versions, so you shouldn’t rely on this. See the [manual on random-number reproducibility](https://docs.julialang.org/en/v1/stdlib/Random/#Reproducibility).

If you need to reproduce certain data for tests, the simplest solution is to save the data. Other options, besides switching to a different random-number package, are discussed in the manual.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 9, 2020, 6:07pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/5 "2020-11-09T18:07:08Z")

</div>

> [@stevengj](#):
>
> simplest solution is to save the data.

In my case the problem is not generating the data, is that the problem to be solved involves generating random coordinates of particles (since you are a physicist: to generate a reference state for a molecular system). The generation of these coordinates and the computation I have to do with them are quite costly, so I cannot rely on obtaining an exhaustive sampling to any reasonable precision during testing (the tests would take too much time). Therefore, I only generate a small sample and compare the rough results obtained with the expected result for that small sample. Of course, without being able to achieve any reasonable precision in that numerical integration with this small sample, I have to reproduce the exact same random coordinates to know that the package is doing always the same thing.

> [@stevengj](#):
>
> The exact sequence of pseudo-random numbers produced from a given seed is explicitly _not_ guaranteed to be reproducible across Julia versions

I was aware of that. That is why I was either waiting to choose what to do when 1.6 arrives, or searching for an alternative to write the tests.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [November 9, 2020, 6:35pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/6 "2020-11-09T18:35:42Z")

</div>

> [@lmiq](#):
>
> Therefore, I only generate a small sample and compare the rough results obtained with the expected result for that small sample. Of course, without being able to achieve any reasonable precision in that numerical integration with this small sample, I have to reproduce the exact same random coordinates to know that the package is doing always the same thing.

Yes, so save the random coordinates of the small test sample. This is the “data” I was referring to — the _input_ data.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 9, 2020, 6:36pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/7 "2020-11-09T18:36:49Z")

</div>

Then I would have to modify the package to read that data, that data is not saved anywhere in an actual run, the coordinates are generated on-the-flight and discarded. (Indeed, those random coordinates are not _input_ data).

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [November 9, 2020, 6:39pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/8 "2020-11-09T18:39:18Z")

</div>

> [@lmiq](#):
>
> Then I would have to modify the package to read that data, that data is not saved anywhere in an actual run, the coordinates are generated on-the-flight and discarded.

If you don’t have a simple API to pass the sample coordinates in externally, I agree that it is trickier. You could write a new RNG that read its inputs from a file rather than generating it, but in that case it is probably easier to use something like the StableRNGs.jl package.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 9, 2020, 6:40pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/9 "2020-11-09T18:40:27Z")

</div>

Indeed, in this case even in the tests we are talking of tenths of thousands of random numbers. StableRNG seems to be the way to go.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 9, 2020, 7:02pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/10 "2020-11-09T19:02:22Z")

</div>

Uhm… but there is something tricky here.

I was using simply `rand()` to generate random numbers, but to add the possibility of using optionally `StableRNGs` I have to use `rand(rng)`. I changed the code to do that, but the problem is that now it allocates memory:

```julia
julia> import Random

julia> rand()
0.30815646522353957

julia> @allocated rand()
0

julia> rng = Random.MersenneTwister(1);

julia> rand(rng)
0.23603334566204692

julia> @allocated rand(rng)
16

```

Thus, to have the option of passing a different `rng ` to `rand()` for some reason it allocates memory even if the option is the default one (and, as I mentioned, I need to generate tenths of thousands of random numbers, so this is really an issue, the code was free from allocations without that).

Edit: Well if I declare `rng` as constant, that allocation goes away. I will see if I can adapt that to my case.

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [November 9, 2020, 7:50pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/11 "2020-11-09T19:50:15Z")

</div>

Just wrap everything in function or in a let block.

```julia
let rng = Random.MersenneTwister(1)
    @allocated rand(rng)
end

```

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 9, 2020, 7:54pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/12 "2020-11-09T19:54:41Z")

</div>

It is a little bit more complicated here, I think. I created a new thread:

> [@Defining const inside struct](https://discourse.julialang.org/t/defining-const-inside-struct/49866):
>
> This is a follow-up of this thread, which ended with this new issue I am not sure how to solve: [https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/10](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/10) The issue is the following: The usage of my package starts with the user defining a series of options which are organized in a struct. Now I want to add to that struct the possibility of the user defining which is the random number generator to be used. I am using Parameters, so I have def…

There may be a smart solution which I am missing, though.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 10, 2020, 9:11am UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/13 "2020-11-10T09:11:47Z")

</div>

> [@lmiq](#):
>
> There may be a smart solution which I am missing, though.

As @stevengj suggested above, it would be best to decouple the deterministic parts from the calculation from the random input generation, and save those “random” inputs and reuse them.

While this may involve some refactoring of the code, when feasible it is the best solution. StableRNGs.jl will guarantee the same random stream, but if your calculations involve floating point, they are not necessarily bit-by-bit reproducible across machines, CPUs, and Julia releases either.

---

<div class="post-metadata">

**Author:** ![rfourquet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rfourquet/32/3610_2.png) [@rfourquet](https://discourse.julialang.org/u/rfourquet)\
**Post date:** [November 10, 2020, 9:48am UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/14 "2020-11-10T09:48:15Z")

</div>

> [@Tamas\_Papp](#):
>
> calculation from the random input generation, and save those “random” inputs and reuse them

It sounds like you suggest to save the random inputs, not the calculation from the random inputs…

> [@Tamas\_Papp](#):
>
> but if your calculations involve floating point, they are not necessarily bit-by-bit reproducible across machines, CPUs, and Julia releases either.

… but this suggests that it’s best to save the result of the calculations themselves.

Sorry, I’m not familiar with this problem setting, would you mind expanding a bit on this?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 10, 2020, 10:42am UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/15 "2020-11-10T10:42:07Z")

</div>

> [@rfourquet](#):
>
> It sounds like you suggest to save the random inputs, not the calculation from the random inputs…

To be specific, I am suggesting that something like

```julia
function f(rng, x)
    y = rand(rng)
    # ... some complex calculation involving x and y
end

```

is refactored to

```julia
function some_complex_calculation(x, y)
    ...
end

```

which is deterministic and can be tested as such.

> [@rfourquet](#):
>
> would you mind expanding a bit on this?

Eg topics like

> [@Different Float64 sum on different architectures](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949/):
>
> I sometimes get different results for summing a vector of Float64 numbers on different Windows machines. The vectors are identical, so it seems sum can produce different results across hardware architectures (even if os is always Windows). Is this expected? Is there a way to guarantee the same results? Would sum\_kbn from KahanSummation, or xsum from Xsum guarantee the same result? Both of these alternative sums produce identical results across machines on my particular data. [https://github…](https://github.com/JuliaMath/KahanSummation.jl)

The issue is nearly impossible to avoid for reasonably complex calculations.

---

<div class="post-metadata">

**Author:** ![rfourquet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rfourquet/32/3610_2.png) [@rfourquet](https://discourse.julialang.org/u/rfourquet)\
**Post date:** [November 10, 2020, 10:54am UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/16 "2020-11-10T10:54:35Z")

</div>

> [@Tamas\_Papp](#):
>
> which is deterministic and can be tested as such.

Oh OK, seems right.

But I don’t understand how from this you derive that it’s best to “save thoses random inputs”, rather than using an RNG to generate them in a reproducible way… or maybe it’s not what you meant?

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 10, 2020, 11:51am UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/17 "2020-11-10T11:51:34Z")

</div>

In my case that option is not reasonable. The number of random numbers is too large, much larger than any of the input data I have to provide to the function. The best analogy is that of a Monte-Carlo algorithm for integration of a irregular volume in space, defined by the distance to a set of points. For instance, this code computes the volume of the region comprised within a `cutoff` distance of a set of points in space defined in `data`:

```julia

julia> function volume(data,cutoff,samples)
         ns = 0
         for i in 1:samples
            x = -10 .+ 20*rand(3)
            for j in 1:length(data)
              d = 0.
              for k in 1:3
                 d += (data[j][k]-x[k])^2 
              end
              if sqrt(d) < cutoff
                ns += 1
                continue
              end
            end
          end
          volume = (ns/samples)*(20^3)
          return volume
       end

julia> data = [rand(3) for i in 1:10];

julia> cutoff = 5.;

julia> samples=10^5;

julia> volume(data,cutoff,samples)
5280.32

```

How can I test this without a stable random number generator?

(edited to fix the code to make it work, please do not mind about the performance of the above, it is just an example)

What I am doing in the testing of my package is using a small number of samples, with a random number generator with predictable results, and comparing the result with a previously computed result which I know is correct. In an actual run of the package the number of samples is large and one can expect to obtain a good precision on that volume estimate, but for testing this takes too long.

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [November 10, 2020, 1:08pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/18 "2020-11-10T13:08:14Z")

</div>

By the way (and it is related to your other questions, I suppose), the usual strategy that I use in functions like this is

```julia
function volume(data, cutoff, samples; rng = Random.GLOBAL_RNG)
    ns = 0
    for i in 1:samples
        x = -10 .+ 20*rand(rng, 3)
...

```

This way users can use your function without worrying about RNG, but you can test function by supplying RNG from custom package (yes, like `StableRNGs.jl` 🙂 )

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 10, 2020, 1:09pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/19 "2020-11-10T13:09:11Z")

</div>

> [@rfourquet](#):
>
> how from this you derive that it’s best to “save thoses random inputs”, rather than using an RNG to generate them in a reproducible way

It really depends on the structure of those inputs — if there is a lot of them, then generating with a stable RNG may be the best option. Hard to say without context.

> [@lmiq](#):
>
> How can I test this without a stable random number generator?

Thanks for providing an MWE — it is much easier to discuss concrete code. I would refactor as

```julia
function volume(data, cutoff, xs; K = length(first(xs)))
  ns = 0
  for x in xs
     for j in 1:length(data)
       d = 0.
       for k in 1:K
          d += (data[j][k]-x[k])^2
       end
       if sqrt(d) < cutoff
         ns += 1
         continue
       end
     end
   end
   volume = (ns/length(xs))*(20^3)
   return volume
end

```

and save the `xs`. Even if their length is small, this should be enough to test that the code _works_, without aiming for accuracy — you simply reproduce the calculation some other way. Also, when (in the original code) you do

```julia
julia> volume(data,cutoff,samples)
5280.32

```

presumably you are comparing to a theoretically calculated value with some error bounds. Then you run into the classic statistical problem of making a trade-off between type I and type II errors. In practice, for very complex calculations and without tests tailored to the particular problem, this either means that you get false alarms in CI or tests that are not very meaningful.

Of course what people like to avoid with reproducible random numbers is the above, usually this means that they get a result, eyeball it, and then hardcode the bounds that make it pass into CI. But this is very brittle.

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [November 10, 2020, 1:13pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/20 "2020-11-10T13:13:31Z")

</div>

I guess it makes sense to add convenience function for end users

```julia
function volume(data, cutoff, samples::Integer; rng = Random.GLOBAL_RNG)
    xs = map(_ -> -10 .+ 20*rand(rng, 3), 1:samples)
    volume(data, cutoff, xs)
end

```

[Next page](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857.md?page=2)