# Nightly build CI failing because rand(UnitRange) changed (when to care?)
**URL:** https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857
**Category:** New to Julia
**Created:** [November 9, 2020, 5:35pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857 "2020-11-09T17:35:28Z")
**Posts on this page:** 9
**Page:** 2
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [November 10, 2020, 1:27pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/21 "2020-11-10T13:27:07Z")
Yes, at the end I ended with something like that, except that the RNG value is not the global one, but one defined inside a function depending on some input parameters (seed is provided or not, reproducible run is desired or not).
> [@Tamas\_Papp](#):
>
> theoretically calculated value with some error bounds
not really, I am comparing with the output of the same calculation in a series of controlled runs, with realistic input. So, yes, I am doing the brittle option probably, but I would feel quite unsafe if the testing was done with toy problems with analytical solution, because there are many many issues than could arise in corner cases of real problems because the actual “shapes” I am integrating are too complicated. For now I think I will stick with this option.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [November 10, 2020, 1:30pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/22 "2020-11-10T13:30:27Z")
> [@lmiq](#):
>
> I would feel quite unsafe if the testing was done with toy problems with analytical solution
Perhaps you misunderstand: the issue is not how you obtain the “true” solution (analytical, or MC runs) you compare to, but how you establish the error bounds for CI.
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [November 10, 2020, 1:38pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/23 "2020-11-10T13:38:03Z")
> [@Tamas\_Papp](#):
>
> how you establish the error bounds for CI
Ah, yes. Well, for the moment the default precision required by `isapprox` seems to be completely safe for the sequential version with the stable random number generator.
Testing the parallel version of the package is a separate issue, where those problems arise more seriously. I am yet to setup a safe testing routine for those runs (while my package has a parallel version which is working quite nice, I do not know yet how to run parallel tests in CI, but just didn’t have time to search for that yet).
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [November 10, 2020, 2:06pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/24 "2020-11-10T14:06:01Z")
> [@lmiq](#):
>
> for the moment the default precision required by `isapprox` seems to be completely safe for the sequential version with the stable random number generator
So you get `√eps` relative precision from a stochastic calculation? That looks suspicious — even for IID draws, you would need a very, very large sample.
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [November 10, 2020, 2:08pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/25 "2020-11-10T14:08:57Z")
> [@Tamas\_Papp](#):
>
> precision from a stochastic calculation?
If the random number sequence is exactly the same, why not?
I mean, this is just a more complicated example of this:
```julia
julia> import Random
julia> Random.seed!(123);
julia> sum(rand() for i in 1:1000)
503.24660050142177
julia> Random.seed!(123);
julia> sum(rand() for i in 1:1000)
503.24660050142177
```
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [November 10, 2020, 2:17pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/26 "2020-11-10T14:17:32Z")
> [@lmiq](#):
>
> If the random number sequence is exactly the same, why not?
Sure, but what are you really testing then? That the same calculation produces the same result? Or is it coded in two different ways, just using the same random stream?
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [November 10, 2020, 3:06pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/27 "2020-11-10T15:06:25Z")
> [@Tamas\_Papp](#):
>
> Or is it coded in two different ways, just using the same random stream?
Generally speaking yes, it is coded different ways. The idea is to have a bunch of tests that assure that whenever I introduce modifications in the package (to improve performance, for example, or new features), I do not break what was working before. Is that different from testing in any other context? Of course some modifications can be breaking in terms of the tests because of these random number sequences, but many (and the most frequent ones) won’t be. So that reassures me that I have not introduced regressions whenever I fix a bug, add some feature, etc.
Edit: but I understand your point. This kind of test is not designed for a major algorithmic modification of the package, for sure, in which case the test should aim the comparison with an expected result with a reasonable precision. I do have some tests of this kind, but they are not part of the automatic test set, because they take too long to run for a safe precision threshold.
For example: if the the `volume` code above I decide to compute `cutoff2 = cutoff^2` and not take the square root of the distance at every iteration of the loop. That saves ~5% of the time and the results are identical. It is good to have quick tests that assure me that I have not done anything wrong when small changes like that are introduced, and that everything continues to work as expected.
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [November 10, 2020, 3:38pm UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/28 "2020-11-10T15:38:45Z")
> [@Tamas\_Papp](#):
>
> Sure, but what are you really testing then?
But since we are there, is there any limitation for CI testing? Can I add a test that takes half an hour to run?
Of course if that is possible I could add tests for which the actual result is compared to the expected precision that a user would expect from the results.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [November 11, 2020, 7:23am UTC](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857/29 "2020-11-11T07:23:25Z")
> [@lmiq](#):
>
> I do not break what was working before. Is that different from testing in any other context?
Generally one would test some invariant that should hold given inputs and outputs.
Hardcoding input/output pairs is occasionally necessary though.
> [@lmiq](#):
>
> is there any limitation for CI testing? Can I add a test that takes half an hour to run?
This depends on your CI setup — most frameworks allow you so set a longer timeout. Of course this will burn up any free tier very quickly.
For some of the economic models I am working on, a CI run takes 2–3 hours. But it is still great because actual estimation takes 1–2 weeks, so catching errors early is valuable. We ended up running CI on our own machine using Gitlab.
[Previous page](https://discourse.julialang.org/t/nightly-build-ci-failing-because-rand-unitrange-changed-when-to-care/49857.md?page=1)