# Statistical unit tests

**URL:** <https://discourse.julialang.org/t/statistical-unit-tests/94210>\
**Category:** General Usage\
**Tags:** question\
**Created:** [February 7, 2023, 1:41pm UTC](https://discourse.julialang.org/t/statistical-unit-tests/94210 "2023-02-07T13:41:06Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![prittjam](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/prittjam/32/21267_2.png) [@prittjam](https://discourse.julialang.org/u/prittjam)\
**Post date:** [February 7, 2023, 1:41pm UTC](https://discourse.julialang.org/t/statistical-unit-tests/94210/1 "2023-02-07T13:41:06Z")

</div>

Are there macros for statistical unit tests for randomized algorithms? E.g., if I’d like to test the mean value of an algorithm over 100 runs, e.g… So a macro similar to @btime that would run the method many times and summarize the result.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [February 7, 2023, 1:44pm UTC](https://discourse.julialang.org/t/statistical-unit-tests/94210/2 "2023-02-07T13:44:30Z")

</div>

Have not seen anything like this, but wouldn’t it just be

```julia
@test mean(my_algo(x, y, z) for _ in 1:1_000) ≈ test_value

```

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [February 7, 2023, 1:47pm UTC](https://discourse.julialang.org/t/statistical-unit-tests/94210/3 "2023-02-07T13:47:01Z")

</div>

On the coding side, I agree something like this is the right approach.

But statistically you really want to check that your mean is within some high-probability interval around the true parameter since randomness implies your result is inaccurate by more than just numerical approximation.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [February 7, 2023, 1:54pm UTC](https://discourse.julialang.org/t/statistical-unit-tests/94210/4 "2023-02-07T13:54:00Z")

</div>

Yes I guess it would be more like

```julia
@test mean(my_algo(x, y, z) for _ in 1:1_000) ≈ test_value atol=0.5

```

with an appropriately chosen tolerance level that reflects the statistical properties rather than just numerical error as you say.

---

<div class="post-metadata">

**Author:** ![prittjam](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/prittjam/32/21267_2.png) [@prittjam](https://discourse.julialang.org/u/prittjam)\
**Post date:** [February 7, 2023, 1:54pm UTC](https://discourse.julialang.org/t/statistical-unit-tests/94210/5 "2023-02-07T13:54:41Z")

</div>

yes, a hypothesis test would be ideal. I wondered if there were some convenience macros. Might be a useful extension.

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [February 7, 2023, 2:11pm UTC](https://discourse.julialang.org/t/statistical-unit-tests/94210/6 "2023-02-07T14:11:03Z")

</div>

Adding a `@clt_test` macro could be useful. Maybe start a package?

I would personally start by creating a function here since it’s not clear to me how often you’d need to test expressions rather than evaluate tests against a distribution object. For example, I can imagine something like:

```julia
clt_test(d, n_samples=1_000, probtol=1e-6)
# Checks that mean(rand(d, n)) in (lower, upper) based on probability of error 1e-6

```

---

<div class="post-metadata">

**Author:** ![prittjam](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/prittjam/32/21267_2.png) [@prittjam](https://discourse.julialang.org/u/prittjam)\
**Post date:** [February 7, 2023, 3:03pm UTC](https://discourse.julialang.org/t/statistical-unit-tests/94210/7 "2023-02-07T15:03:35Z")

</div>

well, maybe one package at a time 🙂 But I think such a macro is perfect for randomized algorithms and algorithms expecting stochastic data inputs.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [February 9, 2023, 9:14am UTC](https://discourse.julialang.org/t/statistical-unit-tests/94210/8 "2023-02-09T09:14:40Z")

</div>

This PR [[WIP] Simulation tests for coverage + p-values by johnmyleswhite · Pull Request #73 · JuliaStats/HypothesisTests.jl · GitHub](https://github.com/JuliaStats/HypothesisTests.jl/pull/73) may be of interest: an attempt to add statistical tests for `HypothesisTests.jl`.