Statistical unit tests

Are there macros for statistical unit tests for randomized algorithms? E.g., if I’d like to test the mean value of an algorithm over 100 runs, e.g… So a macro similar to @btime that would run the method many times and summarize the result.

Have not seen anything like this, but wouldn’t it just be

@test mean(my_algo(x, y, z) for _ in 1:1_000) ≈ test_value

On the coding side, I agree something like this is the right approach.

But statistically you really want to check that your mean is within some high-probability interval around the true parameter since randomness implies your result is inaccurate by more than just numerical approximation.

1 Like

Yes I guess it would be more like

@test mean(my_algo(x, y, z) for _ in 1:1_000)  ≈ test_value atol=0.5

with an appropriately chosen tolerance level that reflects the statistical properties rather than just numerical error as you say.

1 Like

yes, a hypothesis test would be ideal. I wondered if there were some convenience macros. Might be a useful extension.

Adding a @clt_test macro could be useful. Maybe start a package?

I would personally start by creating a function here since it’s not clear to me how often you’d need to test expressions rather than evaluate tests against a distribution object. For example, I can imagine something like:

clt_test(d, n_samples=1_000, probtol=1e-6)
# Checks that mean(rand(d, n)) in (lower, upper) based on probability of error 1e-6
1 Like

well, maybe one package at a time :slight_smile: But I think such a macro is perfect for randomized algorithms and algorithms expecting stochastic data inputs.

This PR [WIP] Simulation tests for coverage + p-values by johnmyleswhite · Pull Request #73 · JuliaStats/HypothesisTests.jl · GitHub may be of interest: an attempt to add statistical tests for HypothesisTests.jl.

1 Like