Are there macros for statistical unit tests for randomized algorithms? E.g., if I’d like to test the mean value of an algorithm over 100 runs, e.g… So a macro similar to @btime that would run the method many times and summarize the result.
Have not seen anything like this, but wouldn’t it just be
@test mean(my_algo(x, y, z) for _ in 1:1_000) ≈ test_value
On the coding side, I agree something like this is the right approach.
But statistically you really want to check that your mean is within some high-probability interval around the true parameter since randomness implies your result is inaccurate by more than just numerical approximation.
1 Like
Yes I guess it would be more like
@test mean(my_algo(x, y, z) for _ in 1:1_000) ≈ test_value atol=0.5
with an appropriately chosen tolerance level that reflects the statistical properties rather than just numerical error as you say.
1 Like
yes, a hypothesis test would be ideal. I wondered if there were some convenience macros. Might be a useful extension.
Adding a @clt_test macro could be useful. Maybe start a package?
I would personally start by creating a function here since it’s not clear to me how often you’d need to test expressions rather than evaluate tests against a distribution object. For example, I can imagine something like:
clt_test(d, n_samples=1_000, probtol=1e-6)
# Checks that mean(rand(d, n)) in (lower, upper) based on probability of error 1e-6
1 Like
well, maybe one package at a time 
But I think such a macro is perfect for randomized algorithms and algorithms expecting stochastic data inputs.
This PR [WIP] Simulation tests for coverage + p-values by johnmyleswhite · Pull Request #73 · JuliaStats/HypothesisTests.jl · GitHub may be of interest: an attempt to add statistical tests for HypothesisTests.jl.
1 Like