HypothesisTests, rand and very low pvalue

Specific Domains Statistics
1

Consider the following code:

using Distributions
import HypothesisTests as HT
HT.pvalue(HT.ApproximateOneSampleKSTest(rand(Normal(), 1_000_000), Normal()))

I expected that the answer will be very close to one, but depending on the run it can be anywhere from 0.01 to 0.98. Why it is so? The same is with OneSampleADTest and other tests.

Using DiscreteUniform is even worse.

HT.pvalue(HT.OneSampleADTest(rand(DiscreteUniform(1, 10), 1_000_000), DiscreteUniform(1, 10)))

routinely gives values like 6e-10.

So what is wrong here?

2

I don’t think anything is wrong: when the null hypothesis is true, the p-value is uniformly distributed. So you have in particular 1% chance of observing a p-value <= 0.01, and 2% of observing something >= 98%.

For example, repeating your experiment 10000 times (with a much smaller sample size to avoid long computations, but this doesn’t change the result):

using UnicodePlots
using Distributions
import HypothesisTests as HT

get_p() = HT.pvalue(HT.ApproximateOneSampleKSTest(rand(Normal(), 1000), Normal()))
p = [get_p() for _ in 1:10_000]

histogram(p)

# Output:
              ┌                                        ┐ 
   [0.0, 0.1) ┤█████████████████████████████▍ 956        
   [0.1, 0.2) ┤███████████████████████████████▌ 1 025    
   [0.2, 0.3) ┤█████████████████████████████▉ 977        
   [0.3, 0.4) ┤██████████████████████████████▏ 978       
   [0.4, 0.5) ┤████████████████████████████▉ 943         
   [0.5, 0.6) ┤███████████████████████████████▌ 1 025    
   [0.6, 0.7) ┤██████████████████████████████▏ 979       
   [0.7, 0.8) ┤██████████████████████████████▊ 1 002     
   [0.8, 0.9) ┤███████████████████████████████▉ 1 040    
   [0.9, 1.0) ┤█████████████████████████████████  1 075  
              └                                        ┘

The histogram is approximtely flat as expected, because all values of the p-value are equally likely.

4 Likes