# Kolmogorov-Smirnov test

**URL:** <https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990>\
**Category:** Statistics\
**Tags:** distributions, hypothesis-tests\
**Created:** [November 9, 2022, 3:38pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990 "2022-11-09T15:38:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![GHTaarn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ghtaarn/32/216007_2.png) [@GHTaarn](https://discourse.julialang.org/u/GHTaarn)\
**Post date:** [November 9, 2022, 3:38pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/1 "2022-11-09T15:38:05Z")

</div>

I have a sample consisting of a set of integers and want to test if can be from a specific distribution. I am not that strong in statistics, but my understanding is that the Kolmogorov-Smirnov test is a good choice for assessing this, so I tried the code below:

```julia
using HypothesisTests, Distributions
n = 50
x = 0.5 .^ (1:n)
d = DiscreteNonParametric(1:n, x ./ sum(x))
sample = vcat(ones(Int64,16), 2*ones(Int64,8), 3*ones(Int64,4), 4,4,5)
display(ExactOneSampleKSTest(sample, d))

```

The problem is that this rejects the hypothesis, even though the sample seems to fit the distribution almost perfectly. Can anyone explain why this happens? The output is given below:

```julia
┌ Warning: This test is inaccurate with ties
└ @ HypothesisTests ~/.julia/packages/HypothesisTests/BgrVj/src/kolmogorov_smirnov.jl:68
Exact one sample Kolmogorov-Smirnov test
----------------------------------------
Population details:
    parameter of interest: Supremum of CDF differences
    value under h_0: 0.0
    point estimate: 0.5

Test summary:
    outcome with 95% confidence: reject h_0
    two-sided p-value: <1e-06

Details:
    number of observations: 31

```

---

<div class="post-metadata">

**Author:** ![awasserman](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@awasserman](https://discourse.julialang.org/u/awasserman)\
**Post date:** [November 9, 2022, 5:38pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/2 "2022-11-09T17:38:31Z")

</div>

It seems to me like the default `tail` value for `pvalue(t::ExactOneSampleKSTest, tail=:both)` is wrong. I think it should use `tail = :right`, since we’re interested in the probability of the KS statistic being less than the value we have.

If we do `pvalue(ExactOneSampleKSTest(sample, d), tail=:right)`, we get `p = 0.92`, which seems more reasonable. To be honest, I’m a little confused by the implementation of this in HypothesisTests.

---

<div class="post-metadata">

**Author:** ![GHTaarn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ghtaarn/32/216007_2.png) [@GHTaarn](https://discourse.julialang.org/u/GHTaarn)\
**Post date:** [November 9, 2022, 7:49pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/3 "2022-11-09T19:49:02Z")

</div>

Thank you, you certainly seem to know much more about this than me, so I think I will file an issue with the developers and see if I can get some clarity on whether this is a bug or just a misunderstanding.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [November 9, 2022, 8:11pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/4 "2022-11-09T20:11:09Z")

</div>

When you file the issue can you post a link here? People might be interested in the discussion, but it is definitely to discuss it there.

---

<div class="post-metadata">

**Author:** ![Paul\_Soderlind](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paul_soderlind/32/1753_2.png) [@Paul\_Soderlind](https://discourse.julialang.org/u/Paul_Soderlind)\
**Post date:** [November 9, 2022, 8:18pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/5 "2022-11-09T20:18:22Z")

</div>

In case you want to double check the results, cell 11 or so in these lecture notes has a simple code for that: [FinancialEconometrics/Ch19\_TestingDistributions.ipynb at master · PaulSoderlind/FinancialEconometrics · GitHub](https://github.com/PaulSoderlind/FinancialEconometrics/blob/master/Ch19_TestingDistributions.ipynb)

---

<div class="post-metadata">

**Author:** ![awasserman](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@awasserman](https://discourse.julialang.org/u/awasserman)\
**Post date:** [November 10, 2022, 12:18am UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/6 "2022-11-10T00:18:22Z")

</div>

Looking at this more closely, I don’t think it’s a problem with the choice of tails, but rather in the computation of the KS statistic.

See in comparison to the implementation in `scipy.stats`

HypothesisTests.jl:

```julia
julia> print(x)
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 5]
julia> print(y)
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 4, 5, 8]
julia> t = ApproximateTwoSampleKSTest(x, y)
┌ Warning: This test is inaccurate with ties
└ @ HypothesisTests ~/.julia/packages/HypothesisTests/BgrVj/src/kolmogorov_smirnov.jl:167
Approximate two sample Kolmogorov-Smirnov test
----------------------------------------------
Population details:
    parameter of interest: Supremum of CDF differences
    value under h_0: 0.0
    point estimate: 0.516129

Test summary:
    outcome with 95% confidence: reject h_0
    two-sided p-value: 0.0005

Details:
    number of observations: [31,31]
    KS-statistic: 2.032002032003047

```

vs scipy stats

```julia
In [13]: stats.kstest(x, y, alternative="two-sided", method="asymp")
Out[13]: KstestResult(statistic=0.032258064516129115, pvalue=1.0)

In [14]: stats.kstest(x, y, alternative="greater", method="asymp")
Out[14]: KstestResult(statistic=0.032258064516129115, pvalue=0.9375209928337668)

In [15]: stats.kstest(x, y, alternative="less", method="asymp")
Out[15]: KstestResult(statistic=0.032258064516129004, pvalue=0.9375209928337671)

```

---

<div class="post-metadata">

**Author:** ![GHTaarn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ghtaarn/32/216007_2.png) [@GHTaarn](https://discourse.julialang.org/u/GHTaarn)\
**Post date:** [November 10, 2022, 5:41am UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/7 "2022-11-10T05:41:33Z")

</div>

I have now filed the issue with the developers [ExactOneSampleKSTest rejects hypothesis incorrectly · Issue #280 · JuliaStats/HypothesisTests.jl · GitHub](https://github.com/JuliaStats/HypothesisTests.jl/issues/280)

Thanks to all that participated in this thread.

---

<div class="post-metadata">

**Author:** ![PharmCat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pharmcat/32/6953_2.png) [@PharmCat](https://discourse.julialang.org/u/PharmCat)\
**Post date:** [November 10, 2022, 9:05am UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/8 "2022-11-10T09:05:19Z")

</div>

> [@awasserman](#):
>
> `t = ApproximateTwoSampleKSTest(x, y)`

SPSS28 test for this data

![image](https://global.discourse-cdn.com/julialang/original/3X/3/1/31bbbdb653c8daa84842cbe2dcb10f44e5adfc44.png)

```julia
NPTESTS 
  /INDEPENDENT TEST (V) GROUP (G) KOLMOGOROV_SMIRNOV 
  /MISSING SCOPE=ANALYSIS USERMISSING=EXCLUDE 
  /CRITERIA ALPHA=0.05 CILEVEL=95.

```

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [November 10, 2022, 12:01pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/9 "2022-11-10T12:01:58Z")

</div>

> [@GHTaarn](#):
>
> ```julia
> ┌ Warning: This test is inaccurate with ties
> └ @ HypothesisTests ~/.julia/packages/HypothesisTests/BgrVj/src/kolmogorov_smirnov.jl:68
> 
> ```

I think this is the case to pay attention to the warning. The calculation of KS statistic involves sorting and because of stability of sort (i.e. preserve order of identical elements) problems can get exacerbated to extreme.

For example:

```julia
julia> pvalue(ApproximateTwoSampleKSTest(sample, sample), tail=:both)
┌ Warning: This test is inaccurate with ties
└ @ HypothesisTests ~/.julia/packages/HypothesisTests/BgrVj/src/kolmogorov_smirnov.jl:167
0.000518320212633612 # rejects strongly

julia> pvalue(ApproximateTwoSampleKSTest(
  sample.+rand(length(sample))/100, 
  sample.+rand(length(sample))/100), tail=:both)

0.8148352946917117 # "nothing to see here"

julia> histogram([pvalue(
  ApproximateTwoSampleKSTest(sample.+rand(length(sample))/100,
  sample.+rand(length(sample))/100), tail=:both) for i in 1:1000])  
  # uses UnicodePlots
              ┌ ┐ 
   [0.0, 0.1) ┤▎ 3                                       
   [0.1, 0.2) ┤▋ 11                                      
   [0.2, 0.3) ┤█▍ 21                                     
   [0.3, 0.4) ┤ 0                                       
   [0.4, 0.5) ┤███▋ 55                                   
   [0.5, 0.6) ┤ 0                                       
   [0.6, 0.7) ┤█████████▏ 134                            
   [0.7, 0.8) ┤ 0                                       
   [0.8, 0.9) ┤█████████████████▍ 257                    
   [0.9, 1.0) ┤███████████████████████████████████ 519  
              └ ┘ 
                               Frequency                 

julia> count([pvalue(
  ApproximateTwoSampleKSTest(
  sample.+rand(length(sample))/100, 
  sample.+rand(length(sample))/100), tail=:both) for i in 1:1000] .< 0.05)
0

```

It is possible to mitigate this tie-breaking problem in the HypothesisTests package (optionally, and in a way that allows reproducability of tie-breaking). But the results should be okay with a little jitter on the caller side (size of jitter dependent on underlying sample and distrubution).

Note:

1. `sample` is `[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 5]`, and jitter is small compared to samples.

2. The distribution of p-values is a bit non-uniform, which merits a bit of investigating.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [November 10, 2022, 2:53pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/10 "2022-11-10T14:53:10Z")

</div>

I vaguely recall that the KS test assumes no ties. Is that perhaps the problem here? Using a stable sort would essentially have the effect of breaking all ties by order of appearance, I’m not sure if that’s sufficient or necessary to fix this.

---

<div class="post-metadata">

**Author:** ![GHTaarn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ghtaarn/32/216007_2.png) [@GHTaarn](https://discourse.julialang.org/u/GHTaarn)\
**Post date:** [November 10, 2022, 3:31pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/11 "2022-11-10T15:31:26Z")

</div>

I tried to add jitter to my example and it removed the warning, but the result was the same.

I am beginning to suspect that the implementation in `ExactOneSampleKSTest` does not work for discrete distributions. According to [The Kolmogorov-Smirnov Goodness-of-fit Test - YouTube](https://www.youtube.com/watch?v=e5zqHxb9un4) the Kolmogorov-Smirnov test can be used on discrete distributions, but I would not expect that code made for continuous distributions would give reliable results for discrete distributions, regardless of whether jitter is added to the sample(s) or not.

---

<div class="post-metadata">

**Author:** ![mschauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mschauer/32/13946_2.png) [@mschauer](https://discourse.julialang.org/u/mschauer)\
**Post date:** [November 10, 2022, 3:40pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/12 "2022-11-10T15:40:01Z")

</div>

Yes, the Kolmogorov Smirnov test uses the continuity of the distribution. For discrete distributions it is more conservative, so not clear why it _rejects_, can you plot the empirical CDF versus the theoretical CDF?

---

<div class="post-metadata">

**Author:** ![GHTaarn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ghtaarn/32/216007_2.png) [@GHTaarn](https://discourse.julialang.org/u/GHTaarn)\
**Post date:** [November 10, 2022, 4:09pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/13 "2022-11-10T16:09:24Z")

</div>

I would expect that it rejects because it bases its decision on the `x` value that has the largest difference between the two CDFs and since the theoretical CDF is discontinuous and e.g. jumps from 0 to 0.5 for `x`=1, then the best case scenario for the largest difference would be ~0.25 which it apparently feels is enough to reject.

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [November 10, 2022, 4:24pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/14 "2022-11-10T16:24:59Z")

</div>

> [@GHTaarn](#):
>
> I am beginning to suspect that the implementation in `ExactOneSampleKSTest` does not work for discrete distributions

They should work with discrete distributions but I think the implementations were written with the assumption of no ties, and so the tie case is not handled gracefully.

> [@GHTaarn](#):
>
> I tried to add jitter to my example and it removed the warning, but the result was the same.

The jitter solution is good for the two-sample case (as shown in my previous response). The one-sample and two-sample cases are implemented separately and need different “fixes”.

Adding below an initial “fix” implementation, which you can test. Tried to pay attention to ties in generalizing the original implementation.

```julia
using HypothesisTests, Distributions, Random, StatsBase

import HypothesisTests: ksstats

#### UPDATE: This function isn't correct. Don't use.
# used in one-sided test
# function ksstats(x::AbstractVector{T}, d::UnivariateDistribution) where T<:Real
# n = length(x)
# cdfs = cdf.(Ref(d), sort(x))
# δp = maximum(competerank(x) ./ n .- cdfs) # fixed ranks from 1:n
# δn = -minimum((n .- competerank(x, lt=(>))) ./ n .- cdfs)
# δ = max(δn, δp)
# (n, δ, δp, δn)
# end

# used in two-sided test
function ksstats(x::AbstractVector{T}, y::AbstractVector{S}) where {T<:Real, S<:Real}
    n_x, n_y = length(x), length(y)
    n = n_x + n_y
    randperm = shuffle(1:n) # add a random perm to random tie-break
    sort_idx = sortperm([x; y][randperm])
    pdf_diffs = [ones(n_x)/n_x; -ones(n_y)/n_y][randperm][sort_idx]
    cdf_diffs = cumsum(pdf_diffs)
    δp = maximum(cdf_diffs)
    δn = -minimum(cdf_diffs)
    δ = max(δp, δn)
    (n_x, n_y, δ, δp, δn)
end

```

With these overwritten `ksstats` function, the original code should/might work.

---

<div class="post-metadata">

**Author:** ![GHTaarn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ghtaarn/32/216007_2.png) [@GHTaarn](https://discourse.julialang.org/u/GHTaarn)\
**Post date:** [November 10, 2022, 4:55pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/15 "2022-11-10T16:55:53Z")

</div>

Thank you, I ran this on the example in the original post and it “failed to reject h\_0”, but it now seems to do this even when I would expect it to reject:

```julia-repl
julia> ExactOneSampleKSTest(fill(31,9999),d)
┌ Warning: This test is inaccurate with ties
└ @ HypothesisTests ~/.julia/packages/HypothesisTests/BgrVj/src/kolmogorov_smirnov.jl:68
Exact one sample Kolmogorov-Smirnov test
----------------------------------------
Population details:
    parameter of interest: Supremum of CDF differences
    value under h_0: 0.0
    point estimate: 0.00010001

Test summary:
    outcome with 95% confidence: fail to reject h_0
    two-sided p-value: 1.0000

Details:
    number of observations: 9999

julia> 

```

using the same `d` as in the original post.

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [November 10, 2022, 5:04pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/16 "2022-11-10T17:04:52Z")

</div>

Oops. Need to have a look. In the meantime, the two-sample test is a little simpler and so probably correct now. It can be used as a stop gap:

```julia
julia> ApproximateTwoSampleKSTest(fill(1,9999), rand(d,9999))
┌ Warning: This test is inaccurate with ties
└ @ HypothesisTests ~/.julia/packages/HypothesisTests/BgrVj/src/kolmogorov_smirnov.jl:167
Approximate two sample Kolmogorov-Smirnov test
----------------------------------------------
Population details:
    parameter of interest: Supremum of CDF differences
    value under h_0: 0.0
    point estimate: 0.50155

Test summary:
    outcome with 95% confidence: reject h_0
    two-sided p-value: <1e-99

Details:
    number of observations: [9999,9999]
    KS-statistic: 35.46317827975053

```

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [November 10, 2022, 6:36pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/17 "2022-11-10T18:36:03Z")

</div>

It seems K-S test is not really suited for discrete distributions as-is, but has been extended to support them. The improved test is Cramer-von-Mises test. A good reference (I haven’t had time to read): [http://www.stat.yale.edu/~jay/EmersonMaterials/DiscreteGOF.pdf](http://www.stat.yale.edu/~jay/EmersonMaterials/DiscreteGOF.pdf).

Perhaps a Chi-square Goodness-of-fit test is worth exploring for this problem.

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [November 11, 2022, 1:26pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/18 "2022-11-11T13:26:12Z")

</div>

Continuing from last post, here is a Chi-squared test for the discussed problem:

```julia
onesampleGOF(sample, d::DiscreteNonParametric) = begin
    cm = [get(countmap(sample),k,0) for k in d.support]
    sum(values(cm)) < length(sample) && error("Sample has elements outside support")
    return ChisqTest(cm, d.p)
end

```

and then it is possible to:

```julia
julia> pvalue(onesampleGOF(2 .* sample, d))
7.485816224539387e-10
julia> pvalue(onesampleGOF(sample, d))
1.0
julia> pvalue(onesampleGOF(fill(31,10), d))
0.0
julia> pvalue(onesampleGOF(rand(d,100), d))
0.9999999116094588
julia> pvalue(onesampleGOF(rand(1:51,10), d))
0.0

```

---

<div class="post-metadata">

**Author:** ![GHTaarn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ghtaarn/32/216007_2.png) [@GHTaarn](https://discourse.julialang.org/u/GHTaarn)\
**Post date:** [November 14, 2022, 7:00pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/19 "2022-11-14T19:00:15Z")

</div>

Thank you, so this is how to do a Chi Squared test with `HypothesisTests`, I actually ended up making my own basic implementation.

---

<div class="post-metadata">

**Author:** ![conalbrown](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/conalbrown/32/30423_2.png) [@conalbrown](https://discourse.julialang.org/u/conalbrown)\
**Post date:** [April 29, 2025, 8:24pm UTC](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990/20 "2025-04-29T20:24:27Z")

</div>

I’ve been digging around in kolmogorov\_smirnov.jl and found that the [pvalue()](https://github.com/JuliaStats/HypothesisTests.jl/blob/c440511c4c67b11e36417e8de36343670db4d56b/src/kolmogorov_smirnov.jl#L178) function calculates the value of _n_ on line 179 differently to other references.

```julia
function StatsAPI.pvalue(x::ApproximateTwoSampleKSTest; tail=:both)
    n = x.n_x*x.n_y/(x.n_x+x.n_y)
    if tail == :left
        exp(-2*n*x.δn^2)
    elseif tail == :right
        exp(-2*n*x.δp^2)
    elseif tail == :both
        pvalue(Kolmogorov(), sqrt(n)*x.δ; tail=:right)
    else
        throw(ArgumentError("tail=$(tail) is invalid"))
    end
end

```

Line 179 is

> n = x.n\_x\*x.n\_y/(x.n\_x+x.n\_y)

The other references I have seen divide the sum by the product (or add the inverses of n and m) [e.g. [1](https://hal.science/hal-01045013v1), [2](https://sparky.rice.edu/astr360/kstest.pdf), [3](http://www.qualityresearchinternational.com/methodology/RRW8pt4pt3pt6pt2pt3KS.php), [4](https://real-statistics.com/non-parametric-tests/goodness-of-fit-tests/two-sample-kolmogorov-smirnov-test/)] , whereas the product of n and m (n\_x and n\_y) is divided by the sum above.

Is there a mistake in the code or am I misinterpreting how the function goes on to calculate the p-value? I suspect the latter as it is calculating the p-value rather than the critical distance.

I notice this issue is still open in Github so replying off this post. Please moderate my post if needed and advise whether I should just post this directly to Github.

[Next page](https://discourse.julialang.org/t/kolmogorov-smirnov-test/89990.md?page=2)