# Fisher's test p-value results appear to differ from matlab, R

Hi,
I was using HypothesisTests.jl and noticed minor, but possibly important differences in the p-value estimate between julia and two other packages when using Fisher’s exact test. In this example, matlab and R users might claim a significant result, whereas the julia user might not. This is a fairly common test, and it would be great if there was agreement. Thanks for considering this comment for discussion.
Scott

julia (v0.6):

julia> FisherExactTest(59, 335, 172, 1366)
Fisher's exact test
-------------------
Population details:
parameter of interest:   Odds ratio
value under h_0:         1.0
point estimate:          1.3984544219625261
95% confidence interval: (0.9980930945998796, 1.9393947540537153)

Test summary:
outcome with 95% confidence: fail to reject h_0
**two-sided p-value:           0.051329212328076565    <------------------------**

Details:
contingency table:
59   335
172  1366


matlab:

>> x = table([59;172],[335;1366])
x =
2×2 table
Var1    Var2
____    ____
59      335
172     1366

>> [h,p,stats]=fishertest(x,'Tail','both','Alpha',0.95)

h =
logical
1

**p =**
**   0.045036387203992    <--------------------------------**

stats =
struct with fields:
OddsRatio: 1.398715723707046
ConfidenceInterval: [1.384515635855077 1.413061452741955]


R:

> x = matrix(c(59,172,335,1366), nrow = 2)
> x
[,1] [,2]
[1,]   59  335
[2,]  172 1366
> fisher.test(x,alternative = "two.sided")

Fisher's Exact Test for Count Data

data:  x
**p-value = 0.04503639                   <------------------------------**
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
0.9980904309 1.9393926010
sample estimates:
odds ratio
1.398453903


I’m afraid I can’t help with the statistics question, but it will be easier for others to read your post and help you if you quote your code with backticks.

2 Likes

Might look into it, but from a quick look, there are adjustments to the test for GLM such as those for count data which lead to the proper test being one-tailed. It might be that software usually take into account those cases as opposed to Julia which might not be aware that it is a count model and the user requests a two-tail version.

For asymmetric distributions, p-values for two-sided alternatives are not well defined. It has been discussed couple of times before. See https://github.com/JuliaStats/HypothesisTests.jl/issues/98#issuecomment-301532196 and the reference there. See also the doc string for the pvalue method which states that

For tail = :both, possible values for method are:

•    :central (default): Central interval, i.e. the p-value is two times the minimum of the
one-sided p-values.

•    :minlike: Minimum likelihood interval, i.e. the p-value is computed by summing all
tables with the same marginals that are equally or less probable:

p_ω = \sum_{f_ω(i)≤ f_ω(a)} f_ω(i)


so you can get the p-value used by R by specifying method, i.e.

julia> pvalue(x, method = :minlike)
0.04503638720401171


This value is not more correct than the other one.

8 Likes

I’m curious, does R or Matlab give the option of calculating the :central value that is the default for Julia?

Robin, thanks for the advice on posting. Jose, thanks for advice on modeling. And Andreas, thank you for a great explanation. I was easily able to reproduce the matlab/R results. For a newbie like me who just used the inline REPL help, an additional sentence in the REPL help giving a little more detail on how alternative p-values can be obtained would be helpful!

R does (my first post, I hope I got it right!):

> library(exact2x2)
> x = matrix(c(59,172,335,1366), nrow = 2)
> x
[,1] [,2]
[1,]   59  335
[2,]  172 1366
>
> exact2x2(x,tsmethod="central")

Central Fisher's Exact Test

data:  x
p-value = 0.05133
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
0.9980904 1.9393926
sample estimates:
odds ratio
1.398454

2 Likes