# A subtle bug?

**URL:** <https://discourse.julialang.org/t/a-subtle-bug/61725>\
**Category:** General Usage\
**Tags:** glm, fast-math\
**Created:** [May 24, 2021, 3:27pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725 "2021-05-24T15:27:46Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [May 24, 2021, 3:27pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/1 "2021-05-24T15:27:46Z")

</div>

I got a `Float64` number from the p-value of a `GLM` model. I could NOT show the fitted model with the following error:

```julia
ERROR: p-values must be in [0; 1]
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] StatsBase.PValue(::Float64) at /Users/Thomas/.julia/packages/StatsBase/EA8Mh/src/statmodels.jl:430

```

after some investigations, I found that some p-value floats are _ **strange** _:

```julia
julia> p = (GLM.coeftable(mod).cols[4])[5] # mod is a model from GLM.lmI()

julia> typeof(p)
Float64

julia> p == 0
true

julia> p >= 0
true

julia> 1 >= p >= 0
true

julia> 0 <= p <= 1 # strange !!!
false

julia> p == 0.0 # strange !!!
false

```

in particular, that strange `false` from `0 <= p <= 1` is the cause of the error messages inside `StatsBase.PValue()`.

I don’t know if it’s a bug from `GLM` or a more general and subtle bug in `Julia`? I’m using v1.5.3, thanks.

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 24, 2021, 3:41pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/2 "2021-05-24T15:41:42Z")

</div>

Without a MWE, I think it will be very difficult to debug your code. Can you please post the result of `dump(p)`?

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [May 24, 2021, 3:51pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/4 "2021-05-24T15:51:27Z")

</div>

```julia
julia> dump(p)
Float64 0.0

```

… there’s a long process to produce the data needed for the linear model… I don’t need how to give a MWE …

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [May 24, 2021, 4:11pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/5 "2021-05-24T16:11:58Z")

</div>

I can’t reproduce

```julia
julia> using StatsBase

julia> StatsBase.PValue(1e-100)
<1e-99

```

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 24, 2021, 4:35pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/6 "2021-05-24T16:35:15Z")

</div>

Would you mind reinstalling Julia using a newer version?

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [May 24, 2021, 5:44pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/7 "2021-05-24T17:44:09Z")

</div>

> [@tomtom](#):
>
> … there’s a long process to produce the data needed for the linear model… I don’t need how to give a MWE …

This is so far from reasonable that you don’t need to worry about the M. Any complete and working example which can reliably reproduce the output from your first message at the end would be good enough.

Pretty much the only ways to explain those results are:

1. It’s not actually the same `p` everywhere.
2. Someone has committed an absolutely heinous act of type piracy and redefined floating point comparisons.

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [May 24, 2021, 5:51pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/8 "2021-05-24T17:51:51Z")

</div>

just installed v1.6.1 and the problem persists.

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [May 24, 2021, 5:55pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/9 "2021-05-24T17:55:14Z")

</div>

is it possible for me to save the data as a file and attach here?

> [@GunnarFarneback](#):
>
> It’s not actually the same `p` everywhere.

it IS the same p everywhere. That’s why it’s “strange”.

===============================================  
I saved the design matrix `X` and response vector `y` into a `.jld` (by `using JLD`).  
Then in a fresh session I loaded them and called `mod = GLM.lm(X, y, true);` and now the problem has gone?! that means I could not reproduce the bug by uploading the data…

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [May 24, 2021, 5:59pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/10 "2021-05-24T17:59:57Z")

</div>

Can you check the following?

```julia
julia> using StatsBase

julia> StatsBase.PValue(0.0)
<1e-99

```

first on a fresh session and then after running your code.

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [May 24, 2021, 6:18pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/11 "2021-05-24T18:18:57Z")

</div>

both on a fresh session and after running my code:

```julia
julia> StatsBase.PValue(0.0)
<1e-99

```

my work flow is:

```julia
julia> mod = GLM.lm(X, y, true);
julia> pvalues = GLM.coeftable(mod).cols[4];
julia> [println(i, " ", pvalues[i], " ", StatsBase.PValue(pvalues[i]) ) for i in 1:length(pvalues)]
1 0.0 <1e-99
2 2.83800947839654e-159 <1e-99
3 0.17914739722604017 0.1791
4 0.00728418728531684 0.0073
ERROR: p-values must be in [0; 1]
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] StatsBase.PValue(v::Float64)
   @ StatsBase ~/.julia/packages/StatsBase/DU1bT/src/statmodels.jl:485

julia> p = pvalues[5] # 1.6.1 shows NaN, 1.5.3 shows 0.0
NaN

julia> p == 0
true

julia> p == 0.0
false

julia> isnan(p)
false

julia> isfinite(p)
true

julia> 0 <= p <= 1
false

julia> 1 >= p >= 0
true

```

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 24, 2021, 6:37pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/12 "2021-05-24T18:37:11Z")

</div>

I have no idea, I could not reproduce it here… Thus, I will start to say random things 😃

Is it possible to execute changing the name of `mod` to something else? `mod` is a function in `Base`. It is not supposed to give you any problem, but I am lacking ideas…

EDIT: Wait, I am not understanding, `p` is `NaN` in 1.6 and still `isnan(p)` is false?

---

<div class="post-metadata">

**Author:** ![viraltux](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viraltux/32/15236_2.png) [@viraltux](https://discourse.julialang.org/u/viraltux)\
**Post date:** [May 24, 2021, 6:57pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/13 "2021-05-24T18:57:07Z")

</div>

Yeah, unfortunately seems calculations of p-values are not quite right in a few key packages, we have the same error with:

```julia
a = [12,10,7,6,3,1]
b = [11,9,8,5,4,2]
MannWhitneyUTest(a,b)

Error showing value of type ExactMannWhitneyUTest{Float64}:
ERROR: p-values must be in [0; 1]
Stacktrace:
...

```

And this is an old problem: [p val \> 1 in ExactMannWhitneyUTest · Issue #126 · JuliaStats/HypothesisTests.jl · GitHub](https://github.com/JuliaStats/HypothesisTests.jl/issues/126)

---

<div class="post-metadata">

**Author:** ![sostock](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sostock/32/5546_2.png) [@sostock](https://discourse.julialang.org/u/sostock)\
**Post date:** [May 24, 2021, 7:27pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/14 "2021-05-24T19:27:09Z")

</div>

Can you post the output of `bitstring(p)` for this weird `p`?

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [May 25, 2021, 3:56am UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/15 "2021-05-25T03:56:43Z")

</div>

```julia
julia> bitstring(p)
"0111111111111000000000000000000000000000000000000000000000000000"

```

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [May 25, 2021, 4:07am UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/16 "2021-05-25T04:07:15Z")

</div>

now I know the cause of the problem: `--math-mode=fast`

I understand that any operation on `NaN` is **unpredictable** in fastmath mode, we **need an important exception** : `isnan(NaN)`.

now in fastmath mode:

```julia
julia> isnan(NaN)
false

```

**and this failure to detect `NaN` is the cause of all confusions.** In this case, the following `isnan(v)` fails to catch the `NaN` and throws an error:

```julia
struct PValue <: Real
    v::Real
    function PValue(v::Real)
        0 <= v <= 1 || isnan(v) || error("p-values must be in [0; 1]")
        new(v)
    end
end

```

we often have no control whether a package would produce `NaN`, and on the other hand, the package has no control if the user is doing fastmath or not. I **strongly recommend that `isnan(NaN)` to return `true` even in fastmath mode!**

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [May 25, 2021, 4:56am UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/17 "2021-05-25T04:56:43Z")

</div>

> [@tomtom](#):
>
> we often have no control whether a package would produce `NaN` , and on the other hand, the package has no control if the user is doing fastmath or not.

Exactly, which is why you should never use global "fast"math in real applications.

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [May 25, 2021, 5:56am UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/18 "2021-05-25T05:56:49Z")

</div>

but why don’t we allow `isnan(NaN) == true` in fastmath mode?

---

<div class="post-metadata">

**Author:** ![Syx\_Pek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/syx_pek/32/6364_2.png) [@Syx\_Pek](https://discourse.julialang.org/u/Syx_Pek)\
**Post date:** [May 25, 2021, 6:20am UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/20 "2021-05-25T06:20:10Z")

</div>

[https://github.com/JuliaLang/julia/issues/21375](https://github.com/JuliaLang/julia/issues/21375)  
" Asking `isnan` correctly is computationally expensive and slow "

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [May 25, 2021, 9:05am UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/21 "2021-05-25T09:05:28Z")

</div>

> [@tomtom](#):
>
> but why don’t we allow `isnan(NaN) == true` in fastmath mode?

Next thing you will get some other error somewhere else because of another assumption of IEEE math in another package. It’s just completely unsafe to use this globally for running code you do not control 100% yourself.

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [May 25, 2021, 1:42pm UTC](https://discourse.julialang.org/t/a-subtle-bug/61725/22 "2021-05-25T13:42:59Z")

</div>

> [@Syx\_Pek](#):
>
> " Asking `isnan` correctly is computationally expensive and slow "

…it is just sad to hear that … 😰

well… seems like I could only **override my own** `isnan()` to cope with the issue…  
a question to ask: how could I detect if the current session is `--math-mode=fast` or not? thanks.

[Next page](https://discourse.julialang.org/t/a-subtle-bug/61725.md?page=2)