# Performance of \`Union{Missing,Float64}\`
**URL:** https://discourse.julialang.org/t/performance-of-union-missing-float64/16348
**Category:** General Usage
**Tags:** question
**Created:** [October 15, 2018, 3:53pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348 "2018-10-15T15:53:03Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@e3c6](https://discourse.julialang.org/u/e3c6)
#### Post date: [October 15, 2018, 3:53pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/1 "2018-10-15T15:53:03Z")
From [https://julialang.org/blog/2018/06/missing:](https://julialang.org/blog/2018/06/missing:)
> For example, missing values still have a significant performance impact for arrays of `Float64` elements, which are essential for numeric computing.
```julia
function sum_nonmissing(X::AbstractArray)
s = zero(eltype(X))
@inbounds @simd for x in X
if x !== missing
s += x
end
end
s
end
julia> Y1 = rand(10_000_000);
julia> Y2 = Vector{Union{Missing, Float64}}(Y1);
julia> Y3 = ifelse.(rand(length(Y2)) .< 0.9, Y2, missing);
julia> @btime sum_nonmissing(Y1);
5.733 ms (1 allocation: 16 bytes)
julia> @btime sum_nonmissing(Y2);
13.854 ms (1 allocation: 16 bytes)
julia> @btime sum_nonmissing(Y3);
17.780 ms (1 allocation: 16 bytes)
```
I still see the performance drop in Julia v1.0.1. But I found no Github issue tracking the performance drop. Is there one open already?
---
### Author:  [@ederag](https://discourse.julialang.org/u/ederag)
#### Post date: [May 18, 2021, 9:34am UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/2 "2021-05-18T09:34:52Z")
At least `missing` is better than `NaN` for this purpose.
(timings under `julia-1.6.1`)
```julia
using BenchmarkTools
function sum_non_nan(X::AbstractArray)
s = zero(eltype(X))
@inbounds @simd for x in X
# it is even slower with isnan()
if x !== NaN
s += x
end
end
s
end
julia> Y1 = rand(10_000_000);
julia> @btime sum_nonmissing($Y1);
2.685 ms (0 allocations: 0 bytes)
julia> @btime sum_non_nan($Y1);
6.807 ms (0 allocations: 0 bytes)
julia> Y2 = Vector{Union{Missing, Float64}}(Y1);
julia> @btime sum_nonmissing($Y2);
7.112 ms (0 allocations: 0 bytes)
# Y2_nan would be identical to Y1, so see timing above (6.771 ms)
julia> Y3 = ifelse.(rand(length(Y2)) .< 0.9, Y2, missing);
julia> Y3_nan = Array{Float64}(replace(x->ismissing(x) ? NaN : x, Y3));
julia> @btime sum_nonmissing($Y3);
12.180 ms (0 allocations: 0 bytes)
julia> @btime sum_non_nan($Y3_nan);
13.216 ms (0 allocations: 0 bytes)
```
---
### Author:  [@Elrod](https://discourse.julialang.org/u/Elrod)
#### Post date: [May 18, 2021, 12:43pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/3 "2021-05-18T12:43:53Z")
```julia
julia> x = rand(10_000_000);
julia> function sum_non_nan(X::AbstractArray)
s = zero(eltype(X))
@inbounds @simd for x in X
# simplify the branch so it can SIMD.
s += isnan(x) ? zero(x) : x
end
s
end
sum_non_nan (generic function with 1 method)
julia> @btime sum_non_nan($x)
3.941 ms (0 allocations: 0 bytes)
5.00030913257381e6
julia> @btime sum($x)
4.268 ms (0 allocations: 0 bytes)
5.0003091325738225e6
```
---
### Author:  [@anon56330260](https://discourse.julialang.org/u/anon56330260)
#### Post date: [May 18, 2021, 12:44pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/4 "2021-05-18T12:44:19Z")
Is your benchmark correct?
```julia
julia> @btime sum_non_nan($Y3);
16.211 ms (0 allocations: 0 bytes)
```
Here `Y3` should be `Y3_nan`? ~~Also `x!==NaN` is not the same thing as `isnan`, since `NaN!=NaN`.~~ No, NaN===NaN, but NaN != NaN, thanks @fph for pointing out this.
---
### Author:  [@fph](https://discourse.julialang.org/u/fph)
#### Post date: [May 18, 2021, 12:48pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/5 "2021-05-18T12:48:04Z")
> [@anon56330260](#):
>
> Also `x!==NaN` is not the same thing as `isnan` , since `NaN!=NaN` .
Note that this uses `===`-comparison. It is not the same thing as `isnan`, but it should work as long as one does not use NaNs with payloads and signaling NaNs and that kind of stuff.
---
### Author:  [@ederag](https://discourse.julialang.org/u/ederag)
#### Post date: [May 18, 2021, 12:57pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/6 "2021-05-18T12:57:14Z")
Indeed it should have been `Y3_nan`. Thanks, fixed now.
---
### Author:  [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [May 18, 2021, 8:12pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/7 "2021-05-18T20:12:47Z")
```julia
julia> function sum_non_nan(X::AbstractArray)
s = zero(eltype(X))
@inbounds @simd for x in X
# simplify the branch so it can SIMD.
s += isnan(x) ? zero(x) : x
end
s
end
julia> function sum_nonmissing(X::AbstractArray)
s = zero(eltype(X))
@inbounds @simd for x in X
s += ismissing(x) ? zero(x) : x
end
s
end
julia> Y1 = rand(10_000_000);
julia> Y2 = Vector{Union{Missing, Float64}}(Y1);
julia> Y3 = ifelse.(rand(length(Y2)) .< 0.9, Y2, missing);
julia> Y3_nan = Array{Float64}(replace(x->ismissing(x) ? NaN : x, Y3));
julia> @btime sum_nonmissing($Y1)
9.132 ms (0 allocations: 0 bytes)
4.999213478955774e6
julia> @btime sum_non_nan($Y1)
10.114 ms (0 allocations: 0 bytes)
4.999213478955774e6
julia> @btime sum_nonmissing($Y2);
17.643 ms (0 allocations: 0 bytes)
julia> @btime sum_nonmissing($Y3);
13.534 ms (0 allocations: 0 bytes)
julia> @btime sum_non_nan($Y3_nan);
10.156 ms (0 allocations: 0 bytes)
```
The only time `sum_nonmissing` seems more efficient to me is with `@btime sum_nonmissing($Y1)`. That’s because we can ignore the `ismissing` call when applied to `Vector{Float64}`.
---
### Author:  [@Vasily\_Pisarev](https://discourse.julialang.org/u/Vasily_Pisarev)
#### Post date: [May 18, 2021, 9:42pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/8 "2021-05-18T21:42:59Z")
> [@fph](#):
>
> it should work as long as one does not use NaNs with payloads and signaling NaNs and that kind of stuff.
If the NaNs result from an arithmetic operation, they may or may not be equivalent to the literal NaN.
```julia
julia> 0.0 / 0.0
NaN
julia> ans === NaN
false
```
Also, `=== NaN` ignores that there are non-64-bit floats while `isnan()` is generic in that respect.
---
### Author:  [@jzr](https://discourse.julialang.org/u/jzr)
#### Post date: [May 18, 2021, 9:59pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/9 "2021-05-18T21:59:32Z")
> [@Vasily\_Pisarev](#):
>
> ```julia
> julia> ans === NaN
> false
>
> ```
Why does this happen?
---
### Author:  [@Vasily\_Pisarev](https://discourse.julialang.org/u/Vasily_Pisarev)
#### Post date: [May 18, 2021, 10:14pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/10 "2021-05-18T22:14:07Z")
> [@jzr](#):
>
> Why does this happen?
From Wikipedia page on NaN:
> For example, a bit-wise IEEE 754 single precision (32-bit) NaN would be
> `s111 1111 1xxx xxxx xxxx xxxx xxxx xxxx`
> where _s_ is the sign (most often ignored in applications) and the _x_ sequence represents a non-zero number (the value zero encodes infinities)
Thus, the bit sequences of two NaNs are not necessarily the same, and my guess is that `===` for two floats compares their bit sequences.
---
### Author:  [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [May 19, 2021, 9:54am UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/11 "2021-05-19T09:54:44Z")
This is fun.
```julia
julia> @less isnan(0.0 / 0.0)
isnan(x::AbstractFloat) = x != x
julia> x = 0.0 / 0.0
NaN
julia> x != x
true
julia> NaN != NaN
true
julia> 0.0 / 0.0 === -NaN
true
julia> bitstring(0.0/0.0)
"1111111111111000000000000000000000000000000000000000000000000000"
julia> bitstring(NaN)
"0111111111111000000000000000000000000000000000000000000000000000"
julia> bitstring(-0.0/0.0)
"1111111111111000000000000000000000000000000000000000000000000000"
julia> 0.0 / 0.0 === -NaN
true
```
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [May 24, 2021, 12:20pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/12 "2021-05-24T12:20:52Z")
> [@mkitti](#):
>
> `0.0 / 0.0 === -NaN`
Note that here you are really relying on `-NaN === NaN` — there is no “negative `NaN`”.
Also, `isnan` is the recommended way of `NaN` comparisons, as they may carry a payload. See
> **[GitHub - JeffreySarnoff/QNaNs.jl: Use of quiet NaNs carrying different payloads.](https://github.com/JeffreySarnoff/QNaNs.jl)**
>
> Use of quiet NaNs carrying different payloads. Contribute to JeffreySarnoff/QNaNs.jl development by creating an account on GitHub.
---
### Author:  [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [May 24, 2021, 7:47pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/13 "2021-05-24T19:47:54Z")
> [@Tamas\_Papp](#):
>
> Note that here you are really relying on `-NaN === NaN` — there is no "negative `NaN` ".
>
> Also, `isnan` is the recommended way of `NaN` comparisons, as they may carry a payload. See
To be clear, that statement evaluates to `false`.
```julia
julia> -NaN === NaN
false
julia> -NaN === -NaN
true
```
Are you sure there is no “negative `NaN`”? There definitely seems to be different `NaN`s to me and QNaNs.jl seems to further that point.
---
### Author:  [@Sukera](https://discourse.julialang.org/u/Sukera)
#### Post date: [May 24, 2021, 9:18pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/14 "2021-05-24T21:18:40Z")
What @Tamas_Papp means is that there’s no “negative NaN” in the sense that negativity doesn’t make sense for `NaN`. NaNess is a property irrespective of the sign bit.
Also:
```julia
julia> reinterpret(Int, -NaN) |> bitstring
"1111111111111000000000000000000000000000000000000000000000000000"
julia> reinterpret(Float64, reinterpret(Int, -NaN) | 0xff)
NaN
julia> reinterpret(Float64, reinterpret(Int, -NaN) | 0xff) |> bitstring
"1111111111111000000000000000000000000000000000000000000011111111"
julia> reinterpret(Float64, reinterpret(Int, -NaN) | 0xff) |> isnan
true
julia> reinterpret(Float64, reinterpret(Int, -NaN) | 0xff) === NaN
false
julia> reinterpret(Float64, reinterpret(Int, -NaN) | 0xff) === -NaN
false
```
Note how the significand of the second `NaN` is not `0`, but the number is still `NaN`. It’s the same with the sign bit - all that makes it `NaN` is an exponent of all ones (which is the intended way `NaN`s should work according to IEEE 754).
> [@mkitti](#):
>
> ```julia
> julia> -NaN === NaN
> false
>
> ```
That comparison compares bitwise patterns, not equality. It’s asking whether or not the two values are _exactly_ the same, not whether they’re semantically the same. `==` (semantic equivalence) indeed gives `false`:
```julia
julia> -NaN |> bitstring
"1111111111111000000000000000000000000000000000000000000000000000"
julia> NaN |> bitstring
"0111111111111000000000000000000000000000000000000000000000000000"
julia> (-NaN) == NaN
false
# according to IEEE 754, any logical/semantic comparison with NaN should be false
julia> NaN == NaN
false
```
[The wikipedia article on NaN](https://en.wikipedia.org/wiki/NaN) has a lot of useful info about how NaN can be compared and what the result should be, as well as what NaNs are sometimes used for if there’s no other means of signaling/error checking available.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [May 25, 2021, 2:47pm UTC](https://discourse.julialang.org/t/performance-of-union-missing-float64/16348/15 "2021-05-25T14:47:04Z")
> [@mkitti](#):
>
> Are you sure there is no "negative `NaN` "?
My understanding of IEEE 754 is that `-NaN` flipping the sign bit is implementation-dependent, and the sign bit of `NaN` results may be accidental anyway in conforming implementations when both the input and the output are `NaN` (it is not part of the payload, and is generally ignored, except for a few special cases enumerated in the standard). See Section 6.3 of IEEE 754-2008.
In any case, I would not rely on this.