# Replacing missing values in a matrix is super slow

**URL:** <https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213>\
**Category:** General Usage\
**Tags:** question\
**Created:** [December 16, 2021, 4:22pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213 "2021-12-16T16:22:36Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [December 16, 2021, 4:22pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/1 "2021-12-16T16:22:36Z")

</div>

I have a matrix with the size of 360x180x1200. It contains lots of values that are equal to 100000002004087734272. I assume those are missing value indicators. I tried to use the below command to replace them with NaNs, but it is taking forever to complete.  
A[A .\> 100] .= NaN;

In comparison, when I tried to do the same thing in Matlab (see below), it was blazingly fast and literally took no time at all. It was done as soon as I finished clicking Enter.  
A(A\>100) = NaN;

What am I missing? Is there a faster way to do this in Julia?

Many thanks!

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [December 16, 2021, 4:44pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/2 "2021-12-16T16:44:08Z")

</div>

Here’s what I see:

```julia
using BenchmarkTools

A = 200*rand(360, 180, 1200);
@btime $A[$A .> 100] .= NaN
  55.450 ms (7 allocations: 9.27 MiB)

```

That’s pretty fast 🙂 so please give a complete MWE that reproduces the slow thing you are seeing…

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [December 16, 2021, 4:44pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/3 "2021-12-16T16:44:40Z")

</div>

Hi!

I wouldn’t call that super slow.

```julia
julia> x = randn((360, 180, 1200));

julia> f(x) = x[x .>100] .= NaN;

julia> @time f(x);
  0.640111 seconds (2.93 M allocations: 143.932 MiB, 1.21% gc time, 89.95% compilation time)

julia> @time f(x);
  0.063607 seconds (8 allocations: 9.274 MiB)

```

Do you see something different on your machine? Do you have an array with Integer of Float values?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [December 16, 2021, 4:52pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/4 "2021-12-16T16:52:13Z")

</div>

as a side note: using `replace!` would allocate less.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [December 16, 2021, 4:56pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/5 "2021-12-16T16:56:58Z")

</div>

I guess a relevant question might be: what is `typeof(A)`? Given that

```julia
julia> 100000002004087734272 > typemax(Int64)
true

```

maybe there’s a `BigInt` issue?

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [December 16, 2021, 4:57pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/6 "2021-12-16T16:57:01Z")

</div>

I tried the randomly generated matrix and it is indeed not that slow. However, for my own matrix, it is still taking forever.

It is a model output matrix of Float32, with missing values as 1e20.

> [@roflmaostc](#):
>
> `x[x .>100] .= NaN;`

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [December 16, 2021, 4:58pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/7 "2021-12-16T16:58:44Z")

</div>

Please show the type of your matrix, better a MWE.

If the type is not concrete like `Matrix{Float64` that’s bad.

Maybe you have `Matrix{Real} ` which is a matrix of many different types.

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [December 16, 2021, 4:59pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/8 "2021-12-16T16:59:30Z")

</div>

Thanks! Here you go:

typeof(A) = Array{Union{Missing, Float32}, 3}

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [December 16, 2021, 5:00pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/9 "2021-12-16T17:00:24Z")

</div>

I don’t know anything about NetCDF, but maybe it is a memory mapped format? You could try `collect(A)` and see if that turns it into a plain old `Matrix{Float32}`

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [December 16, 2021, 5:03pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/10 "2021-12-16T17:03:04Z")

</div>

Unfortunately, the command of `A = collect(A)` is taking forever to execute as well.

It is now done, and the type of my new A is now shown as the below:

typeof(A) = Array{Union{Missing, Float32}, 3}

I guess the issue could be related to the NCDatasets package that did not load the values out cleanly.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [December 16, 2021, 5:05pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/11 "2021-12-16T17:05:30Z")

</div>

For loops in Julia are faster than Matlab-style tricks, that Julia can also do and faster.

Try for example (_improved with input from_ @DNF):

```julia
function capnan!(A, v)
    for i in eachindex(A)
        (A[i] > v) && (A[i] = NaN)
    end
end

```

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [December 16, 2021, 5:10pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/12 "2021-12-16T17:10:08Z")

</div>

Thanks! Unfortunately, I got the below error:

```julia
LoadError: TypeError: non-boolean (Missing) used in boolean context

Stacktrace:

[1] **capnan!(** A::NCDatasets.CFVariable{Union{Missing, Float32}, 3, NCDatasets.Variable{Float32, 3, NCDataset{Nothing}}, NCDatasets.Attributes{NCDataset{Nothing}}, NamedTuple{(:fillvalue, :scale_factor, :add_offset, :calendar, :time_origin, :time_factor), Tuple{Float32, Nothing, Nothing, Nothing, Nothing, Nothing}}}, v::Int64 **)**

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 16, 2021, 5:12pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/13 "2021-12-16T17:12:08Z")

</div>

`&&` and `||` error with missing values. Actually, now I’m confused about why your original code works if you have `missing`s.

The package [MissingsAsFalse.jl](https://github.com/pdeffebach/MissingsAsFalse.jl) is helpful here.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [December 16, 2021, 5:12pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/14 "2021-12-16T17:12:53Z")

</div>

> [@leon](#):
>
> 100000002004087734272

This is not the same as `missing` value.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [December 16, 2021, 5:14pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/15 "2021-12-16T17:14:35Z")

</div>

> [@rafael.guerra](#):
>
> `@inbounds for k = 1:size(A,3), j = 1:size(A,2), i = 1:size(A,1)`

This is what `eachindex` is for😉

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [December 16, 2021, 5:14pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/16 "2021-12-16T17:14:39Z")

</div>

I keep getting the below error. Maybe Julia consider that large value as a missing value?

`unable to check bounds for indices of type Missing`

typeof(A) = Array{Union{Missing, Float32}, 3}

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 16, 2021, 5:16pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/17 "2021-12-16T17:16:25Z")

</div>

No, you have `missing` values in your array. `missing` is different from `NaN`.

```julia
julia> x = [1 2; missing 4]
2×2 Matrix{Union{Missing, Int64}}:
 1 2
  missing 4

julia> x[x .> 1] .= 100
ERROR: ArgumentError: unable to check bounds for indices of type Missing

```

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [December 16, 2021, 5:17pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/18 "2021-12-16T17:17:54Z")

</div>

I think this is the exact problem I’m having and I’m trying to replace the missing values with NaNs.

typeof(A) = Array{Union{Missing, Float32}, 3}

Any idea about how to do that? I tried the below and it did not work:  
`replace!(A, missing=>NaN);`

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 16, 2021, 5:22pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/19 "2021-12-16T17:22:18Z")

</div>

Are you trying to replace `missing` values with `NaN`? Earlier you said you were trying to replace the number `100000002004087734272` with `missing`.

Use `replace!`

```julia
julia> x = [1 2; missing 4]
2×2 Matrix{Union{Missing, Int64}}:
 1 2
  missing 4

julia> replace!(x, missing => 5)
2×2 Matrix{Union{Missing, Int64}}:
 1 2
 5 4

```

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [December 16, 2021, 5:24pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/20 "2021-12-16T17:24:55Z")

</div>

My code:

```julia
@show typeof(A);
replace!(A, missing=>NaN);
@show typeof(A);

```

Results:

```julia
typeof(A) = Array{Union{Missing, Float32}, 3}
typeof(A) = Array{Union{Missing, Float32}, 3}

```

Does that mean it did not work?

[Next page](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213.md?page=2)