# Replacing missing values in a matrix is super slow

**URL:** <https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213>\
**Category:** General Usage\
**Tags:** question\
**Created:** [December 16, 2021, 4:22pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213 "2021-12-16T16:22:36Z")\
**Posts on this page:** 19\
**Page:** 2

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 16, 2021, 5:30pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/21 "2021-12-16T17:30:52Z")

</div>

It did work. `replace!` changes the array in-place. This means it doesn’t change it’s type. Use `replace` (with no `!`), to make a new array that is just `Array{Float32, 3}`

```julia
julia> replace(x, missing => 5)
2×2 Matrix{Int64}:
 1 2
 5 4

```

but be warned, this matrix can no longer hold `missing`s. So you won’t be able to do your imputation `100000002004087734272 => missing` in-place after.

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [December 16, 2021, 5:30pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/22 "2021-12-16T17:30:54Z")

</div>

Note that this is not the problem here, and your approach is even slower and more verbose 😃

```julia
julia> function capnan!(A, v)
           for i in eachindex(A)
               (A[i] > v) && (A[i] = NaN)
           end
       end
capnan! (generic function with 1 method)

julia> f(x, v) = x[x .> v] .= NaN;

julia> x = randn((360, 180, 1200)) * 1000000;

julia> using BenchmarkTools

julia> @btime capnan!($x, 100);
  92.596 ms (0 allocations: 0 bytes)

julia> @btime f($x, 100);
  67.427 ms (7 allocations: 9.27 MiB)

julia> @btime replace!(e -> e > 100 ? NaN : e, x);
55.464 ms (1 allocation: 16 bytes)

```

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [December 16, 2021, 5:31pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/23 "2021-12-16T17:31:34Z")

</div>

Dear All,

Many thanks for your big help. I did the below after replacing the missing values and now it is working! Hooray

`replace!(A, missing => NaN)`

`A = convert.(Float64, A)`

`typeof(A) = Array{Float64, 3}`

BTW, it is blazingly fast as well.

Happy holidays!

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [December 16, 2021, 5:35pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/24 "2021-12-16T17:35:33Z")

</div>

Strange, benchmarked it here to be about 2 times faster. Julia 1.7, Win11. Will check again later. It might have to do with inplace function benchmarking.

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [December 16, 2021, 5:36pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/25 "2021-12-16T17:36:15Z")

</div>

Interesting

```julia
julia> versioninfo()
Julia Version 1.7.0
Commit 3bf9d17731 (2021-11-30 12:12 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 4

```

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [December 16, 2021, 6:11pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/26 "2021-12-16T18:11:59Z")

</div>

> [@leon](#):
>
> A = convert.(Float64, A)

Did you intend to convert it to `Float64` instead of `Float32`? Otherwise it would be better to just directly do

```julia
replace(A, missing=>NaN) #or NaN32? 

```

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [December 16, 2021, 6:26pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/27 "2021-12-16T18:26:08Z")

</div>

they seem to be doing pretty much the same thing:

```julia
julia> @benchmark begin replace!(A, missing => NaN); A = convert.(Float64, A) end setup=(A=rand([missing, 0.3], 1000, 1000))
BenchmarkTools.Trial: 375 samples with 1 evaluation.
 Range (min … max): 3.881 ms … 5.811 ms ┊ GC (min … max): 0.00% … 21.07%
 Time (median): 3.936 ms ┊ GC (median): 0.00%
 Time (mean ± σ): 4.467 ms ± 755.305 μs ┊ GC (mean ± σ): 5.62% ± 7.53%

  ▆█                                                           
  ██▅▃▂▃▁▁▁▁▁▁▁▂▁▃▂▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁▃▃▂▃▃▂▂▁▁▁▁▁▁▁▃▇▆▃ ▂
  3.88 ms Histogram: frequency by time 5.7 ms <

 Memory estimate: 7.63 MiB, allocs estimate: 2.

julia> @benchmark begin A=replace(A, missing => NaN); end setup=(A=rand([missing, 0.3], 1000, 1000))
BenchmarkTools.Trial: 391 samples with 1 evaluation.
 Range (min … max): 3.244 ms … 5.190 ms ┊ GC (min … max): 0.00% … 10.32%
 Time (median): 3.306 ms ┊ GC (median): 0.00%
 Time (mean ± σ): 3.823 ms ± 763.539 μs ┊ GC (mean ± σ): 6.61% ± 8.74%

  ▅█▅▁ ▂ ▄▄▁  
  ████▁▅▁▁▁▄▁▁▁▁▁▁▁▁▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▆█▇█▇▅▄▁▁▁▁▁▁▁▁▁▁▁███ ▆
  3.24 ms Histogram: log(frequency) by time 5.11 ms <

 Memory estimate: 7.63 MiB, allocs estimate: 2.

```

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [December 16, 2021, 6:29pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/28 "2021-12-16T18:29:24Z")

</div>

And then one should choose the simpler solution. But they’re not quite the same, considering the output type.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [December 16, 2021, 9:45pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/29 "2021-12-16T21:45:39Z")

</div>

> [@roflmaostc](#):
>
> Note that this is not the problem here, and your approach is even slower and more verbose

@roflmaostoc, as a paranthesis to this post and in order to check the efficiency of the proposed `for` loop compared to matlab style bitmap indexing, I performed the following benchmarks for inplace functions using `setup`:

```julia
capnan0!(A,v) = A[A .> v] .= NaN

function capnan1!(A, v)
    for i in eachindex(A)
        (A[i] > v) && (A[i] = NaN)
    end
end

using BenchmarkTools
@btime capnan0!(A, 100) setup=(A=200*rand(360,180,1200)) evals=1 # 180 ms (8 allocations: 305 MiB)
@btime capnan1!(A, 100) setup=(A=200*rand(360,180,1200)) evals=1 # 44 ms (0 allocations: 0 bytes)

```

As you can see, the `for` loop is ~4x faster, assuming the benchmarks are done correctly.

> **see versioninfo() details here**
>
> julia\> versioninfo()  
> Julia Version 1.7.0  
> Commit 3bf9d17731 (2021-11-30 12:12 UTC)  
> Platform Info:  
> OS: Windows (x86\_64-w64-mingw32)  
> CPU: Intel(R) Core™ i7-1065G7 CPU @ 1.30GHz  
> WORD\_SIZE: 64  
> LIBM: libopenlibm  
> LLVM: libLLVM-12.0.1 (ORCJIT, icelake-client)  
> Environment:  
> JULIA\_PKG\_USE\_CLI\_GIT = true  
> JULIA\_STACKFRAME\_FUNCTION\_COLOR = blue  
> JULIA\_WARN\_COLOR = cyan  
> JULIA\_EDITOR = code.cmd -g  
> JULIA\_NUM\_THREADS = 8

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [December 16, 2021, 10:56pm UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/30 "2021-12-16T22:56:47Z")

</div>

I see what was going on:  
Previous benchmarks were affected by branch prediction (or stuff like this) since from the second pass on, the function was not doing something effectively (in place update).

```julia
julia> Ac=copy(A); @time capnan0!(Ac, 100); @time capnan0!(Ac, 100); @time capnan0!(Ac, 100);
  0.264360 seconds (9 allocations: 305.862 MiB)
  0.066205 seconds (8 allocations: 9.274 MiB)
  0.066188 seconds (8 allocations: 9.274 MiB)

julia> Ac=copy(A); @time capnan1!(Ac, 100); @time capnan1!(Ac, 100); @time capnan1!(Ac, 100);
  0.062892 seconds
  0.050146 seconds
  0.053584 seconds

```

But still, I’m confused why the first call allocates ` 305.862 MiB` and further calls not 😕

I mean, it make sense that there is one allocation for the bit array

```julia
julia> x = A1 .> 100;

julia> varinfo(r"x")
  name size summary                 
  –––– ––––––––– ––––––––––––––––––––––––
  x 9.270 MiB 360×180×1200 BitArray{3}

```

But why another `~300MiB`?

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [December 17, 2021, 12:17am UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/32 "2021-12-17T00:17:12Z")

</div>

> [@roflmaostc](#):
>
> But still, I’m confused why the first call allocates ` 305.862 MiB` and further calls not 😕

For the same reason it takes about 4x the time of the other runs. It is counting both time and allocations from the compilation.

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [December 17, 2021, 12:18am UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/33 "2021-12-17T00:18:54Z")

</div>

> [@roflmaostc](#):
>
> `Ac=copy(A); @time capnan0!(Ac, 100); @time capnan0!(Ac, 100); @time capnan0!(Ac, 100);`

It’s definitely not compilation.  
See

```julia
julia> Ac=copy(A); @time capnan0!(Ac, 100); @time capnan0!(Ac, 100); @time capnan0!(Ac, 100);
  0.548873 seconds (9 allocations: 305.963 MiB)
  0.099174 seconds (8 allocations: 9.274 MiB)
  0.084230 seconds (8 allocations: 9.274 MiB)

julia> Ac=copy(A); @time capnan0!(Ac, 100); @time capnan0!(Ac, 100); @time capnan0!(Ac, 100);
  0.525876 seconds (9 allocations: 305.963 MiB, 15.80% gc time)
  0.068308 seconds (8 allocations: 9.274 MiB)
  0.073729 seconds (8 allocations: 9.274 MiB)

```

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [December 17, 2021, 12:25am UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/34 "2021-12-17T00:25:28Z")

</div>

This agrees with my findings that the for loops solution is way faster. For example in this function I left a note that it’s 5x faster (seeking for zeros)

[https://github.com/GenericMappingTools/GMT.jl/blob/master/src/common\_options.jl#L3924](https://github.com/GenericMappingTools/GMT.jl/blob/master/src/common_options.jl#L3924)

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [December 17, 2021, 12:26am UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/35 "2021-12-17T00:26:02Z")

</div>

I see now, sorry for misunderstanding. The pattern evokes some lazy behaviour, maybe copy is not really doing the copy until the object is forced to? Or the OS is giving a pointer and a promise of memory but does allocate it until use? (I had seen this behavior a lot in C++ with very large arrays of undefined values.) The problem of this last hypothesis is that the `Array` has values inside already, no?

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [December 17, 2021, 12:29am UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/36 "2021-12-17T00:29:16Z")

</div>

Yeah, maybe

But, that memory should be already allocated and really be there

```julia
julia> A .= 200*rand(360,180,1200);;

julia> A .+= 1;

julia> A .+= 1.0;

julia> @time capnan0!(A, 100); @time capnan0!(A, 100); @time capnan0!(A, 100);
  0.267676 seconds (9 allocations: 311.902 MiB)
  0.072806 seconds (8 allocations: 9.274 MiB)
  0.074731 seconds (8 allocations: 9.274 MiB)

julia> @time capnan0!(A, 100); @time capnan0!(A, 100); @time capnan0!(A, 100); # values are changed
  0.072465 seconds (8 allocations: 9.274 MiB)
  0.072763 seconds (8 allocations: 9.274 MiB)
  0.072659 seconds (8 allocations: 9.274 MiB)

```

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [December 17, 2021, 12:32am UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/37 "2021-12-17T00:32:53Z")

</div>

Doesn’t the first run perform all the assignments by replacing the values with NaNs and the following runs have “nothing” to do?

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [December 17, 2021, 12:36am UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/38 "2021-12-17T00:36:47Z")

</div>

But even the first run should not allocate an array of the full size.  
Everything is done in-place.

See this, that works but is doing the same!

```julia
julia> capnan0!(A,B,v) = B[A .> v] .= NaN
capnan0! (generic function with 2 methods)

julia> A = randn((128, 512, 256));

julia> B = copy(A);

julia> capnan0!(A, B, 100);

julia> @time capnan0!(A, B, 100); A .= B; @time capnan0!(A, B, 100); A .= B; @time capnan0!(A, B, 100); #
  0.018046 seconds (8 allocations: 2.004 MiB)
  0.014085 seconds (8 allocations: 2.004 MiB)
  0.014415 seconds (8 allocations: 2.004 MiB)

```

There are some [aliasing bugs](https://discourse.julialang.org/t/memory-allocation-with-view-and-array-assignment/67109) which might occur also here?

EDIT: The mechanism maybe works in the second run since we do no assignments then.

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [December 17, 2021, 12:37am UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/39 "2021-12-17T00:37:43Z")

</div>

Yes, but the problem is, why when you refill the `Array` with random values the first run takes a lot of memory again? What is the extra 300MiB allocation?

```julia
julia> A = 200 .* rand(360,180,1200);

julia> @time capnan0!(A, 100); @time capnan0!(A, 100); @time capnan0!(A, 100);
  0.733982 seconds (2.07 M allocations: 398.443 MiB)
  0.062742 seconds (8 allocations: 9.274 MiB)
  0.062664 seconds (8 allocations: 9.274 MiB)

julia> A .= 200 .* rand(360,180,1200);

julia> @time capnan0!(A, 100); @time capnan0!(A, 100); @time capnan0!(A, 100);
  0.266805 seconds (9 allocations: 305.897 MiB, 17.05% gc time)
  0.063220 seconds (8 allocations: 9.274 MiB)
  0.066316 seconds (8 allocations: 9.274 MiB)

julia> capnan0!(A,v) = A[A .> v] .= NaN;

julia> A .= 200 .* rand(360,180,1200);

julia> @time capnan0!(A, 100); @time capnan0!(A, 100); @time capnan0!(A, 100);
  0.231621 seconds (9 allocations: 305.903 MiB)
  0.063112 seconds (8 allocations: 9.274 MiB)
  0.062455 seconds (8 allocations: 9.274 MiB)

```

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [December 17, 2021, 12:50am UTC](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213/40 "2021-12-17T00:50:46Z")

</div>

I created now [this issue](https://github.com/JuliaLang/julia/issues/43442).

Thanks for helping 🙂

[Previous page](https://discourse.julialang.org/t/replacing-missing-values-in-a-matrix-is-super-slow/73213.md?page=1)