# Is it possible to reduce the allocation in replace missing value?

**URL:** <https://discourse.julialang.org/t/is-it-possible-to-reduce-the-allocation-in-replace-missing-value/79524>\
**Category:** Performance\
**Tags:** memory-allocation\
**Created:** [April 15, 2022, 10:12am UTC](https://discourse.julialang.org/t/is-it-possible-to-reduce-the-allocation-in-replace-missing-value/79524 "2022-04-15T10:12:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tiZ](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tiz/32/211800_2.png) [@tiZ](https://discourse.julialang.org/u/tiZ)\
**Post date:** [April 15, 2022, 10:12am UTC](https://discourse.julialang.org/t/is-it-possible-to-reduce-the-allocation-in-replace-missing-value/79524/1 "2022-04-15T10:12:16Z")

</div>

Hi , I wanna replace the missing value to 0 in a large vector, but I got a hugh memory allocation

for example:

```julia
A = rand(10_000_000);
@time replace!(A, -Inf => 0, Inf => 0, NaN => 0);

```

this takes only 0.065884 seconds in first run (44.28 k allocations: 2.426 MiB, 39.79% compilation time)

but if I include the missing replace :

```julia
@time replace!(A, -Inf => 0, Inf => 0, missing => 0, NaN => 0);

```

this will take 1.328083 seconds (60.00 M allocations: 2.533 GiB, 21.87% gc time)

Is it anything I wrong? Thanks.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [April 15, 2022, 10:54am UTC](https://discourse.julialang.org/t/is-it-possible-to-reduce-the-allocation-in-replace-missing-value/79524/2 "2022-04-15T10:54:39Z")

</div>

This performs much better, although I’m not sure which are the tradeoffs, or if `replace!` could do better there:

```julia
julia> @time map!(x -> (ismissing(x) | isnan(x) | isinf(x)) ? 0 : x, A, A);
  0.056996 seconds (16.95 k allocations: 961.940 KiB, 57.23% compilation time)

julia> f(x) = (ismissing(x) | isnan(x) | isinf(x)) ? 0 : x
f (generic function with 1 method)

julia> @time map!(f, A, A);
  0.034643 seconds (16.91 k allocations: 958.653 KiB, 33.26% compilation time)

julia> @time map!(f, A, A);
  0.022496 seconds

julia> @btime map!($f, A, A) setup=(A = Union{Missing,Float64}[(rand() > 0.1 ? rand() : missing) for _ in 1:10^7]) evals=1
  24.532 ms (0 allocations: 0 bytes)

```

(the compilation time in the first example is associated to the anonymous function, on every run)

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [April 15, 2022, 1:28pm UTC](https://discourse.julialang.org/t/is-it-possible-to-reduce-the-allocation-in-replace-missing-value/79524/3 "2022-04-15T13:28:20Z")

</div>

There some things that are a little strange in this MWE.

1. The vector `A` simply cannot have `-Inf`, `Inf`, or `NaN`. So you are always just testing how a pass that do not replace anything should go.
2. The `0` literal is interpreted as `Int` (which is an alias for either `Int32` or `Int64` in your system), so there is probably an automatic conversion (maybe optimized away) to the `0.0`, otherwise `replace!` should not work, as it changes the vector in place and you cannot save a `Int` in a `Vector{Float64}` without converting.
3. `rand` returns a `Vector{Float64}` such type simply cannot have `missing` values, another red flag is the number of allocations, there is no reason for `replace!` to allocate anything, I believe what is happening here is that something inside `replace!` became type-unstable because of the strange types in the pairs, ideally all pairs should have a `Float64` in the right and the left side, unless your vector is a `Vector{Float64,Missing}`.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [April 15, 2022, 2:45pm UTC](https://discourse.julialang.org/t/is-it-possible-to-reduce-the-allocation-in-replace-missing-value/79524/4 "2022-04-15T14:45:03Z")

</div>

without prejudice to all doubts about what really happens, here is another proposal

> [@lmiq](#):
>
> `repl(x) = (ismissing(x) | isnan(x) | isinf(x)) ? 0 : x`

```julia
julia> @btime replace!(repl, A);
  13.607 ms (1 allocation: 16 bytes)

```

in my laptop

```julia
julia> @btime map!($f, A, A) setup=(A = Union{Missing,Float64}[(rand() > 0.1 ? rand() : missing) for _ in 1:10^7]) evals=1;
  53.493 ms (0 allocations: 0 bytes)

```

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [April 15, 2022, 3:09pm UTC](https://discourse.julialang.org/t/is-it-possible-to-reduce-the-allocation-in-replace-missing-value/79524/5 "2022-04-15T15:09:23Z")

</div>

I guess, without checking, that the Dict that is being created by `replace` from the arguments is a `Dict{Any,Any}` when the missing is present in the example.

Probably you won’t get allocations if using interpolations:

> [@rocco\_sprmnt21](#):
>
> `@btime replace!($repl, $A);`

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [April 18, 2022, 8:25am UTC](https://discourse.julialang.org/t/is-it-possible-to-reduce-the-allocation-in-replace-missing-value/79524/6 "2022-04-18T08:25:49Z")

</div>

The problem seems to be a type-unstable for-loop inside `Base.replace_pairs!`. Using recursion instead of a loop solves the problem for this case.

```julia
julia> _new(x) = x
_new (generic function with 1 method)

julia> _new(x, p, ps...) = isequal(first(p), x) ? last(p) : _new(x, ps...)
_new (generic function with 2 methods)

julia> function Base.replace_pairs!(res, A, count::Int, old_new::Tuple{Vararg{Pair}})
           Base._replace!(res, A, count) do x
               _new(x, old_new...)
           end
       end

julia> @time replace!(A, -Inf => 0, Inf => 0, missing => 0, NaN => 0);
  0.074939 seconds (68.23 k allocations: 3.670 MiB, 31.72% compilation time)

```