# Converting to an array that accepts 'missing' type

**URL:** <https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111>\
**Category:** New to Julia\
**Tags:** question, arrays, missing-values\
**Created:** [February 6, 2023, 12:10am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111 "2023-02-06T00:10:41Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![austin-putz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/austin-putz/32/2583_2.png) [@austin-putz](https://discourse.julialang.org/u/austin-putz)\
**Post date:** [February 6, 2023, 12:10am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/1 "2023-02-06T00:10:41Z")

</div>

So I have a matrix with mostly 0/1/2 values, but missing values are usually set = 5 for other programs. How can I set these values to missing? It seems simple but I cannot find an answer with julia docs ([Missing Values · The Julia Language](https://docs.julialang.org/en/v1/manual/missing/)). And searching discourse I’ve found very similar things but when I apply them it doesn’t work.

```julia
A = [ 0.0 1.0 2.0
       0.0 2.0 1.0
       1.0 5.0 1.0]

# set element 3, 2 to missing
A[3, 2] = missing # throws error

# set all elements > 2.5 = missing (should only be 5's that represent missing)
A[A .> 2.5] = missing # as one would do in R, but throws an error

```

Any help? I don’t understand this `Array{Union{Missing, String}}` notation I see in the docs or how to make a matrix accept missing values.

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [February 6, 2023, 12:16am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/2 "2023-02-06T00:16:17Z")

</div>

It seems `allowmissing` is what you’re after:

```julia
julia> A = allowmissing(A)
3×3 Matrix{Union{Missing, Float64}}:
 0.0 1.0 2.0
 0.0 2.0 1.0
 1.0 5.0 1.0

julia> A[3,2] = missing
missing

```

or since the above allocates anyway, you can go with:

```julia
julia> A = map(x -> x > 2.5 ? missing : x, A)
3×3 Matrix{Union{Missing, Float64}}:
 0.0 1.0 2.0
 0.0 2.0 1.0
 1.0 missing 1.0

```

---

<div class="post-metadata">

**Author:** ![austin-putz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/austin-putz/32/2583_2.png) [@austin-putz](https://discourse.julialang.org/u/austin-putz)\
**Post date:** [February 6, 2023, 12:20am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/3 "2023-02-06T00:20:31Z")

</div>

Thank you so much, I’m not sure why this was so difficult to find. ChatGPT kept giving me weird things and I kept going to Discourse and didn’t find what I needed. This worked great. Thanks!

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [February 6, 2023, 12:21am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/4 "2023-02-06T00:21:33Z")

</div>

After training on this forum, the next version of ChatGPT will hopefully get it.

---

<div class="post-metadata">

**Author:** ![austin-putz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/austin-putz/32/2583_2.png) [@austin-putz](https://discourse.julialang.org/u/austin-putz)\
**Post date:** [February 6, 2023, 12:48am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/5 "2023-02-06T00:48:12Z")

</div>

I hope so! It gives me a lot of wrong answers due to older versions of Julia.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [February 6, 2023, 12:58am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/6 "2023-02-06T00:58:28Z")

</div>

> [@austin-putz](#):
>
> I’m not sure why this was so difficult to find

Linking [this related post](https://discourse.julialang.org/t/how-to-replace-all-values-in-an-array-with-missing-values/75467).

We need to load Missings.jl to use `allowmissing()` (or DataFrames which seems to re-export it).

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [February 6, 2023, 1:14am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/7 "2023-02-06T01:14:08Z")

</div>

> [@Dan](#):
>
> ```julia
> 0.0 1.0 2.0
> 0.0 2.0 1.0
> 1.0 missing 1.0
> 
> ```

Why is this better than?

```julia
 0.0 1.0 2.0
 0.0 2.0 1.0
 1.0 NaN 1.0

```

NaN is a Float64 so your array will be contiguous in memory but `missing` is not a Float64 so who (me 🙂 ) knows how that array is stored.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [February 6, 2023, 9:14pm UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/8 "2023-02-06T21:14:38Z")

</div>

> [@Dan](#):
>
> `map(x -> x > 2.5 ? missing : x, A)`

Is best when you want to set `missing` by condition, not by index. This is perfectly general and works for values other than `missing` (eg `nothing`), no need to learn specialized functions like `allowmissing`.

Setting a value of potentially-different type by index is probably most straightforward with `Accessors`:

```julia
julia> using Accessors

julia> A_new = @set A[3, 2] = missing
3×3 Matrix{Union{Missing, Float64}}:
 0.0 1.0 2.0
 0.0 2.0 1.0
 1.0 missing 1.0

```

Again, perfectly general and works for any type.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [February 6, 2023, 10:05pm UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/9 "2023-02-06T22:05:25Z")

</div>

> [@aplavin](#):
>
> Is best when you want to set `missing` by condition, not by index

Perhaps something like this:

```julia
map(x -> x.I == (3,2) ? missing : A[x], CartesianIndices(A))

```

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [February 7, 2023, 12:43am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/10 "2023-02-07T00:43:36Z")

</div>

How silly is to do this:

```julia
(A = convert(Matrix{Union{Missing, eltype(A)}}, A))[3,2] = missing 

```

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [February 7, 2023, 2:09am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/11 "2023-02-07T02:09:41Z")

</div>

> [@rafael.guerra](#):
>
> `map(x -> x.I == (3,2) ? missing : A[x], CartesianIndices(A))`

This is a tangent, but the quoted statement often occurs in other contexts: A function is supplied to transform a value, but it would be super convenient to use the index as a parameter to the transformation.

Something like:

```julia
mapind((I, x) -> ( I == (3,2) || x > 2.5 ? missing : A[x] ), A)

```

Essentially,

```julia
mapind( f, A, ...) "==" map(f, enumerate(A), ...)

```

Many of the uses of `enumerate` are from exactly this case in other functions too). But `enumerate` needs to add multidim indices instead of the default linear indices (i.e. `zip(CartesianIndices(A), A)` instead of `enumerate(A)` )

The simplest solution I see is to use:

```julia
map( ((I,v),) -> ...., pairs(A) )

```

which is pretty neat. So maybe it is enough.

A MWE for the case is creating a matrix of complex values with norm relative to distance from matrix center and random phase:

```julia
N = 5
phase = rand(N,N)
map( ((I,v),) -> exp(im*v)*hypot((Tuple(I).-(N-N÷2))...), collect(pairs(phase)) )

```

but this isn’t so nice.

---

<div class="post-metadata">

**Author:** ![hexaeder](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hexaeder/32/24403_2.png) [@hexaeder](https://discourse.julialang.org/u/hexaeder)\
**Post date:** [February 7, 2023, 8:40am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/12 "2023-02-07T08:40:32Z")

</div>

> NaN is a Float64 so your array will be contiguous in memory but missing is not a Float64 so who (me 🙂 ) knows how that array is stored.

According to the [docs](https://docs.julialang.org/en/v1/manual/missing/#Arrays-With-Missing-Values), arrays with missing values are still fast and contiguous in memory, since the missing information is stored alongside the main data rather than in place.

Using NaN is a bit of a hack since thats not what NaN is supposed to represent.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [February 7, 2023, 9:08am UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/13 "2023-02-07T09:08:30Z")

</div>

> [@hexaeder](#):
>
> According to the [docs](https://docs.julialang.org/en/v1/manual/missing/#Arrays-With-Missing-Values), arrays with missing values are still fast

`missing`s are fast in the simplest cases, but generally they can have huge overhead wrt NaNs — see a simple example at [Is there any reason to use NaN instead of missing? - #4 by aplavin](https://discourse.julialang.org/t/is-there-any-reason-to-use-nan-instead-of-missing/84396/4).  
If you know that you have floating point data, nans are perfectly fine, and they are also convenient to work with in julia.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [February 7, 2023, 12:07pm UTC](https://discourse.julialang.org/t/converting-to-an-array-that-accepts-missing-type/94111/14 "2023-02-07T12:07:38Z")

</div>

> [@hexaeder](#):
>
> Using NaN is a bit of a hack since thats not what NaN is supposed to represent.

Imagine the data is to be transmitted to an external library written in C, Fortran, whatever, like a plotting lib for example. It can’t because those libs know nothing abut `missings`. So the `missings` have to be replaced by a float. That (I think) implies either a copy or memory shuffling.