# Why are missing values not ignored by default?

**URL:** <https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756>\
**Category:** Internals & Design\
**Tags:** data, missing-values\
**Created:** [November 26, 2023, 1:45am UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756 "2023-11-26T01:45:36Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![alfaromartino](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alfaromartino/32/52986_2.png) [@alfaromartino](https://discourse.julialang.org/u/alfaromartino)\
**Post date:** [November 26, 2023, 1:45am UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/1 "2023-11-26T01:45:36Z")

</div>

I love Julia. The only feature that I don’t like is the treatment of missing values. It really hinders your work when working with data analysis.

Comparison operators like \> and ==, and aggregate functions like sum and mean will return missing if there is at least one missing value in the dataset. So you need to write `mean(skipmissing(x))` rather than `mean(x)`, or `isequal(x,y)` rather than `x .== y`.

I understand that this approach was intended to ensure code safety. In fact, I think returning `NaN` when there is at least one `NaN` value in x is useful in general, as it potentially points out an unintended result. However, missing values only arise when you work in data analysis, and you always know whether there are missing values in your dataset. For me, it’s never been true that `mean(x)` returning missing is pointing out any unintended result.

Overall, I would have preferred `skipmissing` to be the default approach. Ultimately, it’s a matter of preference. However, if you’re like me, handling missing values feels like a drag after two hours of serious work, deviating your attention all the time from the actual analysis.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [November 26, 2023, 1:56am UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/2 "2023-11-26T01:56:49Z")

</div>

I agree 100% with every point you have made

beyond `mean(skipmissing(x))` it gets even more painful with `cor`, when one must write `skipmissing` twice…

I cannot help but feel that the `missing` semantics were designed by those who (while well-intentioned) do not actually do data science work themselves

---

<div class="post-metadata">

**Author:** ![liuyxpp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/liuyxpp/32/9870_2.png) [@liuyxpp](https://discourse.julialang.org/u/liuyxpp)\
**Post date:** [November 26, 2023, 2:06am UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/3 "2023-11-26T02:06:09Z")

</div>

I am kind of curious how to do mean when there are missing values. Say x has N elements and M missing values, the subset of x contains no missing values is y. Which mean do you like: sum(y) / N or sum(y) / (N-M)?

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [November 26, 2023, 2:18am UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/4 "2023-11-26T02:18:04Z")

</div>

> [@liuyxpp](#):
>
> sum(y) / (N-M)

this one of course

---

<div class="post-metadata">

**Author:** ![alfaromartino](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alfaromartino/32/52986_2.png) [@alfaromartino](https://discourse.julialang.org/u/alfaromartino)\
**Post date:** [November 26, 2023, 2:18am UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/5 "2023-11-26T02:18:42Z")

</div>

It’s not only mean, but any function that reduces a vector to a scalar.

Example:

```julia
x = [0, 1, missing]
sum(x) # this gives you `missing`
sum(skipmissing(x))

```

Similarly, some functions return error if there’s a missing value. Say you have a dataset that it loaded numbers as strings. To convert them to numbers, you have to use `passmissing`, otherwise it errors if there’s a missing value

```julia
x = ["0", "1", missing]
tryparse.(Int, x) # this errors
passmissing(tryparse).(Int, x)

```

To avoid these problems, you also end up writing a lot of code like this for a dataframe `df`, so that you work with a subset of values that are not missing:

```julia
temp = view(df, (!ismissing).(:var1), :)
#or 
temp = dropmissing(df, :var1, view=true)

```

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [November 26, 2023, 2:41pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/6 "2023-11-26T14:41:26Z")

</div>

I understand that writing this out all the time is frustrating. But if you know what semantics you need, you can convert your data to use a custom type right after loading it, and then gradually build a package with the dispatches you need as you hit those MethodErrors.  
Libraries for loading data could be made to accept a custom type for missing data.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [November 26, 2023, 3:54pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/7 "2023-11-26T15:54:04Z")

</div>

> [@simsurace](#):
>
> then gradually build a package with the dispatches you need

“sure, Julia’s behavior for basic statistical operations is atypical and clunky, but you can always rewrite your own libraries!”

I’m not trying to be too too snarky, but just imagine reading that rebuttal as a Python-wielding tourist to this thread. I’m not sure it would inspire much confidence in Julia’s commitment to make data analysis feel ergonomic.

---

<div class="post-metadata">

**Author:** ![Eben60](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eben60/32/13475_2.png) [@Eben60](https://discourse.julialang.org/u/Eben60)\
**Post date:** [November 26, 2023, 4:42pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/8 "2023-11-26T16:42:38Z")

</div>

> [@alfaromartino](#):
>
> It’s not only mean, but any function that reduces a vector to a scalar.
> 
> Example:
> 
> ```julia
> x = [0, 1, missing]
> sum(x) # this gives you `missing`
> sum(skipmissing(x))
> 
> ```

What then should in your opinion be the proper behavior of the `sum` function in this case? Silently skip the `missing`s?

---

<div class="post-metadata">

**Author:** ![mpeters2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mpeters2/32/202247_2.png) [@mpeters2](https://discourse.julialang.org/u/mpeters2)\
**Post date:** [November 26, 2023, 4:44pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/9 "2023-11-26T16:44:17Z")

</div>

How are these missing values coded? Is there a package that allows arrays with missing values? I’m curious, as I’ve always handcoded this: one array with actual data, and another coding whether an element in the first array should be used or not.

---

<div class="post-metadata">

**Author:** ![Eben60](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eben60/32/13475_2.png) [@Eben60](https://discourse.julialang.org/u/Eben60)\
**Post date:** [November 26, 2023, 5:01pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/10 "2023-11-26T17:01:02Z")

</div>

> [@mpeters2](#):
>
> How are these missing values coded? Is there a package that allows arrays with missing values?

> **[ABC of handling missing values in Julia](https://bkamins.github.io/julialang/2021/09/03/missing.html)**
>
> Introduction

> **[The Zen of Missing in Julia](https://bkamins.github.io/julialang/2022/06/17/missing.html)**
>
> Introduction

And especially in this connection may I strongly disagree with your following sweeping statement?

> [@What don't you like about Julia for "serious work"?](https://discourse.julialang.org/t/what-dont-you-like-about-julia-for-serious-work/54591/175):
>
> a culture of terrible documentation.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [November 26, 2023, 5:06pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/11 "2023-11-26T17:06:31Z")

</div>

> [@Eben60](#):
>
> Silently skip the `missing`s?

yup

---

<div class="post-metadata">

**Author:** ![Eben60](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eben60/32/13475_2.png) [@Eben60](https://discourse.julialang.org/u/Eben60)\
**Post date:** [November 26, 2023, 5:17pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/12 "2023-11-26T17:17:31Z")

</div>

No, not for me!

Especially of you sum positive values (a common case) you will silently get not only definitively wrong result, but even a wrong estimation. For an estimation I’d replace each missing by a mean _or_ median of available values.

But first of all that should be my responsibility, and the responsibility of the software is to compute correctly. Which it does now.

---

<div class="post-metadata">

**Author:** ![mpeters2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mpeters2/32/202247_2.png) [@mpeters2](https://discourse.julialang.org/u/mpeters2)\
**Post date:** [November 26, 2023, 5:17pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/13 "2023-11-26T17:17:31Z")

</div>

Thanks for the info: i didn’t know that ‘missing’ was part of the language.

I agree that having to use skipmissing() all the time would get annoying. Knowing myself, I’d probably write my own library for mean, stdev, etc. that hides all of the skipmissing() verbosity and ugliness.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [November 26, 2023, 8:48pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/14 "2023-11-26T20:48:32Z")

</div>

8 posts were split to a new topic: [Off topic asides and snark on missing’s docs](https://discourse.julialang.org/t/off-topic-asides-and-snark-on-missing-s-docs/106760)

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [November 26, 2023, 5:22pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/15 "2023-11-26T17:22:05Z")

</div>

> [@Eben60](#):
>
> you will silently get not only definitively wrong result, but even a wrong estimation.

it’s not “wrong” it’s just a design choice

sometimes the user might want to use behavior different than default design choice, in which case one can `coalesce` missings to a different value

that is indeed the user’s responsibility and default behavior should be whatever is most standard and ergonomic, and that is to skip missings

---

<div class="post-metadata">

**Author:** ![Eben60](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eben60/32/13475_2.png) [@Eben60](https://discourse.julialang.org/u/Eben60)\
**Post date:** [November 26, 2023, 5:30pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/16 "2023-11-26T17:30:53Z")

</div>

> [@adienes](#):
>
> that is indeed the user’s responsibility and default behavior should be whatever is most standard and ergonomic, and that is to skip missings

Well, that’s your priorities, your personal opinion about what is most standard, and your expectation of how the `sum` function must behave. For me, if some summands are unknown, then the sum is simply unknown as well.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [November 26, 2023, 5:34pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/17 "2023-11-26T17:34:31Z")

</div>

> [@Eben60](#):
>
> Well, that’s your priorities, your personal opinion about what is most standard, and your expectation of how the `sum` function must behave.

agreed

nonetheless, I would bet that these opinions and priorities are shared by sizeable majority of those doing data science work

especially for stuff like `cor` it quickly becomes a nightmare

note how `describe` has the nice `skipmissing` behavior

---

<div class="post-metadata">

**Author:** ![Eben60](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eben60/32/13475_2.png) [@Eben60](https://discourse.julialang.org/u/Eben60)\
**Post date:** [November 26, 2023, 5:46pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/18 "2023-11-26T17:46:57Z")

</div>

> [@adienes](#):
>
> I would bet that these opinions and priorities are shared by sizeable majority of those doing data science work

I doubt that, but could be. For me it would be another confirmation that data science is not science 😛. In any case I would bet that these opinions and priorities are **NOT** shared by sizeable majority of those doing (proper) science and engineering work.

See also the pool results in the “[The Zen of Missing in Julia](https://bkamins.github.io/julialang/2022/06/17/missing.html)”

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [November 26, 2023, 5:48pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/19 "2023-11-26T17:48:44Z")

</div>

> [@Eben60](#):
>
> doing (proper) science and engineering

this feels pretty condescending and dismissive of a ton of work by a ton of well-educated and trained professionals. not sure if you intended it to come off this way?

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [November 26, 2023, 5:56pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/20 "2023-11-26T17:56:14Z")

</div>

The solution here is pretty simple, write a macro to insert the skipmissings.

[Next page](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756.md?page=2)