# Why are missing values not ignored by default?

**URL:** <https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756>\
**Category:** Internals & Design\
**Tags:** data, missing-values\
**Created:** [November 26, 2023, 1:45am UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756 "2023-11-26T01:45:36Z")\
**Posts on this page:** 1\
**Showing post:** 35

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [November 26, 2023, 8:58pm UTC](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756/35 "2023-11-26T20:58:02Z")

</div>

> [@stevengj](#):
>
> The basic issue is that `sum` for arrays can use [pairwise summation](https://en.wikipedia.org/wiki/Pairwise_summation), which has very slow (logarithmic) error accumulation but relies on random access.
> 
> For a generic iterator (like what is returned by `skipmissing`), in contrast, `sum` has to loop over the contents in sequence, which accumulates errors more quickly, and the difference is quite noticeable in low precision. Similarly, `mean` falls back to an in-sequence loop for generic iterators.
> 
> In general, for `reduce` functions the order/associativity of the reduction is implementation-defined and can vary for different container types, which affects floating-point roundoff errors.
> 
> Of course, we could implement a special-case pairwise reduction for `SkipMissing{<:AbstractArray}` iterators, and this might be a good idea because of how often this iterator type is probably used in statistics.

Actually we already have special `mapreduce` methods for `SkipMissing`, which are be fast while still using pairwise summation ([Add optimized mapreduce implementation for SkipMissing by nalimilan · Pull Request #27743 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/27743)). But since `mean` no longer uses `sum`, it doesn’t benefit from them. We should probably add a special method for it.

---

_[View the full topic](https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756)._