Why are missing values not ignored by default?

[nalimilan]

@stevengj

The basic issue is that sum for arrays can use pairwise summation, which has very slow (logarithmic) error accumulation but relies on random access.

For a generic iterator (like what is returned by skipmissing), in contrast, sum has to loop over the contents in sequence, which accumulates errors more quickly, and the difference is quite noticeable in low precision. Similarly, mean falls back to an in-sequence loop for generic iterators.

In general, for reduce functions the order/associativity of the reduction is implementation-defined and can vary for different container types, which affects floating-point roundoff errors.

Of course, we could implement a special-case pairwise reduction for SkipMissing{<:AbstractArray} iterators, and this might be a good idea because of how often this iterator type is probably used in statistics.

Actually we already have special mapreduce methods for SkipMissing, which are be fast while still using pairwise summation (Add optimized mapreduce implementation for SkipMissing by nalimilan · Pull Request #27743 · JuliaLang/julia · GitHub). But since mean no longer uses sum, it doesn’t benefit from them. We should probably add a special method for it.