I love Julia. The only feature that I don’t like is the treatment of missing values. It really hinders your work when working with data analysis. Comparison operators like > and ==, and aggregate functions like sum and mean will return missing if there is at least one missing value in the dataset. …

[image] Mason: skipmissing can cause changes to the algorithm which can cause catastrophic loss of precision unexpectedly. For example: This is quite surprising and should probably be filed as a bug?

I am a university teacher and was a PhD doing research and using Julia to make data analysis and I am completely 100% favorable to the current approach. Skipping missing by default is a nightmare that I do not want to deal with.

It’s not a bug, it’s the intended behaviour of mean hitting a generic fallback. There are so many different wrapper types out there in the world that specializing all of them to account for skipmissing is an unending game of whack-a-mole. That said, I’m sure if you submit a PR adding a specialized…

fair enough, I retract that still unfortunately […] in SAS and Stata, x < 100 will silently return true or false even if x is missing right, it should be false, because it is false there are many ways to be wrong. the onus is on the user to handle missing values in their data correctly, and id…

[image] dlakelan: This is quite surprising and should probably be filed as a bug? The basic issue is that sum for arrays can use pairwise summation , which has very slow (logarithmic) error accumulation but relies on random access. For a generic iterator (like what is returned by skipmissing)…

We’ve talked about this before. The options we’ve come up with are Macros to skip missings. For example, @miss Statistics.mean(::Vector) can insert skipmissing. Different statistics modules for different missing behaviors. Copy the interface of Statistics and StatsBase into MissingStats with funct…

Yeah the 2nd and 3rd approaches are quite easy to implement in a separate package. Assuming there is indeed a sizeable fraction of people who want this behavior more easily accessible, it’s even a bit strange there’s still no such package.

If you are able to work with a table structure for data science you might better off using sql within Julia. You could register your DataFrame in duckdb (I think that’s zero copy) and then query the in memory dataset using sql. The aggregation functions will ignore null (missing) values by default …

[image] stevengj: The basic issue is that sum for arrays can use pairwise summation , which has very slow (logarithmic) error accumulation but relies on random access. For a generic iterator (like what is returned by skipmissing), in contrast, sum has to loop over the contents in sequence, which…

Don’t get me wrong, I understand the usefulness of not skipping missing values. My point is that should be the option, not the default behavior. Overall, this is quite opinionated, and this post was actually part of the thread what I don’t like about Julia . In some cases, you want to ensure not add…

Why are missing values not ignored by default?

Internals & Design

dlakelan November 26, 2023, 6:13pm 25

This is quite surprising and should probably be filed as a bug?

Topic		Replies	Views
What workflows for missing values are more ergonomic in Julia? Internals & Design	2	453	November 30, 2023
How does StatsBase.skewness work? Data	29	3053	January 29, 2019
A modest `missing`s 2.0 proposal Data	20	1483	October 31, 2020
Missing or NaN General Usage	26	12831	August 1, 2018
DataFramesMeta.jl and the state of the DataFrames ecosystem Data	36	4581	April 24, 2020

Why are missing values not ignored by default?

Related topics