I love Julia. The only feature that I don’t like is the treatment of missing values. It really hinders your work when working with data analysis. Comparison operators like > and ==, and aggregate functions like sum and mean will return missing if there is at least one missing value in the dataset. …

It’s not only mean, but any function that reduces a vector to a scalar. Example: x = [0, 1, missing] sum(x) # this gives you `missing` sum(skipmissing(x)) Similarly, some functions return error if there’s a missing value. Say you have a dataset that it loaded numbers as strings. To c…

I understand that writing this out all the time is frustrating. But if you know what semantics you need, you can convert your data to use a custom type right after loading it, and then gradually build a package with the dispatches you need as you hit those MethodErrors. Libraries for loading data c…

[image] simsurace: then gradually build a package with the dispatches you need “sure, Julia’s behavior for basic statistical operations is atypical and clunky, but you can always rewrite your own libraries!” I’m not trying to be too too snarky, but just imagine reading that rebuttal as a Pyt…

[image] alfaromartino: It’s not only mean, but any function that reduces a vector to a scalar. Example: x = [0, 1, missing] sum(x) # this gives you `missing` sum(skipmissing(x)) What then should in your opinion be the proper behavior of the sum function in this case? Silently s…

How are these missing values coded? Is there a package that allows arrays with missing values? I’m curious, as I’ve always handcoded this: one array with actual data, and another coding whether an element in the first array should be used or not.

[image] mpeters2: How are these missing values coded? Is there a package that allows arrays with missing values? ABC of handling missing values in Julia Introduction The Zen of Missing in Julia Introduction An…

[image] Ben Elkin: Silently skip the missings? yup

No, not for me! Especially of you sum positive values (a common case) you will silently get not only definitively wrong result, but even a wrong estimation. For an estimation I’d replace each missing by a mean or median of available values. But first of all that should be my responsibility, and th…

Thanks for the info: i didn’t know that ‘missing’ was part of the language. I agree that having to use skipmissing() all the time would get annoying. Knowing myself, I’d probably write my own library for mean, stdev, etc. that hides all of the skipmissing() verbosity and ugliness.

[image] Ben Elkin: you will silently get not only definitively wrong result, but even a wrong estimation. it’s not “wrong” it’s just a design choice sometimes the user might want to use behavior different than default design choice, in which case one can coalesce missings to a different valu…

Why are missing values not ignored by default?

Internals & Design

Eben60 November 26, 2023, 5:01pm 10

And especially in this connection may I strongly disagree with your following sweeping statement?

Topic		Replies	Views
What workflows for missing values are more ergonomic in Julia? Internals & Design	2	447	November 30, 2023
How does StatsBase.skewness work? Data	29	3029	January 29, 2019
A modest `missing`s 2.0 proposal Data	20	1477	October 31, 2020
Missing or NaN General Usage	26	12808	August 1, 2018
DataFramesMeta.jl and the state of the DataFrames ecosystem Data	36	4551	April 24, 2020

Why are missing values not ignored by default?

Related topics