# Missing or NaN Data in GLM (e.g., in DataFrame, @formula)

**URL:** <https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783>\
**Category:** Statistics\
**Tags:** glm\
**Created:** [September 10, 2018, 9:10pm UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783 "2018-09-10T21:10:14Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 10, 2018, 9:10pm UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783/1 "2018-09-10T21:10:14Z")

</div>

If a series contains missing or NaN values, R will ignore the missing observations in its `lm` command. This is convenient when one runs different models, which have different observations due to variations in whether the series with missing is included or not. (R is somewhat inconsistent, in that other functions (than lm), like mean, seem never to have heard about missing variables.)

How should end users deal with missing values in series? This is especially pertinent when they want to use formulas on data frames. Is the expectation that they create a new DataFrame for each lm?

advice appreciated.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [September 10, 2018, 11:31pm UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783/2 "2018-09-10T23:31:31Z")

</div>

Missings.jl offers some convenience functions for dealing with `missing` values. I think the general consensus is that you are responsible for handling them on your own. So if you try to `mean` something that has missing values, you get an error by default, but you can use the provided functionality (like `skipmissing`) when passing arguments to easily remove the missings.

> **[GitHub - JuliaData/Missings.jl: Missing value support for Julia](https://github.com/JuliaData/Missings.jl)**
>
> Missing value support for Julia. Contribute to JuliaData/Missings.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 11, 2018, 1:37am UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783/3 "2018-09-11T01:37:24Z")

</div>

it’s considerably more painful for new users. Fortunately, in the dataframe glm context, `completecases` can fix part of the problem. Unfortunately, it still leavies the “other” missing value NaN to be dealt with. (and completecases does not work for matrices.)

even then, every time a new model is run, it requires creating a new data frame, because the variable responsible for the missing content in one variable may or may not be in the model. I presume one needs to write a function that takes an AbstractDataFrame, copies it, and returns another sanitized DataFrame. Otherwise, I do not see how one can use the formula interface.

of course, it can be handled, but it is a roadblock compared to other stats packages I know of.

If the GLM calculations are aware of missing observations, this becomes much more convenient.

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 11, 2018, 2:19am UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783/4 "2018-09-11T02:19:24Z")

</div>

uggh, no. this does not do it. the formula interface needs to be known to the data frame sanitizer.

so, I am out of my league here.

this is really problematic for using Julia in the social sciences, where missing values are very common, and data frames can be considered a basic necessary data type. all the nice formula stuff goes out of the window.

this shortcoming puts julia at a big disadvantage relative to R, Stata, etc., in this domain.

---

<div class="post-metadata">

**Author:** ![pmarg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pmarg/32/936_2.png) [@pmarg](https://discourse.julialang.org/u/pmarg)\
**Post date:** [September 11, 2018, 5:41am UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783/5 "2018-09-11T05:41:30Z")

</div>

I agree with iwelch on this. Regression tables report the number of observations so if a researcher adds variables in steps and the number of observations falls then it is easy for the reader to understand what is going on. Whether the population is the same between regressions because of an inclusion of an additional variable that has missing values is up for debate but this is a research question that arises transparently and explicitly from the regression table.

Some people don’t just explain the use of completecases() or dropna() etc, but consider them to be the only logical thing to do if you have missing values across different observations in different columns. I believe it’s because of different practices across disciplines. In my area this is definitely not the obvious thing to do most of the time.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [September 11, 2018, 10:09am UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783/6 "2018-09-11T10:09:35Z")

</div>

> [@iwelch](#):
>
> If a series contains missing or NaN values, R will ignore the missing observations in its `lm` command.

In Julia `NaN` values are _not_ missing values. They correspond to an invalid number or to the result of an invalid computation such as `0/0`. Missing values should be represented using `missing`, and observations with a `missing` value in one of the variables are skipped automatically by GLM. For example:

```julia
data = DataFrame(X=[1,2,3,5,10], Y=[2,4,missing,3,5])
ols = lm(@formula(Y ~ X), data)
nobs(ols) # Gives 4

```

If you have a dataset with `NaN` to represent missing values, apply `replace(x, NaN => missing)` to its columns after importing it.

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 11, 2018, 5:08pm UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783/7 "2018-09-11T17:08:49Z")

</div>

I think asking users to replace NaN with missings is ok.

mea culpa on the data. see, I concluded from

```julia
julia> lm( data[:,1] ~ data[:,2] )
ERROR: MethodError: no method matching ~(::Array{Union{Missing, Int64},1}, ::Array{Union{Missing, Int64},1})

```

that `lm` was not set up for missing data. incorrect.

so, somehow there is magic in the formula/dataframe interface to lm that would be nice to back-propagate into the plain lm.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 11, 2018, 7:01pm UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783/8 "2018-09-11T19:01:42Z")

</div>

The problem with this approach is that in Julia `~` is a bitwise not. Julia has non-standard evaluation of expressions, but it is done through macros, which is why we use `@formula`.

If you are working with a dataframe, it’s not clear why you wouldn’t simply write `lm(@formula(y ~ x), data)`. Or if you want to work with objects themselves, you can always write

```julia
lm([df.x1 df.x2], df.x3)

```

There are benefits to this distinction. It means that code is more clear. For example, `lm(y ~ x)` and `lm(y ~ x, data)` would mean two very different things if you had vectors `y` and `x` defined, while also having `data.y` and `data.x` defined.

One thing you frustratingly _can’t_ do is use a Vector as your `X` in an `lm` call. Perhaps there should be an additional method that is

```julia
fit(model, X::AbstractVector, y) = fit(model, reshape(X, :, 1), y) # now X is a matrix

```

EDIT: For anyone frustrated by this, remember that `X[:,:]` will turn any `Array{T, 1}` into an `Array{T, 2}`.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [September 11, 2018, 7:19pm UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783/9 "2018-09-11T19:19:21Z")

</div>

Just a general comment: in R `NA` and `NaN` are also separate things, what is worse you are not guaranteed what you will get as a result of operating on them as R documentation explains:

> Computations involving `NaN` will return `NaN` or perhaps `NA` : which of those two is not guaranteed and may depend on the **R** platform (since compilers may re-order computations).

**R** handles this by implementing `is.na` function in the following way:

> The default method for `is.na` applied to an atomic vector returns a logical vector of the same length as its argument `x` , containing `TRUE` for those elements marked `NA` or, for numeric or complex vectors, `NaN` , and `FALSE` otherwise.

It is easy enough to get the same in Julia:

```julia
is_na(x::Number) = ismissing(x) || isnan(x)
is_na(x) = ismissing(x)

```

if you wanted it.

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 11, 2018, 9:39pm UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783/10 "2018-09-11T21:39:17Z")

</div>

the problem is that the solution is not what one would expect, if someone has tried `lm` with ordinary numeric vectors and matrices. once someone knows the solution is just to place it into a data frame and then it works, it is just fine.

there is something to be said for getting what one expects…

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 12, 2018, 2:26pm UTC](https://discourse.julialang.org/t/missing-or-nan-data-in-glm-e-g-in-dataframe-formula/14783/11 "2018-09-12T14:26:29Z")

</div>

One problem with that approach is that many people expect a variety of different things, this leads to an exponential growth of interfaces to support. Using `?lm` and then seeing that the way to get a regression with arrays is `lm(X, y)` doesn’t seem like a high barrier to entry.