# Loess with missing values

**URL:** <https://discourse.julialang.org/t/loess-with-missing-values/90921>\
**Category:** Data\
**Tags:** missing-values, loess\
**Created:** [November 27, 2022, 10:47pm UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921 "2022-11-27T22:47:11Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 27, 2022, 10:47pm UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/1 "2022-11-27T22:47:11Z")

</div>

I want to use the Loess smoothing with missing values in y (and then fill those with the loess estimate). I hope I would be able to do it with `pairwise` from `StatsBase`, but did not manage.

The code below works but feels a bit cumbersome:

```
using Loess

function loess4Miss(x, y; span=0.3)  
  ok = findall(@. !ismissing(x) & !ismissing(y))
  model = loess(disallowmissing(x[ok]), disallowmissing(y[ok]); span=span, degree=1)
  predict(model, x)
end

x = rand(100);
y = sin.(x*π) + 0.03*randn(100)
y = allowmissing(y)
y[10] = missing
loess4Miss(x,y)

```

Is there a better solution, e.g. with `pairwise`?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 27, 2022, 11:18pm UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/2 "2022-11-27T23:18:31Z")

</div>

Can you please open an issue for this in Loess.jl as I also think requiring `AbstractFloat` eltype is a bit cumbersome.

As a small comment: your code will not work when `x` has `missing` value as `predict` fail I think (not tested).

What I typically do is the following:

```julia
using DataFrames
function loess4Miss(x, y; span=0.3)  
  df = dropmissing(DataFrame(x=x, y=y, copycols=false), :y)
  model = loess(df.x, df.y; span=span, degree=1)
  predict(model, x)
end

```

(in general working on multiple columns is usually easier when using DataFrames.jl)

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 28, 2022, 9:49pm UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/3 "2022-11-28T21:49:48Z")

</div>

Thanks! - only the `df2` (the “2”) is a typo.

I also found `skipmissings`, essentially very similar, without DataFrame, but would remove also x missings

```
  function loess4Miss(x, y; span=0.3)
    xok, yok = collect.(skipmissings(x, y))
    model = loess(xok, yok; span=span, degree=1)
    Loess.predict(model, x)
  end

```

> [@bkamins](#):
>
> As a small comment: your code will not work when `x` has `missing` value as `predict` fail I think (not tested).

Yes I think so too. Since `x` was a timeseries or index, that did not matter here.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 28, 2022, 10:09pm UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/4 "2022-11-28T22:09:52Z")

</div>

`skipmissings` is similar indeed.

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [November 28, 2022, 10:16pm UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/5 "2022-11-28T22:16:46Z")

</div>

Note that if you have y with missing x and x with missing y you should really do the DataFrames version because you can be completely screwing up your pairings otherwise

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 29, 2022, 7:24am UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/6 "2022-11-29T07:24:33Z")

</div>

Sorry, I don’t get it. I thought `skipmissings` removes casewise.

I believe the problem you refer to only happens with separate `skipmissing`for each variable (of course).  
cf. [Skipmissing no working in cor function - #3 by pdeffebach](https://discourse.julialang.org/t/skipmissing-no-working-in-cor-function/71293/3)

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [November 29, 2022, 7:57am UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/7 "2022-11-29T07:57:02Z")

</div>

> [@mreichMPI-BGC](#):
>
> separate `skipmissing`for each variable (of course).

Yes correct. That’s what I meant.

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 29, 2022, 8:01am UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/8 "2022-11-29T08:01:35Z")

</div>

But I also read somewhere, that `skipmissings` (with **s** ) will be discontinued? So should one use the DataFrame way? Looks a bit like a detour, but maybe I just need to get used to it. Any “general” advice welcome, e.g. @bkamins. Btw thanks for all your helpful comments and blogs.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 29, 2022, 9:32am UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/9 "2022-11-29T09:32:09Z")

</div>

> [@mreichMPI-BGC](#):
>
> But I also read somewhere, that `skipmissings` (with **s** ) will be discontinued?

There is a discussion abut a better design, but probably `skipmissings` will stay to avoid breaking changes.

* * *

My major comment is that I do not think that you have to use DataFrames.jl, but I would recommend you to use any table-aware storage type, so that it ensures synchronization of rows between several vectors if these vectors are logically connected out of the box (as this is what you essentially need here). Such design is in my opinion cleaner conceptually.

(in general I think that is why “data frame” concept become so popular everywhere - it makes thinking about such cases easier)

---

<div class="post-metadata">

**Author:** ![ayushpatnaikgit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ayushpatnaikgit/32/25229_2.png) [@ayushpatnaikgit](https://discourse.julialang.org/u/ayushpatnaikgit)\
**Post date:** [June 28, 2023, 12:21pm UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/10 "2023-06-28T12:21:56Z")

</div>

Could this be done in StatsModels.jl? We can add `missing_action`, and it can be useful for all kinds of models. Just like R has `na.action`.

Something like:

```julia
glm(@formula(y ~ x, missing_action = :skip), df, Binomial(), LogitLink());

```

Then we can implement StatsModels API in Loess.

```julia
loess(@formula(y ~ x, missing_action = :skip), df);

```

Or, perhaps like this:

```julia
loess(@formula(y ~ x), df; missing_action = :skip);

```

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [June 29, 2023, 4:20pm UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/11 "2023-06-29T16:20:52Z")

</div>

Is the [`na.action` mechanism](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/na.fail.html) really useful in R? I’ve never seen people really using it, as the default behavior of skipping missing values seems to be enough for everyone.

In Julia, adding a `skipmissing::Bool` argument like it exists for several functions would be enough IMO. See for example [Propagation of missing values · Issue #496 · JuliaStats/GLM.jl · GitHub](https://github.com/JuliaStats/GLM.jl/issues/496).

---

<div class="post-metadata">

**Author:** ![ayushpatnaikgit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ayushpatnaikgit/32/25229_2.png) [@ayushpatnaikgit](https://discourse.julialang.org/u/ayushpatnaikgit)\
**Post date:** [June 30, 2023, 3:07pm UTC](https://discourse.julialang.org/t/loess-with-missing-values/90921/12 "2023-06-30T15:07:52Z")

</div>

skipmissing should be enough in most cases. I think it would be useful to have it in a consistent way perhaps through StatsModels.