# Skipmissing no working in cor function

**URL:** <https://discourse.julialang.org/t/skipmissing-no-working-in-cor-function/71293>\
**Category:** New to Julia\
**Tags:** question\
**Created:** [November 10, 2021, 11:01pm UTC](https://discourse.julialang.org/t/skipmissing-no-working-in-cor-function/71293 "2021-11-10T23:01:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jsaraviadrago](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jsaraviadrago/32/20358_2.png) [@jsaraviadrago](https://discourse.julialang.org/u/jsaraviadrago)\
**Post date:** [November 10, 2021, 11:01pm UTC](https://discourse.julialang.org/t/skipmissing-no-working-in-cor-function/71293/1 "2021-11-10T23:01:33Z")

</div>

Hi everyone,

I am trying to make a correlation in Julia but one line of missing values. (My code is below).

I use the skipmissing and the result is skipmissing(missing), if I don’t put that it says missing.

I tried the correlation function with a complete DataFrame and everything works properly.

Any help would be appreciated.

```julia
using DataFrames, Pkg, CSV, Gadfly, HypothesisTests, Statistics, Missings

df = CSV.read("/Users/home/Documents/MP blog 2021/Data/Eliminatorias/CONMEBOL/Conmebol_partidos_2022.csv", DataFrame, normalizenames=true)

skipmissing(cor(df.Remates_arco, df.Pases))

``
```

---

<div class="post-metadata">

**Author:** ![aramirezreyes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aramirezreyes/32/42573_2.png) [@aramirezreyes](https://discourse.julialang.org/u/aramirezreyes)\
**Post date:** [November 10, 2021, 11:13pm UTC](https://discourse.julialang.org/t/skipmissing-no-working-in-cor-function/71293/2 "2021-11-10T23:13:27Z")

</div>

You are applying the skipmissing too late:

You are first applying the cor, which returns missing because your dataframes have missing values. Then you are applying the skipmissing, but the only thing you have at that moment is a missing value. Hence what you are doing is `skipmissing(missing)`

If both columns have the same missing values, I suppose you could do:

`cor( skipmissing(df.Remates_arco), skipmissing(df.Pases) )`

I am not sure if that works because I don’t use Dataframes a lot, but looking at [their documentation](https://dataframes.juliadata.org/stable/man/missing/), you could also do:

```julia
newdf = dropmissing(df)
cor(newdf.Remates_arco, newdf.Pases))

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 11, 2021, 12:24am UTC](https://discourse.julialang.org/t/skipmissing-no-working-in-cor-function/71293/3 "2021-11-11T00:24:13Z")

</div>

First, you are applying `skipmissing` too late.

But even if you were to correct it, `cor` wouldn’t work. This is a long-standing annoyance.

> [@aramirezreyes](#):
>
> `cor( skipmissing(df.Remates_arco), skipmissing(df.Pases) )`

This is not a good idea, since the observations are _not_ guaranteed to be matched.

We don’t have a good solution for this at the moment. Missings.jl (which is re-exported by DataFrames) provides `skipmissings`.

```julia
julia> using Missings, Statistics

julia> x = [rand() < .2 ? missing : rand() for i in 1:10];

julia> y = [rand() < .2 ? missing : rand() for i in 1:10];

julia> sx, sy = collect.(skipmissings(x, y));

julia> cor(sx, sy)
-0.32257867573052007

```

But `skipmissings` is not guaranteed to exist in the future. It’s deliberately documented as such even though Missings.jl is past 1.0.

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [November 11, 2021, 1:54am UTC](https://discourse.julialang.org/t/skipmissing-no-working-in-cor-function/71293/4 "2021-11-11T01:54:00Z")

</div>

You can do it without the Missings package:

> pos = (.!ismissing.(x)) .& (.!ismissing.(y))  
> cor(x[pos], y[pos])

It would be more difficult to create a replacement for skipmissings if we had multiple variables instead of just two.

---

<div class="post-metadata">

**Author:** ![aramirezreyes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aramirezreyes/32/42573_2.png) [@aramirezreyes](https://discourse.julialang.org/u/aramirezreyes)\
**Post date:** [November 11, 2021, 2:00am UTC](https://discourse.julialang.org/t/skipmissing-no-working-in-cor-function/71293/6 "2021-11-11T02:00:37Z")

</div>

This one suffers from the same problem mentioned by @pdeffebach, (it doesn’t guarantee that the observations will match).

---

<div class="post-metadata">

**Author:** ![aramirezreyes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aramirezreyes/32/42573_2.png) [@aramirezreyes](https://discourse.julialang.org/u/aramirezreyes)\
**Post date:** [November 11, 2021, 4:57pm UTC](https://discourse.julialang.org/t/skipmissing-no-working-in-cor-function/71293/8 "2021-11-11T16:57:50Z")

</div>

My apologies. As far as I can see your method would work. I misread it.