# Summarizing grouped DataFrame where a group is entirely missing

**URL:** <https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920>\
**Category:** Data\
**Tags:** dataframes, missing-values, splitapplycombine\
**Created:** [November 27, 2022, 10:38pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920 "2022-11-27T22:38:14Z")\
**Posts on this page:** 3\
**Page:** 2

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [December 6, 2022, 10:00pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/21 "2022-12-06T22:00:32Z")

</div>

Sorry, now facing another problem… I want to have the option to calculate the linear trend of the variable (in time = index) and do it like that:

```
function trendOfVec(y::AbstractArray{Union{Missing, T}}) where T<:Number
   xy = DataFrame(; x = range(1.0, length(y)), y=y)
   lreg = lm(@formula(y~x), xy)
   coef(lreg)[2]
end

```

That does not work with `skipmissing`, in the call `fun`\circ `skipmissing` because there is no `length` of `skipmissing` type defined. On the other hand, `lm` can work with missings directly (contrary to `mean` or `median` which need the `skipmissing`). So I tried to define

```
function trendOfVec(y::Base.SkipMissing{Vector{Union{Missing, T}}}) where T<:Number
  trendOfVec(y.x)
end

```

but that does not work because the Type given to `savefun` by `skipmissing` in

```julia
@combine {outvar} = (savefun ∘ skipmissing)({vb}) 

```

is more complicated.

I ended up with

```
pre = fun == trendOfVec ? identity : skipmissing

...

{outvar} = (savefun ∘ pre)({vbl})

```

but don’t find this satisfying.

Basically I find it not very intuitive that

```julia
@combine {outvar} = (savefun ∘ skipmissing)({vb}) 

```

does not give Base.SkipMissing{Vector{Union{Missing, T}} to savefun, and don’t know how to achieve it.

Hope I am clear enough…

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [December 18, 2022, 6:29am UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/22 "2022-12-18T06:29:01Z")

</div>

I was probably not clear enough… but by browsing discourse I found by chance in a post of @pdeffebach , that `eachindex` returns the original indices, which solves the original problem:

```
 ## This works with skipmissing type
  function trendOfVec(y)
     xy = DataFrame(; x = eachindex(y) |> collect, y=y |> collect)
     lreg = lm(@formula(y~x), xy)
     coef(lreg)[2]
  end

```

Still I believe “harmonizing” the ecosystem, such that all statistical functions do the “same” when missings are present, would be very helpful. Currently `lm` skips them and `mean` etc. returns `missing`.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 18, 2022, 8:41pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/23 "2022-12-18T20:41:43Z")

</div>

You might be interested in the following stalled [PR](https://github.com/JuliaData/DataFramesMeta.jl/pull/276) to DataFramesMeta.jl

[Previous page](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920.md?page=1)