# Summarizing grouped DataFrame where a group is entirely missing

**URL:** <https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920>\
**Category:** Data\
**Tags:** dataframes, missing-values, splitapplycombine\
**Created:** [November 27, 2022, 10:38pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920 "2022-11-27T22:38:14Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 27, 2022, 10:38pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/1 "2022-11-27T22:38:14Z")

</div>

Dear community,  
when I want to summarize a grouped DataFrame e.g. with `median` and a group contains only `missing`, I get an error, because `skipmissing` returns an empty array and `median(Float64[])` throws an error.

The code below works, but is there a better way than defining an extra function (`savefun`)?

```
using DataFrameMacros, Statistics, Chain

d = @chain begin
  DataFrame(x=rand(12))
  @transform :gr = @bycol repeat('A':'D'; inner=3)
  @transform :x_miss=:gr == 'A' ? missing : :x ## make one group missing completely
end

function calc(df, vbl, gr, fun)
  savefun(x) = try fun(x) catch y missing end
  outvar = string(vbl)*"_"*string(fun)
  @chain df begin
    @groupby {gr}
    @combine {outvar} = (savefun ∘ skipmissing)({string(vbl)}) 
  end
end

calc(d, :x_miss, :gr, median)

```

Thanks!

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [November 27, 2022, 10:41pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/2 "2022-11-27T22:41:12Z")

</div>

If you `skipmissing` before the `groupby`, the missing group will not be created. Is that a good answer?

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 27, 2022, 10:53pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/3 "2022-11-27T22:53:43Z")

</div>

Probably not so good for my use case, because I would not like to drop entire rows, but I might not fully understand, what you meant.

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [November 27, 2022, 11:02pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/4 "2022-11-27T23:02:40Z")

</div>

Yes, this would drop lines, as you mentioned. Since the `try`/`catch` feels bad to me, perhaps (can be adapted to macro notation):

```julia
combine(groupby(d,:gr), :x_miss => 
  (x-> all(ismissing,x) ? missing : median(x)) =>
  :x_miss_median)

```

A little verbose, but not too bad.  
Result:

```julia
4×2 DataFrame
 Row │ gr x_miss_median  
     │ Char Float64?       
─────┼──────────────────────
   1 │ A missing        
   2 │ B 0.33751
   3 │ C 0.771823
   4 │ D 0.726377

```

Or (same result):

```julia
combine(groupby(d,:gr), :x_miss => 
  ((x-> isempty(x) ? missing : median(x))∘ skipmissing) =>
  :x_miss_median)

```

ADDITION:  
Since `median` of vector containing `missing` is `missing`, why not go for the simple:

```julia
julia> combine(groupby(d,:gr), :x_miss => median => :x_miss_median)
4×2 DataFrame
 Row │ gr x_miss_median  
     │ Char Float64?       
─────┼──────────────────────
   1 │ A missing        
   2 │ B 0.33751
   3 │ C 0.771823
   4 │ D 0.726377

```

Keeping the non-`missing` median of a `missing` infested group is dangerous by itself (from data science perspective).

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [November 27, 2022, 11:50pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/5 "2022-11-27T23:50:13Z")

</div>

> [@mreichMPI-BGC](#):
>
> ```julia
> @combine {outvar} = (savefun ∘ skipmissing)({string(vbl)}) 
> 
> ```

BTW this works for me only without `{` `}` on `outvar`.

---

<div class="post-metadata">

**Author:** ![alfaromartino](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alfaromartino/32/52986_2.png) [@alfaromartino](https://discourse.julialang.org/u/alfaromartino)\
**Post date:** [November 28, 2022, 12:05am UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/6 "2022-11-28T00:05:01Z")

</div>

Not sure if this is just what you’re looking for, but you could use `dropmissing` with the option `view=true`.  
Example: suppose the following dataframe

```julia
df = DataFrame(a = [1,1,2,2,missing], b = [1,2,3,4,5])

julia> df
5×2 DataFrame
 Row │ a b     
     │ Int64? Int64 
─────┼────────────────
   1 │ 1 1
   2 │ 1 2
   3 │ 2 3
   4 │ 2 4
   5 │ missing 5

```

Then you can create a view without missing values and calculate the median

```julia
temp = dropmissing(df, view=true)
transform!(groupby(temp,:a), :b => median)

```

with output

```julia
julia> df
5×3 DataFrame
 Row │ a b b_median  
     │ Int64? Int64 Float64?  
─────┼───────────────────────────
   1 │ 1 1 1.5
   2 │ 1 2 1.5
   3 │ 2 3 3.5
   4 │ 2 4 3.5
   5 │ missing 5 missing   

```

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [November 28, 2022, 12:20am UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/7 "2022-11-28T00:20:41Z")

</div>

That’s cool. I think you get exactly what @mreichMPI-BGC meant if you setup with `df = DataFrame(a = [1,1,2,2,3], b = [1,2,3,4,missing])`

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 28, 2022, 9:28pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/8 "2022-11-28T21:28:04Z")

</div>

Thanks for all the discussion. At the end IMO only this solution works as intended:

> [@Dan](#):
>
> ```julia
> combine(groupby(d,:gr), :x_miss => 
> (x-> all(ismissing,x) ? missing : median(x)) =>
> :x_miss_median)
> 
> ```

Or in the original example:

```
d = @chain begin
      DataFrame(x=rand(12))
      @transform :gr = @bycol repeat('A':'D'; inner=3)
      @transform :x_miss=:gr == 'A' ? missing : :x
    end

function calc(df, vbl, gr, fun)
  safefun(x) = all(ismissing.(x)) ? missing : fun(x)
  outvar = string(vbl)*"_"*string(fun)
  @chain df begin
    @groupby {gr}
    @combine begin
    {outvar} = (safefun ∘ skipmissing)({string(vbl)}) ## Variant 1 with explicit function
    {outvar*"_2"} = ((x -> all(ismissing.(x)) ? missing : fun(x)) ∘ skipmissing)({string(vbl)}) ## Variant 2 with anonymous function
    end
  end
end

```

The other solutions don’t work, because I don’t want missing if only few entries are missing or because I want to `combine`, not `transform``.

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [November 28, 2022, 10:40pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/9 "2022-11-28T22:40:55Z")

</div>

Nice. I think you missed the predicate argument for `all()`, which would make it:

```julia
safefun(x) = all(ismissing, x) ? missing : fun(x)

```

which is nicer (and like my orig suggestion).  
P.S. beyond nicer, it doesn’t materialize a vector of Bools and short-circuits faster.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 29, 2022, 9:12am UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/10 "2022-11-29T09:12:18Z")

</div>

You could also define

```julia
safe(fun) = x -> (all(ismissing, x) ? missing : fun(x))

```

which you can then apply to any function to create a safe version of it. So you could call `safe(median)(vec)` or `safe(mean)(vec)`.

As a side note, you don’t need to do `{string(vbl)}`, you don’t gain anything from `vbl` being turned into a string if it’s a symbol. And I would probably find it confusing if other objects like `Int`s were being turned into strings such that `vbl = 1` would not return the first column but a column named `"1"`.

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 29, 2022, 9:22am UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/11 "2022-11-29T09:22:45Z")

</div>

Great! I was thinking of that, coming from Tidyverse: [Capture side effects. — safely • purrr](https://purrr.tidyverse.org/reference/safely.html) .

Would it be possible to propagate arbitrary keyword arguments to fun?

```julia
safe(fun, kwargs...) = x -> (all(ismissing, x) ? missing : fun(x, kwargs...))

```

In R one could use `...` often for this. Sorry, hve no code editor open.

Regarding the string “thing” - thanks, need to check why I thought I needed it. Maybe a leftover, from when I wanted to manipulate the name before.

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 29, 2022, 9:25am UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/12 "2022-11-29T09:25:03Z")

</div>

Btw I believe making such convenience functions available in a high-level package, would probably increase the user base of Julia a lot. Basically transferring R `purrr` to Juila…

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 29, 2022, 9:33am UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/13 "2022-11-29T09:33:44Z")

</div>

Yes, but you need a `;` to signal that you want keyword arguments, not trailing positional:

```julia
safe(fun) = (x; kwargs...) -> (all(ismissing, x) ? missing : fun(x; kwargs...))

```

I have not used `purrr` but my experience with Julia has been that after learning the primitives, the need for convenience functions is reduced because they are usually not that hard to build on the fly. Of course, if one keeps redefining the same helpers, a package would be better. The space of possible helper functions is just very large, and if people don’t know that one specific function exists in a package, they will redefine it anyway.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 29, 2022, 9:35am UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/14 "2022-11-29T09:35:18Z")

</div>

I have opened [Consider allowing default in quantile and median · Issue #132 · JuliaStats/Statistics.jl · GitHub](https://github.com/JuliaStats/Statistics.jl/issues/132) to discuss how these cases should be handled in Statistics.jl.

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 29, 2022, 1:09pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/15 "2022-11-29T13:09:25Z")

</div>

I really only have the user perspective but IMO for many people the high-level, well-structured, consistent and intuitive functions in the `tidyverse` really made a difference. Of course (and that’s nice) many exist in Julia already natively, but still…

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 29, 2022, 2:04pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/16 "2022-11-29T14:04:31Z")

</div>

How does `purrr` solve this problem? The functionality of `purrr` is pretty much fully captured by `map` and `broadcast` in Julia. It doesn’t help handling `missing`s iirc.

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 30, 2022, 12:26pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/17 "2022-11-30T12:26:01Z")

</div>

Yes, that what I meant with

> [@mreichMPI-BGC](#):
>
> Of course (and that’s nice) many exist in Julia already natively,

Just in `purrr` exist a some “higher level” convenience functions, such a `safely`, `possibly` and `quietly` or `map_df` etc. Cf. [Wrap a function to capture errors — safely • purrr](https://purrr.tidyverse.org/reference/safely.html?q=safely#null).

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 30, 2022, 12:32pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/18 "2022-11-30T12:32:48Z")

</div>

That brings me to the question whether something like

```
function safe(fun) 
   (x; kwargs...) -> try fun(x; kwargs...) catch y missing end
end

```

Is something “good”. The advantage is that it would also catch other issues than “just” the “all-missing” problem.

What are disadvantages? Performance? Type-stable this should be, if `fun` is Type-stable?  
From a puristic developing point of view probably not good style, but for some “big” data science tasks at least convenient.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 30, 2022, 4:07pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/19 "2022-11-30T16:07:33Z")

</div>

Try catch comes with a performance penalty as far as I know, also you’d catch any sort of error with this, even plain bugs like UndefVarErrors. So usually it’s not a good idea.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 30, 2022, 4:15pm UTC](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920/20 "2022-11-30T16:15:27Z")

</div>

@mreichMPI-BGC - see discussion in [Consider allowing default in quantile and median · Issue #132 · JuliaStats/Statistics.jl · GitHub](https://github.com/JuliaStats/Statistics.jl/issues/132) (and maybe comment there what you think from user’s perspective). That discussion is exactly about how to design such things correctly (by correctly I mean to e.g. avoid exceptions when indeed they should be avoided, but at the same time not cover exceptions that you want to be raised unconditionally like `OutOfMemory` exception, and at the same time ensure that operations would be still fast)

[Next page](https://discourse.julialang.org/t/summarizing-grouped-dataframe-where-a-group-is-entirely-missing/90920.md?page=2)