# Dropmissing(!) is undefined for GroupedDataFrames
**URL:** https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037
**Category:** Data
**Tags:** dataframes
**Created:** [April 5, 2022, 9:15am UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037 "2022-04-05T09:15:48Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@Joseph\_Bradley](https://discourse.julialang.org/u/Joseph_Bradley)
#### Post date: [April 5, 2022, 9:15am UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/1 "2022-04-05T09:15:48Z")
Hi,
The following code throws an error:
```nohighlight
df.ticker = vcat(["AAPL" for i in 1:10], ["TSLA" for i in 1:10])
df.prices = vcat(rand(19), missing)
gdf = groupby(df, :ticker)
dropmissing!(gdf)
```
The only solution I have at the moment is:
```julia
df = combine(gdf, valuecols(gdf))
dropmissing!(df)
gdf = groupby(df, :ticker)
```
Which seems awkward and inefficient. I’ve also tried broadcasting drop missing over the grouped data frame, but that seems to be deliberately disallowed.
Does anyone have a better workaround, or know why `dropmissing` isn’t defined for GDFs?
Thanks!
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 5, 2022, 9:24am UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/2 "2022-04-05T09:24:09Z")
Hi,
Why would you want to do `dropmissing` on `gdf` rather than on `df` before grouping it?
What I mean is that in your original code I would thought you would do:
```julia
df.ticker = vcat(["AAPL" for i in 1:10], ["TSLA" for i in 1:10])
df.prices = vcat(rand(19), missing)
dropmissing!(df)
gdf = groupby(df, :ticker)
```
---
### Author:  [@Joseph\_Bradley](https://discourse.julialang.org/u/Joseph_Bradley)
#### Post date: [April 5, 2022, 9:43am UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/3 "2022-04-05T09:43:37Z")
Because in reality I have an intermediate function that requires groups, that is guaranteed to return missing values:
```julia
function calculate_log_return(X)
N = size(X, 1)
out = Vector{Union{Missing, Float64}}(undef, N)
out[1] = missing
for i in 2:N
@inbounds out[i] = log(X[i]) - log(X[i - 1])
end
out
end
transform!(gdf, :prices => calculate_log_return => :log_return)
```
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 5, 2022, 9:58am UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/4 "2022-04-05T09:58:52Z")
But the result of your `transform!` function is a data frame (not a grouped data frame), so you can drop missings from it.
So while I am still unsure what you need exactly note that you can write e.g. `combine(gdf, dropmissing, ungroup=false)` and you should get a `GroupedDataFrame` with dropped missings - is this what you wanted?
---
### Author:  [@Joseph\_Bradley](https://discourse.julialang.org/u/Joseph_Bradley)
#### Post date: [April 5, 2022, 10:37am UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/5 "2022-04-05T10:37:28Z")
Ah, there’s a subtlety here that you’ve just made me realise:
`transform!` does actually return a value and this value is an ungrouped data frame, but it also leaves its argument (appropriately transform) grouped. So we get:
```julia
gdf = groupby(df, :ticker)
new_df = transform!(gdf, :prices => calculate_log_return => :log_return)
```
where `gdf` is still a grouped data frame, but `new_df` is not.
So yes, `combine(gdf, dropmissing, ungroup=false)` is exactly what I need, thanks. However, I think it should be considered a bug that an in place function ‘returns’ values of two different types!
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 5, 2022, 2:13pm UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/6 "2022-04-05T14:13:06Z")
> [@Joseph\_Bradley](#):
>
> However, I think it should be considered a bug that an in place function ‘returns’ values of two different types!
Could you please explain what you mean is a bug? `transform!` by default returns a data frame. If you passed `ungroup=false` to `transform!` it would return `GroupedDataFrame`. It is your choice what value you want returned.
The fact that `transform!` is in-place means that it updates the passed argument (and it does it).
In place function does not have to return its argument (`transform!` is consistent in what it returns with `transform`).
To show you an example from Base Julia that in-place function does not return its argument consider for example:
```julia
julia> x = [1, 2, 3]
3-element Vector{Int64}:
1
2
3
julia> pop!(x)
3
julia> x
2-element Vector{Int64}:
1
2
```
---
### Author:  [@Joseph\_Bradley](https://discourse.julialang.org/u/Joseph_Bradley)
#### Post date: [April 5, 2022, 2:51pm UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/7 "2022-04-05T14:51:54Z")
Firstly, I didn’t expect `transform!` to return _anything_ (and I don’t think it should). I get that it’s convenient (e.g. `pop!`) for in place functions for return values as well, and that there might be cases when the type of the return value is different to the type of the mutated argument. However, I feel like this should be for very special cases that are made explicit. Beyond my personal gripes with having an in-place function return a value, my reasons for calling this behaviour with `transform!` and `dropmissing!` a bug are:
1. If I `transform!` a gdf I expect to get a gdf left in place. If I also wanted to store the result (like `new_df = transform!(...)` then I would expect that to be of the same type, a gdf. I think the fact that the return value of `transform!` is different to the in-place value is a bug because it’s unexpected behaviour. If this is supposed to be a convenience trick, to let you carry on with an ungrouped df once you’re finishing `transform`ing, then I think that should be made clear in the docs. It would be quite strange if `D = mul!(A, B, C)` left `A` as a `Matrix{Float64}` but returned `D` as a `Vector{Vector{Float64}}`.
2. Either way, `dropmissing!` still isn’t defined for gdfs, which is a least a shame and would be great to see. There are lots of cases (see above) where a split-apply operation will produce `missing` values and the author of the code would like to keep a gdf with missing values removed. It looks like it’s as simple as `dropmissing!(gdf::GroupedDataFrame) = combine!(gdf, dropmissing, ungroup=false)`.
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [April 5, 2022, 3:02pm UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/8 "2022-04-05T15:02:04Z")
> [@Joseph\_Bradley](#):
>
> I think the fact that the return value of `transform!` is different to the in-place value is a bug because it’s unexpected behaviour.
One thing to note, “unexpected behavior” is different than “undocumented behavior”. Bugs are when the behavior violates the contract described in the documentation. So this technically doesn’t qualify as a bug.
One other note, having `transform!` return a value really helps with chaining.
```julia
using Chain
@chain df begin
transform!(df, :x => normalize => :y)
select(:y)
end
```
---
### Author:  [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)
#### Post date: [April 5, 2022, 3:11pm UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/9 "2022-04-05T15:11:25Z")
> [@Joseph\_Bradley](#):
>
> I think the fact that the return value of `transform!` is different to the in-place value is a bug because it’s unexpected behaviour.
I think you meant _counterintuitive_ behavior, and I kinda of agree.
---
### Author:  [@Joseph\_Bradley](https://discourse.julialang.org/u/Joseph_Bradley)
#### Post date: [April 5, 2022, 3:22pm UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/10 "2022-04-05T15:22:56Z")
@pdeffebach I’ve not used Chain before but that example looks a little odd - surely you only need `transform` (not in-place) if you want to immediately pass that on to `select`? Unless you also want the mutated DataFrame to be available after you’ve inspected `:y`? I suppose there should be a performance gain from working in place and then keep `:y` is free. I do love the dplyr syntax though! I will check it out.
@Henrique_Becker I’m not convinced that there’s much of a difference here between counterintuitive and unexpected, but sure! I think I would just point back the the `mul!` example I made earlier. And I don’t mind if it’s not officially a ‘bug’, it’s still worth pointing out and IMHO still worth changing.
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [April 5, 2022, 3:44pm UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/11 "2022-04-05T15:44:04Z")
There is a _major_ performance gain. I think there is a huge benefit in being able to write slow code, with `transform` and then with one simple change make all of it fast, by adding a `!`.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 5, 2022, 4:07pm UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/12 "2022-04-05T16:07:53Z")
Let me add some comments:
1. As I have already explained `transform` and `transform!` both can return either a data frame or a grouped data frame; the question is which of them is returned by default; this issue was discussed a lot before we made this decision and most of the users preferred data frame by default (admit that it was not an easy decision - different people have different expectations about default behavior). As @pdeffebach noted in the end you just need to learn what is the default and what is the opt-in.
2. The choice between `transform!` and `transform` when chaining is mostly the performance issue as you comment. However, exactly for this reason we do not provide `combine!` (which you have used in your example); having `combine!` most likely would not give any performance benefit for the user.
3. Now why `dropmissing!` does not work on `GroupedDataFrame`? The reason is that `dropmissing!` potentially drops rows from a `GroupedDataFrame` (even it could drop whole groups if you removed all rows from some group). This means that doing it in-place would require re-grouping the `GroupedDataFrame`. So there would not be much benefit from this in general. However, adding `dropmissing` and potentially also `dropmissing!` for `GroupedDataFrame` object could be considered. Could you please open an issue about it if you really feel they should be added so that we can discuss it? (the point is - exactly as you say it would be just a one liner `combine(gdf, dropmissing, ungroup=false)` for a non-in-place case so the question would be if it is worth adding it - but maybe it is - let us discuss)
---
### Author:  [@Joseph\_Bradley](https://discourse.julialang.org/u/Joseph_Bradley)
#### Post date: [April 5, 2022, 9:12pm UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/13 "2022-04-05T21:12:34Z")
Thanks for this! To summarise:
1. I think it’s great that `transform` offers the choice over the return type, I just think that it shouldn’t be inconsistent with it’s in-place value, or this should be made very clear.
2. Out of interest, why doesn’t `combine` benefit from being in place?
3. I see your point about the regrouping issues for `dropmissing!`. I will open an issue later this week to suggest a `dropmissing` implementation - it’s small but I do think it’s worth having, and would put DataFrames in line with dplyr:
```julia
library(tidyverse)
data <- as_tibble(airquality)
gdf <- group_by(data, Month)
gdf <- drop_na(gdf)
```
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [April 5, 2022, 9:23pm UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/14 "2022-04-05T21:23:43Z")
In DataFrames.jl with Chain this is
```julia
julia> using DataFrames, Chain;
julia> df = DataFrame(a = [1, 1, 1, 2, 2, 2], b = [4, missing, 5, 8, missing, 9]);
julia> @chain df begin
groupby(:b)
combine(dropmissing)
end
4×2 DataFrame
Row │ b a
│ Int64? Int64
─────┼───────────────
1 │ 4 1
2 │ 5 1
3 │ 8 2
4 │ 9 2
```
The distinction between grouped data frames and data frames is very clear in Julia, and that’s a really nice feature imo. A `GroupedDataFrame` is a collection of `AbstractDataFrame`s and just has the methods needed for that.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 5, 2022, 9:36pm UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/15 "2022-04-05T21:36:53Z")
> [@Joseph\_Bradley](#):
>
> why doesn’t `combine` benefit from being in place?
Because `combine` changes number of rows in general. This means that essentially implementation of `combine!` would have the following steps:
1. run `combine` and store the result on a side
2. replace the contents of the source data frame with the result of step 2.
E.g. `transform!` benefits from being in place as it does not touch the columns that are already present in the data frame (while `transform` would copy them by default).
Regarding what @pdeffebach said I think it is important to highlight that `GroupedDataFrame` is not just “data frame with a set grouping variable”, but rather it is “a collection of data frames”. The point is that you can rearrange grouped data frame or subset it. For example:
```julia
julia> df = DataFrame(a=1:3)
3×1 DataFrame
Row │ a
│ Int64
─────┼───────
1 │ 1
2 │ 2
3 │ 3
julia> gdf = groupby(df, :a)
GroupedDataFrame with 3 groups based on key: a
First Group (1 row): a = 1
Row │ a
│ Int64
─────┼───────
1 │ 1
⋮
Last Group (1 row): a = 3
Row │ a
│ Int64
─────┼───────
1 │ 3
julia> gdf[[3, 1]]
GroupedDataFrame with 2 groups based on key: a
First Group (1 row): a = 3
Row │ a
│ Int64
─────┼───────
1 │ 3
⋮
Last Group (1 row): a = 1
Row │ a
│ Int64
─────┼───────
1 │ 1
```
In other words it is better to think of `GroupedDataFrame` like a vector of data frames it is not a vector because it has some extra features that vectors do not have, e.g. you can index it with column keys:
```julia
julia> df = DataFrame(a='a':'c')
3×1 DataFrame
Row │ a
│ Char
─────┼──────
1 │ a
2 │ b
3 │ c
julia> gdf = groupby(df, :a)
GroupedDataFrame with 3 groups based on key: a
First Group (1 row): a = 'a'
Row │ a
│ Char
─────┼──────
1 │ a
⋮
Last Group (1 row): a = 'c'
Row │ a
│ Char
─────┼──────
1 │ c
julia> gdf[('b',)]
1×1 SubDataFrame
Row │ a
│ Char
─────┼──────
1 │ b
```
This is the reason why “by default” we have not added `dropmissing` for `GroupedDataFrame` (as it is collection of data frames), but as said we can discuss adding it as a convenience method.
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [April 6, 2022, 7:35am UTC](https://discourse.julialang.org/t/dropmissing-is-undefined-for-groupeddataframes/79037/16 "2022-04-06T07:35:54Z")
Also if we add `dropmissing(::GroupedDataFrame)`, we should review the API to check whether similar methods should be added for consistency. `allowmissing` comes to mind, but there are also more controversial cases like `unique`, `deleteat!`, `insertcols!` and so on. It’s hard to decide where to stop, which can be a reason not to start adding such methods in the first place. 🙂