# Summarize all or selected columns in grouped DataFrame

**URL:** <https://discourse.julialang.org/t/summarize-all-or-selected-columns-in-grouped-dataframe/90509>\
**Category:** Data\
**Tags:** dataframes, dataframesmeta\
**Created:** [November 19, 2022, 5:20pm UTC](https://discourse.julialang.org/t/summarize-all-or-selected-columns-in-grouped-dataframe/90509 "2022-11-19T17:20:13Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 19, 2022, 5:20pm UTC](https://discourse.julialang.org/t/summarize-all-or-selected-columns-in-grouped-dataframe/90509/1 "2022-11-19T17:20:13Z")

</div>

Dear community,

how can I summarize all columns of a GroupedDataFrame with DataFramesMeta?

Like this in DataFrames:

```julia
df = DataFrame(rand(20,5), :auto)
df[!, :gr] = repeat(1:4; inner=5)
dsum=combine(groupby(df, :gr), names(df) .=> (x->sum(log.(x))) .=> names(df).*"_logsum" )

```

Additionally (both for DataFrames and Meta), how could I select columns by type for a summary (e.g. if char would not work woth log und sum).  
Basically I am looking for equivalents of TidyR’s `summarize_if` and `summarize_at` .

New to Julia, I am happy for any hints beyond the standard documentation.

Thanks!

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 19, 2022, 5:43pm UTC](https://discourse.julialang.org/t/summarize-all-or-selected-columns-in-grouped-dataframe/90509/2 "2022-11-19T17:43:00Z")

</div>

You can do `name(df, Number)` to get all numeric columns.

```julia
julia> df = DataFrame(a = [1], b = ["a string"]);

julia> combine(df, names(df, Number) => (t -> sum(log.(t))))
1×1 DataFrame
 Row │ a_function 
     │ Float64    
─────┼────────────
   1 │ 0.0

```

You can also pass custom functions to `describe`

```julia
julia> df = DataFrame(a = [1], b = ["a string"])
1×2 DataFrame
 Row │ a b        
     │ Int64 String   
─────┼─────────────────
   1 │ 1 a string

julia> describe(df, (t -> sum(log.(t))) => :sum_log)
2×2 DataFrame
 Row │ variable sum_log 
     │ Symbol Union…  
─────┼───────────────────
   1 │ a 0.0
   2 │ b                 

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 19, 2022, 6:46pm UTC](https://discourse.julialang.org/t/summarize-all-or-selected-columns-in-grouped-dataframe/90509/3 "2022-11-19T18:46:08Z")

</div>

DataFramesMeta.jl was designed with intention of explicit passing of column names. If you want, you can pass arbitrarily complex expression to it also escaping it with `$` (though it is useful only in complex cases - here it is just a noise and probably using `combine` is better):

```julia
@combine(groupby(df, :gr), $(names(df) .=> (x->sum(log.(x))) .=> x -> x * "_logsum" ))

```

(note that it also shows you that you can pass a function that renames your input columns).

As @pdeffebach commented column selection having a specified type is achieved using the `names` function.

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 19, 2022, 7:29pm UTC](https://discourse.julialang.org/t/summarize-all-or-selected-columns-in-grouped-dataframe/90509/4 "2022-11-19T19:29:32Z")

</div>

Thanks a lot! - I learned at least 5 helpful things, also the hint with the function to rename was very useful.

I should have looked at the `names` documentation more closely. The possibilities with using `eachcol` seem powerful. More than R?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 19, 2022, 10:12pm UTC](https://discourse.julialang.org/t/summarize-all-or-selected-columns-in-grouped-dataframe/90509/5 "2022-11-19T22:12:32Z")

</div>

In [add an option to pass multiple column selectors to names by bkamins · Pull Request #3224 · JuliaData/DataFrames.jl · GitHub](https://github.com/JuliaData/DataFrames.jl/pull/3224) we are discussing how to improve this functionality in the future.

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 20, 2022, 6:37am UTC](https://discourse.julialang.org/t/summarize-all-or-selected-columns-in-grouped-dataframe/90509/6 "2022-11-20T06:37:45Z")

</div>

Thanks! - I do like your suggestion: [add an option to pass multiple column selectors to names by bkamins · Pull Request #3224 · JuliaData/DataFrames.jl · GitHub](https://github.com/JuliaData/DataFrames.jl/pull/3224#issuecomment-1318217535)

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 20, 2022, 9:06am UTC](https://discourse.julialang.org/t/summarize-all-or-selected-columns-in-grouped-dataframe/90509/7 "2022-11-20T09:06:57Z")

</div>

You could also have a look at [GitHub - jkrumbiegel/DataFrameMacros.jl: Macros that simplify working with DataFrames.jl](https://github.com/jkrumbiegel/Dataframemacros.jl) which can run expressions over column sets, in your case that would be `@combine(gdf, "{}_logsum" = sum(log.({All()})))` I think.

---

<div class="post-metadata">

**Author:** ![mreichMPI-BGC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mreichmpi-bgc/32/43775_2.png) [@mreichMPI-BGC](https://discourse.julialang.org/u/mreichMPI-BGC)\
**Post date:** [November 20, 2022, 4:32pm UTC](https://discourse.julialang.org/t/summarize-all-or-selected-columns-in-grouped-dataframe/90509/8 "2022-11-20T16:32:42Z")

</div>

It seems df-meta and df-macro are exclusive, i.e. one cannot use them together?  
Nice of df-meta IMO is that is it not always rowwise, allowing something like `x.-mean(x)`,  
while what you describe and the `@group_by(iseven(:b))` is nice in df-macro.

If they are currently exclusive, is there a chance to merge them?

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 20, 2022, 6:56pm UTC](https://discourse.julialang.org/t/summarize-all-or-selected-columns-in-grouped-dataframe/90509/9 "2022-11-20T18:56:20Z")

</div>

You can switch to by column as well in DataFrameMacros, it’s kind of the other way around because I found in my own use, transform style commands use row wise more often.

You cannot merge them really as they make some fundamentally different choices in their implementation, even though the surface syntax is similar (as both just use the DataFrames verbs as macros). I think one of the biggest differences is that everything in DataFrameMacros is automatically broadcast which gives you these multi column operations.