# A minor group-by benchmark - DataFrames.jl plenty fast
**URL:** https://discourse.julialang.org/t/a-minor-group-by-benchmark-dataframes-jl-plenty-fast/45622
**Category:** General Usage
**Created:** [August 27, 2020, 1:56pm UTC](https://discourse.julialang.org/t/a-minor-group-by-benchmark-dataframes-jl-plenty-fast/45622 "2020-08-27T13:56:22Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [August 27, 2020, 1:56pm UTC](https://discourse.julialang.org/t/a-minor-group-by-benchmark-dataframes-jl-plenty-fast/45622/1 "2020-08-27T13:56:22Z")
I once made a FastGroupBy.jl to do some fast group by but now DataFrames.jl is plenty fast by itself!
DataFrames.jl has done lots of optimizations, I wonder what’s been done.
[Database-like ops benchmark](https://h2oai.github.io/db-benchmark/) is showing really promising stuff about DataFrames.jl’s performance.
For 50G size data, DataFrames.jl has some catching up to do with data.table.
Next stop fast joins!
PS as usually, Query.jl’s performance would potentially be an issue with large datasets. That’s the reason why I don’t use it still.
> **\*\*Expand to see code\*\***
>
> using DataFrames, Statistics
>
> using BenchmarkTools
>
> using Pipe
>
> using DataFramesMeta
>
> using FastGroupBy
>
> using Query
>
> using Plots
>
> function plot\_bench(df; title = “”)
>
> ```
> time2 = @belapsed @pipe df |>
>
> groupby(_, :a) |>
>
> combine(_, meanb = :b => mean) # 83ms
>
> time3 = @belapsed @pipe df |>
>
> @by(_, :a, meanb = mean(:b)) #158
>
> time4 = @belapsed fastby(mean, df, :a, :b)
>
> time1 = @belapsed df |>
>
> @groupby(_.a) |>
>
> @map({meanb=mean(_.b)}) |>
>
> DataFrame # 266.353
>
> plot(
>
> ["Query.jl", "DataFrames.jl", "DataFramesMeta.jl", "FastGroupBy.jl"],
>
> [time1, time2, time3, time4];
>
> title = title,
>
> seriestype = :bar)
>
> ```
>
> end
>
> df = DataFrame(a=rand(1:100\_000, 10\_000\_000), b=rand(10\_000\_000))
>
> plot\_bench(df; title = “Group By a (100\_000 groups) mean(b)”)
>
> savefig(“grp100000.png”)
>
> df10 = DataFrame(a=rand(1:8, 10\_000\_000), b=rand(10\_000\_000))
>
> plot\_bench(df10; title = “Group By a (10 groups) mean(b)”)
>
> savefig(“grp10.png”)
---
### Author:  [@derekmahar](https://discourse.julialang.org/u/derekmahar)
#### Post date: [August 27, 2020, 2:04pm UTC](https://discourse.julialang.org/t/a-minor-group-by-benchmark-dataframes-jl-plenty-fast/45622/2 "2020-08-27T14:04:06Z")
What is the difference between group-by in [DataFrames.jl](https://github.com/JuliaData/DataFrames.jl) and [DataFramesMeta.jl](https://github.com/JuliaData/DataFramesMeta.jl)? What table data structures do [Query.jl](https://github.com/queryverse/Query.jl) and [FastGroupBy.jl](https://github.com/xiaodaigh/FastGroupBy.jl) use?
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [August 27, 2020, 2:12pm UTC](https://discourse.julialang.org/t/a-minor-group-by-benchmark-dataframes-jl-plenty-fast/45622/3 "2020-08-27T14:12:30Z")
You can expand the code section to see some details.
All use the same input which is a `DataFrames.DataFrame`.
I DataFrames.jl uses `groupby` and `combine` while DataFramesMeta.jl uses the `@by` which could be using an older algorithm.
I believe Query.jl uses a row-based algorithm which isn’t optimized for the fact that we are dealing with column vectors.
---
### Author:  [@derekmahar](https://discourse.julialang.org/u/derekmahar)
#### Post date: [August 27, 2020, 2:18pm UTC](https://discourse.julialang.org/t/a-minor-group-by-benchmark-dataframes-jl-plenty-fast/45622/4 "2020-08-27T14:18:03Z")
> [@xiaodai](#):
>
> You can expand the code section to see some details.
Which code section do you mean? Do you mean the _Details_ section at the bottom of [Database-like ops benchmark](https://h2oai.github.io/db-benchmark/)?
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [August 27, 2020, 2:24pm UTC](https://discourse.julialang.org/t/a-minor-group-by-benchmark-dataframes-jl-plenty-fast/45622/5 "2020-08-27T14:24:37Z")
I included the source code in this post
---
### Author:  [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [August 27, 2020, 3:00pm UTC](https://discourse.julialang.org/t/a-minor-group-by-benchmark-dataframes-jl-plenty-fast/45622/6 "2020-08-27T15:00:39Z")
Yes the new manipulation functions `combine`, `select`, etc are very speedy compared to previous versions!