# Using the groupby function

**URL:** <https://discourse.julialang.org/t/using-the-groupby-function/40839>\
**Category:** Data\
**Created:** [June 5, 2020, 11:38pm UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839 "2020-06-05T23:38:25Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chris\_Anderson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chris_anderson/32/9465_2.png) [@Chris\_Anderson](https://discourse.julialang.org/u/Chris_Anderson)\
**Post date:** [June 5, 2020, 11:38pm UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/1 "2020-06-05T23:38:25Z")

</div>

I have a dataframe that I would like to group by a categorical variable (4 values, let’s say a, b, c and d) and then by a continuous variable (values between 0 and 40). I’d like the continuous variable to be in groups too. Zero to five, five to ten, ten to fifteen, etc. Ideally I’d end up with 32 different dataframes by the end of this process. I know how to us groupby for categorical variables, but not grouping by continuous variables.  
Thank you.

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [June 5, 2020, 11:54pm UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/2 "2020-06-05T23:54:52Z")

</div>

I believe that you’re conflating 2 steps. You need to first bin your continuous values into your desired bins and then use groupby, which simply groups by distinct values, regardless of whether the variables are discrete or continuous.

You could write your own binning function or perhaps use the fit function from here:  
[https://juliastats.org/StatsBase.jl/v0.33/empirical/#Histograms-1](https://juliastats.org/StatsBase.jl/v0.33/empirical/#Histograms-1)

---

<div class="post-metadata">

**Author:** ![jack\_rabbit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jack_rabbit/32/10914_2.png) [@jack\_rabbit](https://discourse.julialang.org/u/jack_rabbit)\
**Post date:** [June 6, 2020, 12:12am UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/3 "2020-06-06T00:12:07Z")

</div>

Somebody else may give a better answer. For this specific problem, you may try something similar

```nohighlight
# load libraries
using DataFrames, Statistics, Distributions

# create a demo DataFrame
df = DataFrame(a = repeat('a':'d', outer = 100), b = rand(Uniform(0,40), 400), c = randn(400))
# generate a group variable based on the continuous variable, subject to specific problem
df[:, :group] = map(x -> div(x, 5.0), df.b)
# create a groupby DataFrame based on variables a and group
gdf = groupby(df, [:a, :group])
# get some summary Statistics by group
combine(gdf, :c=>mean)

```

---

<div class="post-metadata">

**Author:** ![Mattriks](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mattriks/32/351_2.png) [@Mattriks](https://discourse.julialang.org/u/Mattriks)\
**Post date:** [June 6, 2020, 12:36am UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/4 "2020-06-06T00:36:18Z")

</div>

Also for binning:

```julia
using CategoricalArrays
?cut

```

`using DataFrames` also works (as cut is reexported from CategoricalArrays)

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [June 6, 2020, 7:07am UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/5 "2020-06-06T07:07:37Z")

</div>

Yes, cut is nice:

```julia
julia> by(transform(DataFrame(x=rand([:a,:b], 10^6), y=rand(10^6)), :y => (y->cut(y,3)) => :y2), [:x,:y2], :y => mean)
6×3 DataFrame
│ Row │ x │ y2 │ y_mean │
│ │ Symbol │ CategoricalValue… │ Float64 │
├────┼───────┼────────────────────────┼─────────┤
│ 1 │ b │ Q3: [0.666464, 0.999998] │ 0.833538 │
│ 2 │ b │ Q2: [0.333284, 0.666464) │ 0.499986 │
│ 3 │ a │ Q2: [0.333284, 0.666464) │ 0.499864 │
│ 4 │ a │ Q3: [0.666464, 0.999998] │ 0.833266 │
│ 5 │ a │ Q1: [1.59801e-6, 0.333284) │ 0.166748 │
│ 6 │ b │ Q1: [1.59801e-6, 0.333284) │ 0.167013 │

```

---

<div class="post-metadata">

**Author:** ![jack\_rabbit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jack_rabbit/32/10914_2.png) [@jack\_rabbit](https://discourse.julialang.org/u/jack_rabbit)\
**Post date:** [June 6, 2020, 1:01pm UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/6 "2020-06-06T13:01:28Z")

</div>

Using `cut` is a good idea. But your method will not guarantee that it cuts at 5, 10, 15, etc.  
A slightly modified version of creating a group variable using `cut` is

```nohighlight
df[:,:g] = cut(df.b, 0:5:40)

```

if all values in column `b` falls into [0,40].

---

<div class="post-metadata">

**Author:** ![Chris\_Anderson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chris_anderson/32/9465_2.png) [@Chris\_Anderson](https://discourse.julialang.org/u/Chris_Anderson)\
**Post date:** [June 6, 2020, 2:51pm UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/7 "2020-06-06T14:51:13Z")

</div>

Thank you for all of the replies, this is very useful information.

Using cut seems the most straightforward, so this is what I’ve gone with. However, after using cut I am not able to then groupby on the new column. The new column has arrays, like [5,10) and [10,15). So it throws an error and won’t group. Any tips on how to deal with this part?

Thanks

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [June 6, 2020, 3:02pm UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/8 "2020-06-06T15:02:55Z")

</div>

Can you show the full error you get?

---

<div class="post-metadata">

**Author:** ![Chris\_Anderson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chris_anderson/32/9465_2.png) [@Chris\_Anderson](https://discourse.julialang.org/u/Chris_Anderson)\
**Post date:** [June 6, 2020, 3:15pm UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/9 "2020-06-06T15:15:25Z")

</div>

My code:  
df[:,:g] = cut(df.Years, 0:5:40)  
gdf = groupby(df, :g)

Error:  
ERROR: LoadError: BoundsError: attempt to access 9-element Array{Int64,1} at index [2301742737]

An aside, I don’t know how to put code in the grey box thing, sorry.

---

<div class="post-metadata">

**Author:** ![Chris\_Anderson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chris_anderson/32/9465_2.png) [@Chris\_Anderson](https://discourse.julialang.org/u/Chris_Anderson)\
**Post date:** [June 6, 2020, 3:28pm UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/10 "2020-06-06T15:28:23Z")

</div>

There were missing values, I’ve figured it out now. Thank you for the help.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [June 6, 2020, 3:40pm UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/11 "2020-06-06T15:40:18Z")

</div>

Hmm, that shouldn’t happen anyway. Would you be able to post code to reproduce the problem on my machine? What version of DataFrames are you using? Does the following code give an error on your machine (it doesn’t on mine)?

```julia
df = DataFrame(Years=replace(rand(0:40, 10_000), 40=>missing))
df[:,:g] = cut(df.Years, 0:5:40)
gdf = groupby(df, :g)

```

---

<div class="post-metadata">

**Author:** ![Chris\_Anderson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chris_anderson/32/9465_2.png) [@Chris\_Anderson](https://discourse.julialang.org/u/Chris_Anderson)\
**Post date:** [June 6, 2020, 3:52pm UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/12 "2020-06-06T15:52:42Z")

</div>

I have DataFrames v0.18.4.  
Yes that code does give an error on my machine.  
I’ve removed/replaced the missing values and everything works now. Why would it work on yours and not mine?

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [June 6, 2020, 4:12pm UTC](https://discourse.julialang.org/t/using-the-groupby-function/40839/13 "2020-06-06T16:12:33Z")

</div>

OK, I think I fixed this in CategoricalArrays 0.8. You should upgrade to DataFrames 0.21 when you can.