# Nested groupby

**URL:** <https://discourse.julialang.org/t/nested-groupby/78958>\
**Category:** Data\
**Tags:** question, inmemorydatasets, combine\
**Created:** [April 3, 2022, 5:26pm UTC](https://discourse.julialang.org/t/nested-groupby/78958 "2022-04-03T17:26:24Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [April 3, 2022, 5:26pm UTC](https://discourse.julialang.org/t/nested-groupby/78958/1 "2022-04-03T17:26:25Z")

</div>

trying to understand the role of kwarg isgathered, I practiced this exercise.

```julia
julia> ds = Dataset(id = [1,1,1,1,1,2,2,2,3,3,3],
                    date = Date.(["2019-03-05", "2019-03-12", "2019-04-10",
                            "2019-04-29", "2019-05-10", "2019-03-20",
                            "2019-04-22", "2019-05-04", "2019-11-01",
                            "2019-11-10", "2019-12-12"]),
                    outcome = [false, false, false, true, false, false,
                               true, false, true, true, true])
11×3 Dataset
...

julia> gb=gatherby(ds, [1, 3], isgathered = true)  
11×3 View of GatherBy Dataset, Gathered by: id ,outcome        
 id date outcome  
 identity identity identity
 Int64? Date? Bool?
────────────────────────────────
        1 2019-03-05 false
        1 2019-03-12 false
        1 2019-04-10 false
        1 2019-04-29 true
        1 2019-05-10 false
        2 2019-03-20 false
        2 2019-04-22 true
        2 2019-05-04 false
        3 2019-11-01 true
        3 2019-11-10 true
        3 2019-12-12 true

julia> combine(gb, (:) => last, dropgroupcols = true)
7×3 Dataset
...

```

I was wondering how many groups are generated by

`julia> gb = gatherby (ds, [1, 3], isgathered = true)`

because the output header does not report the information:

`11 × 3 View of GatherBy Dataset, Gathered by: id, outcome`

I saw that the info can be obtained in the following way  
`last (gb.groups),` but I don’t think it’s a recommended procedure.  
Later, having read that groupby and gatherby accept the output of other grouping functions as input, I tried (in all the variations that came to my mind ), unsuccessfully, to do something like this:

```julia
combine (gb, (:) => x-> groupby (x, 1)).

```

so I tried to build the subgroups by hand and, after a few attempts, I found this “solution”

```julia

cgb1 = combine (gb, (:) => x -> [x], (:) => byrow (x-> Dataset (; zip ([: a,: b,: c], x) ...) ), dropgroupcols = true)

```

At this point the question arises why the following “reduced” expression does not work

```julia

cgb1 = combine (gb, (:) => byrow (x-> Dataset (; zip ([: a,: b,: c], [x]) ...)), dropgroupcols = true)

```

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [April 3, 2022, 7:55pm UTC](https://discourse.julialang.org/t/nested-groupby/78958/2 "2022-04-03T19:55:42Z")

</div>

I found this way to get more directly to the form I was looking for.

`cgb1 = combine(gb, (1,2,3) => (x...)-> Dataset(; zip([:a,:b,:c], x) ...))`

I don’t know if there are others even more direct.

So at first thought, a method for combine which allows to pass the entire subgroup to a function for the subsequent transformations, would seem desirable to me.  
eg:  
`combine (gbds, x-> nrow (x))`  
in general  
`combine (gbds, sg-> fun (sg))`  
But I don’t know if it has any unwanted side effects.

---

<div class="post-metadata">

**Author:** ![monopolynomial](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/monopolynomial/32/34782_2.png) [@monopolynomial](https://discourse.julialang.org/u/monopolynomial)\
**Post date:** [April 4, 2022, 12:34am UTC](https://discourse.julialang.org/t/nested-groupby/78958/3 "2022-04-04T00:34:42Z")

</div>

> [@rocco\_sprmnt21](#):
>
> Later, having read that groupby and gatherby accept the output of other grouping functions as input

my take was like this `groupby` can group the output of another `groupby`, like `groupby(groupby(ds,:x1),:x2)`.

---

<div class="post-metadata">

**Author:** ![monopolynomial](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/monopolynomial/32/34782_2.png) [@monopolynomial](https://discourse.julialang.org/u/monopolynomial)\
**Post date:** [April 4, 2022, 12:47am UTC](https://discourse.julialang.org/t/nested-groupby/78958/4 "2022-04-04T00:47:10Z")

</div>

> [@rocco\_sprmnt21](#):
>
> I was wondering how many groups are generated by

my understanding is that 1:3, 4:4, 5:5, 6:6, 7:7, 8:8, 9:11 are starts and ends of groups this means total of 7 groups.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [April 4, 2022, 6:45am UTC](https://discourse.julialang.org/t/nested-groupby/78958/5 "2022-04-04T06:45:55Z")

</div>

From the package documentation I have seen that in some cases the information is exposed.  
I was wondering why in other cases it is not and if it is not the case to standardize the outputs in this respect.  
I believe that info is more important in the case of gatherby than in the case of groupby

```julia
julia> groupby!(ds, 1)
6×2 Grouped Dataset with 2 groups
Grouped by: g
 Row │ g x        
     │ identity identity
     │ Int64? Float64?
─────┼────────────────────
   1 │ 1 12.0
   2 │ 1 11.0
   3 │ 1 15.0
   4 │ 2 12.3
   5 │ 2 13.0
   6 │ 2 13.2

```

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [April 4, 2022, 6:48am UTC](https://discourse.julialang.org/t/nested-groupby/78958/6 "2022-04-04T06:48:59Z")

</div>

Yes. I too understand this. I just wanted to understand the possibilities of the package and how it works.  
A little by reading the documentation (not much though 😃) a little by doing tests.  
I still don’t quite understand what is passed to functions in the part  
cols =\> fun =\> newcols

in different situations.

---

<div class="post-metadata">

**Author:** ![monopolynomial](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/monopolynomial/32/34782_2.png) [@monopolynomial](https://discourse.julialang.org/u/monopolynomial)\
**Post date:** [April 4, 2022, 11:46pm UTC](https://discourse.julialang.org/t/nested-groupby/78958/7 "2022-04-04T23:46:04Z")

</div>

i just look at `col=>fun` as a pipe in linux, “pipe col to fun”.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [April 6, 2022, 10:54am UTC](https://discourse.julialang.org/t/nested-groupby/78958/8 "2022-04-06T10:54:33Z")

</div>

in many cases, I have been able to figure out how the parameters are passed

> **cases working**
>
> ```julia
> ds=Dataset(a=rand(6),a_lim=0.75,b=rand(6),b_lim=0.33,c=rand(6),c_lim=0.25)
> 
> modify(compare(ds[!, r"lim"], ds[!, Not(r"lim")], on = 1:3 .=> 1:3, eq = !isless), 1:3=>byrow(count))
> modify(compare(ds[!, r"lim"], ds[!, Not(r"lim")], on = 1:3 .=> 1:3, eq = !isless), 1:3=>byrow(sum))
> 
> modify(modify(compare(ds[!, r"lim"], ds[!, Not(r"lim")], on = 1:3 .=> 1:3, eq = !isless), 
> 1:3=>x->Int.(x)), 1:3=>byrow(x->sum(x)))
> 
> cds=compare(ds[!, r"lim"], ds[!, Not(r"lim")], on = 1:3 .=> 1:3, eq = !isless)
> mcds=modify(cds,1:3=>x->Int.(x))
> modify(mcds, 1:3=>byrow(sum))
> modify(mcds, 1:3=>byrow(x->sum(x)))
> 
> mmcds=modify(mcds, 1:3=>byrow(x->x .*2) =>:v1,1:3=>byrow(x->x .* 3)=>:v2)[:,[1,2,4,5]]
> names(mmcds)
> rename!(mmcds,[:a,:b,:v1,:v2])
> modify(mmcds,[:v1,:v2]=>byrow(x->sum(x[1]))=>:sum_v1)
> modify(mmcds,:v1=>byrow(x->sum(x))=>:sum_v1)
> modify(mmcds,:v1=>byrow(x->mean(x))=>:mean_v1)
> 
> modify(mmcds,[:v1,:v2]=>byrow(x->mean(x))=>:mean_v1_v2)
> modify(mmcds,[:a,:b]=>byrow(x->mean(x))=>:mean_a_b)
> 
> modify(mmcds,[:v1,:v2]=>byrow(x->sum(x)))
> modify(mmcds,[:v1,:v2]=>byrow(sum))
> modify(mmcds,[:v2]=>byrow(sum))
> modify(mmcds,:v2=>byrow(sum))
> modify(mmcds,:v2=>byrow(x->sum(x)))
> 
> modify(mmcds,[:a,:b]=>byrow(x->sum(x)))
> modify(mmcds,[:a,:b]=>byrow(sum))
> modify(mmcds,[:b]=>byrow(sum))
> 
> modify(mmcds,[:v1,:v2]=>byrow((x,y)->y .+ x[2]))
> 
> modify(mmcds,[:v1,:v2]=>byrow((x,y)->x .+y))
> modify(mmcds,[:a,:b]=>byrow((x,y)->x .+y))
> 
> modify(mmcds,(:v1,:v2)=>(x...)->(x[2]))
> modify(mmcds,(:a,:b)=>(x...)->(x[2]))
> 
> modify(mmcds,(:v1,:v2)=>(x,y)-> string(x...,y...)[end-30:end])
> modify(mmcds,(:a,:b)=>(x,y)-> string(x...," | ",y...))
> 
> modify(mmcds,(:v1,:v2)=>(x,y)-> (x[3],y[3]))
> modify(mmcds,(:a,:b)=>(x,y)-> (x[3],y[3]))
> 
> modify(mmcds,(:v1,:v2)=>(x,y)-> x[1:2])
> modify(mmcds,(:a,:b)=>(x,y)-> x[1:2])
> 
> modify(mmcds,(:v1,:v2)=>(x,y)->sum.(zip(x,y)))
> 
> modify(mmcds,(:a,:b)=>(x,y)->sum.(zip(x,y)))
> 
> byrow(mmcds,sum, [:a,:b])
> 
> ```

but in this I don’t understand why the form with byrow () doesn’t work

```julia
julia> mmcds=mmcds[:,[3,4]]
6×2 Dataset
 Row │ v1 v2        
     │ identity identity
     │ Array…? Array…?
─────┼──────────────────────
   1 │ [0, 2, 0] [0, 3, 0]
   2 │ [2, 0, 0] [3, 0, 0]
   3 │ [2, 0, 2] [3, 0, 3]
   4 │ [2, 2, 0] [3, 3, 0]
   5 │ [2, 0, 0] [3, 0, 0]
   6 │ [2, 0, 0] [3, 0, 0]

julia> modify(mmcds,(:v1,:v2)=>(x,y)->cor.(x,y))
6×3 Dataset
 Row │ v1 v2 function_v1_v2 
     │ identity identity identity
     │ Array…? Array…? Float64?
─────┼──────────────────────────────────────
   1 │ [0, 2, 0] [0, 3, 0] 1.0
   2 │ [2, 0, 0] [3, 0, 0] 1.0
   3 │ [2, 0, 2] [3, 0, 3] 1.0
   4 │ [2, 2, 0] [3, 3, 0] 1.0
   5 │ [2, 0, 0] [3, 0, 0] 1.0
   6 │ [2, 0, 0] [3, 0, 0] 1.0

julia> modify(mmcds,(:v1,:v2)=>byrow(cor))
ERROR: MethodError: no method matching normalize_modify!(::InMemoryDatasets.

```

---

<div class="post-metadata">

**Author:** ![DataFrames](https://avatars.discourse-cdn.com/v4/letter/d/e19b73/32.png) [@DataFrames](https://discourse.julialang.org/u/DataFrames)\
**Post date:** [April 10, 2022, 2:13am UTC](https://discourse.julialang.org/t/nested-groupby/78958/9 "2022-04-10T02:13:41Z")

</div>

use

```julia
modify(mmcds,[:v1,:v2]=>byrow(cor))

```

---

<div class="post-metadata">

**Author:** ![DataFrames](https://avatars.discourse-cdn.com/v4/letter/d/e19b73/32.png) [@DataFrames](https://discourse.julialang.org/u/DataFrames)\
**Post date:** [July 23, 2022, 5:17am UTC](https://discourse.julialang.org/t/nested-groupby/78958/10 "2022-07-23T05:17:36Z")

</div>

> [@rocco\_sprmnt21](#):
>
> a method for combine which allows to pass the entire subgroup to a function for the subsequent transformations, would seem desirable to me.

use `eachgroup` in `InMemoryDatasets`.