# Groupby function?

**URL:** <https://discourse.julialang.org/t/groupby-function/9896>\
**Category:** General Usage\
**Created:** [March 22, 2018, 3:24pm UTC](https://discourse.julialang.org/t/groupby-function/9896 "2018-03-22T15:24:32Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jean\_Michel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jean_michel/32/8282_2.png) [@Jean\_Michel](https://discourse.julialang.org/u/Jean_Michel)\
**Post date:** [March 22, 2018, 3:24pm UTC](https://discourse.julialang.org/t/groupby-function/9896/1 "2018-03-22T15:24:32Z")

</div>

While porting GAP code, I translated the function CollectBy (that I find very useful) as:

```julia
"""
  group items of list l according to the corresponding values in list v

    julia> groupby([31,28,31,30,31,30,31,31,30,31,30,31],
           [:Jan,:Feb,:Mar,:Apr,:May,:Jun,:Jul,:Aug,:Sep,:Oct,:Nov,:Dec])
    Dict{Int64,Array{Symbol,1}} with 3 entries:
      31 => Symbol[:Jan, :Mar, :May, :Jul, :Aug, :Oct, :Dec]
      28 => Symbol[:Feb]
      30 => Symbol[:Apr, :Jun, :Sep, :Nov]

"""  
function groupby(v::AbstractVector,l::AbstractVector)
  res=Dict{eltype(v),Vector{eltype(l)}}()
  for (k,val) in zip(v,l)
    push!(get!(res,k,similar(l,0)),val) 
  end
  res
end

"""
  group items of list l according to the values taken by function f on them

    julia> groupby(iseven,1:10)
    Dict{Bool,Array{Int64,1}} with 2 entries:
      false => [1, 3, 5, 7, 9]
      true => [2, 4, 6, 8, 10]

Note:in this version l is required to be non-empty since I do not know how to
access the return type of a function
"""
function groupby(f,l::AbstractVector)
  res=Dict(f(l[1])=>[l[1]]) # l should be nonempty
  for val in l[2:end] 
    push!(get!(res,f(val),similar(l,0)),val) 
  end
  res
end

```

I choose the name `groupby` since I saw in some messages that there seems to be a function of that name  
doing something similar in some package. I have several questions:

- is there some “standard library” where such a function can be found?
- if not, does my implementation look good? Is it possible to do better/faster?
- In particular, is there a good way to solve the problem of accessing function return type?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [March 22, 2018, 4:58pm UTC](https://discourse.julialang.org/t/groupby-function/9896/2 "2018-03-22T16:58:48Z")

</div>

[Query.jl](https://github.com/davidanthoff/Query.jl) also provides an implementation for [groupby](http://www.david-anthoff.com/Query.jl/stable/experimental.html#The-@groupby-command-1). It should work with pretty much any data source.

---

<div class="post-metadata">

**Author:** ![Jean\_Michel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jean_michel/32/8282_2.png) [@Jean\_Michel](https://discourse.julialang.org/u/Jean_Michel)\
**Post date:** [March 22, 2018, 5:41pm UTC](https://discourse.julialang.org/t/groupby-function/9896/3 "2018-03-22T17:41:54Z")

</div>

In `Query.jl` it seems `groupby` is a macro rather than a function. How does it work in the examples I gave?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [March 22, 2018, 6:23pm UTC](https://discourse.julialang.org/t/groupby-function/9896/4 "2018-03-22T18:23:06Z")

</div>

The standalone query commands version would be:

```julia
1:10 |> @groupby(iseven(_)) |> collect

```

You need the `collect` at the end because `@groupby` returns a lazy iterator.

If you don’t want to use the pipe syntax, you’d do:

```julia
collect(@groupby(1:10, iseven(_), _))

```

The third argument to `@groupby` there is another projection function that is applied to each element before it is placed in a group, in this case just the identity function.

The LINQ style version would be:

```julia
@from i in 1:10 begin
    @group i by iseven(i) into g
    @select g
    @collect
end

```

Both do the same thing.

---

<div class="post-metadata">

**Author:** ![Nosferican](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nosferican/32/9275_2.png) [@Nosferican](https://discourse.julialang.org/u/Nosferican)\
**Post date:** [March 22, 2018, 9:07pm UTC](https://discourse.julialang.org/t/groupby-function/9896/5 "2018-03-22T21:07:42Z")

</div>

In my packages I tend to perform a few transformations on a series of subgroups. Hence, I have those as `Vector{Vector{Vector{T}}} where T <: Integer`. Some transformations are passed the subgroups as `obj[subgroup]`. Would it be possible to use the iterators as a more efficient solution?

---

<div class="post-metadata">

**Author:** ![Jean\_Michel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jean_michel/32/8282_2.png) [@Jean\_Michel](https://discourse.julialang.org/u/Jean_Michel)\
**Post date:** [March 22, 2018, 9:13pm UTC](https://discourse.julialang.org/t/groupby-function/9896/6 "2018-03-22T21:13:40Z")

</div>

Building a `Dict` seems to me the best solution to construct the groups – though perhaps sorting could be faster.  
Certainly `Dict` is a very flexible data structure to use the groups. I am not sure exactly what you mean —  
an example of code would be useful.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [March 23, 2018, 12:47am UTC](https://discourse.julialang.org/t/groupby-function/9896/7 "2018-03-23T00:47:12Z")

</div>

> [@Jean\_Michel](#):
>
> Building a Dict seems to me the best solution to construct the groups

You can also do that with [Query.jl](https://github.com/davidanthoff/Query.jl):

```julia
1:10 |> @groupby(iseven(_)) |> @map(_.key => collect(_)) |> Dict

```

---

<div class="post-metadata">

**Author:** ![Nosferican](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nosferican/32/9275_2.png) [@Nosferican](https://discourse.julialang.org/u/Nosferican)\
**Post date:** [March 23, 2018, 3:24am UTC](https://discourse.julialang.org/t/groupby-function/9896/8 "2018-03-23T03:24:25Z")

</div>

See for example, [JuliaEconometrics/EconUtils.jl, within.jl](https://github.com/JuliaEconometrics/EconUtils.jl/blob/master/src/within.jl). Think of R’s `tapply(X, INDEX, FUN)` or `ddply(.data, .variables, .fun)`. The idea is to have an `object::AbstractVecOrMat`, an index, and a function. The index can either be computed once and stored or be a mapping function. The function takes a subgroup of the data and is passed to the function. I usually generate the index through something like

```julia
Index = map(elem -> find(isequal(elem), obj), unique(obj))

```

where `Index::Vector{Vector{Int64}}`. In some cases for multiple dimensions I use `Index::Vector{Vector{Vector{Int64}}}`. A typical function would have,

```julia
function magic(obj::AbstractMatrix,
               ind::AbstractVector{<:AbstractVector{<:AbstractVector{<:Integer}}})
    for dimension ∈ groups
        for lvl ∈ dimension
            obj[lvl,:] = foo(obj[lvl,:])
        end
    end
end

```