# DataFrame \`by\` function error

**URL:** <https://discourse.julialang.org/t/dataframe-by-function-error/11278>\
**Category:** New to Julia\
**Created:** [May 31, 2018, 5:39am UTC](https://discourse.julialang.org/t/dataframe-by-function-error/11278 "2018-05-31T05:39:25Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [May 31, 2018, 5:39am UTC](https://discourse.julialang.org/t/dataframe-by-function-error/11278/1 "2018-05-31T05:39:25Z")

</div>

My understanding is that function `f` will get an `AbstractDataFrame` and I can do whatever with it. However, it fails when I just return the input. Why? What am I missing?

```julia
df = DataFrame(a = repeat([1, 2, 3, 4], outer=[2]),
               b = repeat([2, 1], outer=[4]),
               c = randn(8));

by(df, :a, d -> d)

WARNING: Duplicate variable names are deprecated: pass makeunique=true to add a suffix automatically.
Stacktrace:
 [1] depwarn(::String, ::Symbol) at ./deprecated.jl:70
 [2] #add_names#18(::Bool, ::Function, ::DataFrames.Index, ::DataFrames.Index) at /Users/tomkwong/.julia/v0.6/DataFrames/src/other/index.jl:190
 [3] (::DataFrames.#kw##add_names)(::Array{Any,1}, ::DataFrames.#add_names, ::DataFrames.Index, ::DataFrames.Index) at ./<missing>:0
 [4] #hcat!#71(::Bool, ::Function, ::DataFrames.DataFrame, ::DataFrames.DataFrame) at /Users/tomkwong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:840
 [5] (::DataFrames.#kw##hcat!)(::Array{Any,1}, ::DataFrames.#hcat!, ::DataFrames.DataFrame, ::DataFrames.DataFrame) at ./<missing>:0
 [6] combine(::DataFrames.GroupApplied{DataFrames.SubDataFrame{Array{Int64,1}}}) at /Users/tomkwong/.julia/v0.6/DataFrames/src/groupeddataframe/grouping.jl:202
 [7] by(::DataFrames.DataFrame, ::Symbol, ::Function) at /Users/tomkwong/.julia/v0.6/DataFrames/src/groupeddataframe/grouping.jl:293

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 31, 2018, 8:39am UTC](https://discourse.julialang.org/t/dataframe-by-function-error/11278/2 "2018-05-31T08:39:06Z")

</div>

How does it “fail”? For me, it just give the warning above, but returns

```julia
8×4 DataFrames.DataFrame
│ Row │ a │ a_1 │ b │ c │
├─────┼───┼─────┼───┼───────────┤
│ 1 │ 1 │ 1 │ 2 │ 0.140573 │
│ 2 │ 1 │ 1 │ 2 │ -1.87573 │
│ 3 │ 2 │ 2 │ 1 │ -0.793972 │
│ 4 │ 2 │ 2 │ 1 │ -0.339131 │
│ 5 │ 3 │ 3 │ 2 │ -1.68218 │
│ 6 │ 3 │ 3 │ 2 │ 0.169798 │
│ 7 │ 4 │ 4 │ 1 │ -0.111596 │
│ 8 │ 4 │ 4 │ 1 │ 0.353701 │

```

> [@tk3369](#):
>
> What am I missing?

Probably the fact that `by` separates by the given columns, does the transformation, and then combines. From the docstring,

> For a DataFrame, cols are combined along columns with the resulting DataFrame.

so the no-op would be

```julia
by(df, [:a], d -> d[:, [:b, :c]])

```

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [May 31, 2018, 10:11am UTC](https://discourse.julialang.org/t/dataframe-by-function-error/11278/3 "2018-05-31T10:11:16Z")

</div>

In JuliaDB the default is to only pass the “non grouping” columns to the anonymous function, but I’m not sure which if there are strong reasons in favor of one option or the other. Otherwise, having some syntactic sugar to select everything except the grouping columns may help.

---

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [May 31, 2018, 4:32pm UTC](https://discourse.julialang.org/t/dataframe-by-function-error/11278/4 "2018-05-31T16:32:00Z")

</div>

It was a surprise to me because I thought the semantic is to summarize data that fits in a single cell for the group. In the above example, I was expecting only 4 rows to be returned and the new column will contain elements that are individual data frames.

I like that fact that the new column can contains objects e.g. passing `x -> size(x)` would return a tuple type in that column. The current behavior seems inconsistent when the function returns a DataFrame.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [May 31, 2018, 4:50pm UTC](https://discourse.julialang.org/t/dataframe-by-function-error/11278/5 "2018-05-31T16:50:20Z")

</div>

I think you’re only seeing this awkward thing with the column names. The problem is that `by` always tries to construct a dataframe with the columns that were grouped by, so if you try to return a `DataFrame` for each group, it will wind up trying to duplicate those columns. There really should be a keyword argument that tells it _not_ to try to include the grouped by columns, perhaps it’s worth thinking about more carefully and opening an issue.

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [May 31, 2018, 5:31pm UTC](https://discourse.julialang.org/t/dataframe-by-function-error/11278/6 "2018-05-31T17:31:15Z")

</div>

Just for reference, the JuliaDB implementation uses the `select` keyword for this. `groupby(df, by, select = Not(by))` is the default and `groupby(df, by, select = All())` would select all columns. To use grouping columns in the inner function the recommended approach is (keep in mind that `groupby` in JuliaDB is the same as `by` in DataFrames):

```julia
groupby(df, by, usekey=true) do key, dd
....
end

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [June 1, 2018, 9:17am UTC](https://discourse.julialang.org/t/dataframe-by-function-error/11278/7 "2018-06-01T09:17:31Z")

</div>

> [@ExpandingMan](#):
>
> There really should be a keyword argument that tells it _not_ to try to include the grouped by columns

I think this should be the default.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [June 1, 2018, 10:01am UTC](https://discourse.julialang.org/t/dataframe-by-function-error/11278/8 "2018-06-01T10:01:13Z")

</div>

That makes sense. Feel free to file an issue/PR. It shouldn’t be hard to do, but we’d better have a quick look at what dplyr and Pandas do too.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [June 1, 2018, 2:00pm UTC](https://discourse.julialang.org/t/dataframe-by-function-error/11278/9 "2018-06-01T14:00:44Z")

</div>

I’ve thought about this a bit more. I think that what we should aim for is for

```julia
by(identity, df, cols)

```

to return the original `DataFrame` except possibly for ordering. This would also agree with @Tamas_Papp suggestion that the default behavior is _not_ to try to insert the grouped by columns.

I’ll try to make a PR, but I’m not sure when I’ll get to it.