### Author:  [@vsoler](https://discourse.julialang.org/u/vsoler)
#### Post date: [January 20, 2024, 9:47am UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/1 "2024-01-20T09:47:52Z")
I’m struggling with transformations in DataFrames.
To make it simple, let’s start with a very basic transformation. Suppose that I want to add columns :b and :c in the following DataFrame:
```julia
df = DataFrame(a=repeat([1,2], outer=3), b=repeat([1,2,3], outer=2), c=1:6)
```
6×3 DataFrame
| Row | a | b | c |
| --- | --- | --- | --- |
| | Int64 | Int64 | Int64 |
| 1 | 1 | 1 | 1 |
| 2 | 2 | 2 | 2 |
| 3 | 1 | 3 | 3 |
| 4 | 2 | 1 | 4 |
| 5 | 1 | 2 | 5 |
| 6 | 2 | 3 | 6 |
```julia
gdf=groupby(df, :a)
transform(gdf, [:b, :c] => ((b, c) -> b + c) => :b_plus_c)
```
6×4 DataFrame
| Row | a | b | c | b\_plus\_c |
| --- | --- | --- | --- | --- |
| | Int64 | Int64 | Int64 | Int64 |
| 1 | 1 | 1 | 1 | 2 |
| 2 | 2 | 2 | 2 | 4 |
| 3 | 1 | 3 | 3 | 6 |
| 4 | 2 | 1 | 4 | 5 |
| 5 | 1 | 2 | 5 | 7 |
| 6 | 2 | 3 | 6 | 9 |
Now, suppose that, instead of adding two columns, I want to multiply them:
```julia
transform(gdf, [:b, :c] => ((b, c) -> b * c) => :b_times_c)
```
Now I get:
MethodError: no method matching \*(::SubArray{Int64, 1, Vector{Int64}, Tuple{SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}}, false}, ::SubArray{Int64, 1, Vector{Int64}, Tuple{SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}}, false})
Closest candidates are:
\*(::Any, ::Any, ::Any, ::Any…)
@ Base operators.jl:578
\*(::AbstractVector, ::LinearAlgebra.AbstractRotation)
@ LinearAlgebra C:\Users\USUARIO\AppData\Local\Programs\Julia-1.9.2\share\julia\stdlib\v1.9\LinearAlgebra\src\givens.jl:19
\*(::LinearAlgebra.Diagonal, ::AbstractVector)
@ LinearAlgebra C:\Users\USUARIO\AppData\Local\Programs\Julia-1.9.2\share\julia\stdlib\v1.9\LinearAlgebra\src\diagonal.jl:242
…
If instead of multiplying, I want to calculate the minimum of :b and :c:
```julia
transform(gdf, [:b, :c] => ((b, c) -> minimum(b,c)) => :minimum_b_c)
```
… I get…
MethodError: objects of type SubArray{Int64, 1, Vector{Int64}, Tuple{SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}}, false} are not callable
Use square brackets for indexing an Array.
My conclusion is that `transform` can only be used to add or subtract two columns, but it is unable to multiply or divide them.
Perhaps I am doing something wrong, but I cannot figure out what it is.
---
### Author:  [@bertschi](https://discourse.julialang.org/u/bertschi)
#### Post date: [January 20, 2024, 10:18am UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/2 "2024-01-20T10:18:30Z")
Well, the function passed to transform gets handed the full column vectors of your data frame, i.e., in your example `b` and `c` will be bound to vectors when executing `((b, c) -> )`. Thus, your function should be designed to operate on vectors and return a vector containing the result of your transformation.
1. `b + c` works, because `+` happens to be defined for vector (as these can be considered as vector spaces where addition makes sense).
2. `b * c` is not defined on vectors as there is no unambiguous definition on (finite) vector spaces. To apply `*` element-wise use explicit [broadcasting](https://docs.julialang.org/en/v1/manual/arrays/#Broadcasting), i.e., write `b .* c`
3. `minimum` usually takes a single argument and reduces its argument to a single minimal value. To compute the elementwise min, you need to broadcast the `min` function, i.e., write `min.(b, c)`.
---
### Author:  [@vsoler](https://discourse.julialang.org/u/vsoler)
#### Post date: [January 20, 2024, 10:29am UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/3 "2024-01-20T10:29:06Z")
Thanks, @bertschi
I have checked your solution on my PC.
> `b .* c` works for me.
> but `min.(b, c)` does not
I get the following error:
```julia
transform(gdf, [:b, :c] => ((b, c) -> min.(b, c)) => :minimum_b_c)
```
> MethodError: objects of type Tuple{Int64, Int64, Int64} are not callable
---
### Author:  [@bertschi](https://discourse.julialang.org/u/bertschi)
#### Post date: [January 20, 2024, 10:34am UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/4 "2024-01-20T10:34:01Z")
Hmm, it does work for me (and I cannot reproduce the exact error when trying some variants).
---
### Author:  [@vsoler](https://discourse.julialang.org/u/vsoler)
#### Post date: [January 20, 2024, 10:54am UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/5 "2024-01-20T10:54:53Z")
I’m running Julia Ver 1.9.2 and DataFrames Ver 1.6.1
Could this explain why it is not working for me?
---
### Author:  [@bertschi](https://discourse.julialang.org/u/bertschi)
#### Post date: [January 20, 2024, 12:01pm UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/6 "2024-01-20T12:01:34Z")
Maybe, but broadcasting is very deeply engrained into Julia and should work in any version.
Can you post the precise interaction, you typed into the REPL, including the code line and the full stacktrace?
---
### Author:  [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [January 20, 2024, 1:51pm UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/7 "2024-01-20T13:51:21Z")
You need to look at the `ByRow` function.
```julia
using DataFrames
df = DataFrame(a=repeat([1,2], outer=3), b=repeat([1,2,3], outer=2), c=1:6)
transform(df,["b","c"]=>ByRow(+))
transform(df,["b","c"]=>ByRow(*))
transform(df,["b","c"]=>ByRow(min))
```
---
### Author:  [@hendri54](https://discourse.julialang.org/u/hendri54)
#### Post date: [January 20, 2024, 2:24pm UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/8 "2024-01-20T14:24:52Z")
Probably easiest is `DataFramesMeta`:
```julia
using DataFrames, DataFramesMeta;
df = DataFrame(a=repeat([1,2], outer=3), b=repeat([1,2,3], outer=2), c=1:6);
gdf = groupby(df, :a);
@rtransform(gdf, :bcmin = min(:b, :c))
```
---
### Author:  [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)
#### Post date: [January 20, 2024, 4:14pm UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/9 "2024-01-20T16:14:14Z")
Have you accidentally done something like this?
```julia
# julia> transform(gdf, [:b, :c] => ((b, c) -> (c...,)(b)) => :minimum_b_c)
# ERROR: MethodError: objects of type Tuple{Int64, Int64, Int64} are not callable
```
In case you need it, know that a function that transforms your input vectors into a scalar is fine too.
```julia
transform(gdf, [:b, :c] => ((b, c) -> b'*c) => :b_dot_c)
```
---
### Author:  [@alfaromartino](https://discourse.julialang.org/u/alfaromartino)
#### Post date: [January 21, 2024, 12:43am UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/10 "2024-01-21T00:43:34Z")
In case you’re just starting with Julia DataFrames, I think it’s important to distinguish between the operations you’re trying to perform.
When you’re operating with pairs of columns, like with +, you don’t need to group the data or apply `transform`. You should simply do:
```julia
dff = DataFrame(a=repeat([1,2], outer=3), b=repeat([1,2,3], outer=2), c=1:6)
dff.b_plus_c = dff.b + dff.c
```
The same holds for `min`, if what you aimed for was to keep the minimum between pairs of two columns
```julia
dff.b_plus_c = min.(dff.b, dff.c)
```
Instead, use `transform` when you want to operate with **whole** columns rather than pairs of columns.
In that case, you could probably want to define what a whole vector should be considered, in the sense of only taking the whole vector by groups.
For example, if you want to take the minimum of `c` for each specific group of `a` and return the result, then you need:
```julia
gdf = groupby(dff, :a)
transform(gdf, :c => minimum => :min_c)
transform(gdf, :c => (a -> minimum(a)) => :min_c) #equivalent, what you're doing with the previous operation
```
Also, use `transform!` if you want to update the original `dff`, rather than creating a new dataframe. Otherwise, all the results performed with `transform` will be lost.
---
### Author:  [@rdavis120](https://discourse.julialang.org/u/rdavis120)
#### Post date: [January 21, 2024, 2:32am UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/11 "2024-01-21T02:32:17Z")
If this syntax is more familiar, the [TidierData.jl](https://tidierorg.github.io/TidierData.jl/latest/) package makes the broadcasting invisible:
```julia
using TidierData
@chain df begin
@mutate(b_plus_c = b + c, b_times_c = b * c, bc_min = min(b,c))
end
```
---
### Author:  [@vsoler](https://discourse.julialang.org/u/vsoler)
#### Post date: [January 21, 2024, 4:21pm UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/12 "2024-01-21T16:21:41Z")
> [@vsoler](#):
>
> `transform(gdf, [:b, :c] => ((b, c) -> minimum(b,c)) => :minimum_b_c)`
Here is what I tried, as per your suggestion:
```julia
df = DataFrame(a=repeat([1,2], outer=3), b=repeat([1,2,3], outer=2), c=1:6)
gdf=groupby(df, :a)
transform(gdf, [:b, :c] => ((b, c) -> min.(b, c)) => :min_b_c)
```
And this this what I’m getting:
MethodError: objects of type Tuple{Int64, Int64, Int64} are not callable
Stacktrace:
[1] \_combine(gd::GroupedDataFrame{DataFrame}, cs\_norm::Vector{Any}, optional\_transform::Vector{Bool}, copycols::Bool, keeprows::Bool, renamecols::Bool, threads::Bool)
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:755
[2] \_combine\_prepare\_norm(gd::GroupedDataFrame{DataFrame}, cs\_vec::Vector{Any}, keepkeys::Bool, ungroup::Bool, copycols::Bool, keeprows::Bool, renamecols::Bool, threads::Bool)
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:87
[3] \_combine\_prepare(gd::GroupedDataFrame{DataFrame}, ::Base.RefValue{Any}; keepkeys::Bool, ungroup::Bool, copycols::Bool, keeprows::Bool, renamecols::Bool, threads::Bool)
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:52
[4] \_combine\_prepare
@ C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:26 [inlined]
[5] select(::GroupedDataFrame{DataFrame}, ::Union{Regex, AbstractString, Function, Signed, Symbol, Unsigned, Pair, Type, All, Between, Cols, InvertedIndex, AbstractVecOrMat}, ::Vararg{Union{Regex, AbstractString, Function, Signed, Symbol, Unsigned, Pair, Type, All, Between, Cols, InvertedIndex, AbstractVecOrMat}}; copycols::Bool, keepkeys::Bool, ungroup::Bool, renamecols::Bool, threads::Bool)
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:892
[6] transform(gd::GroupedDataFrame{DataFrame}, args::Union{Regex, AbstractString, Function, Signed, Symbol, Unsigned, Pair, Type, All, Between, Cols, InvertedIndex, AbstractVecOrMat}; copycols::Bool, keepkeys::Bool, ungroup::Bool, renamecols::Bool, threads::Bool)
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:917
[7] transform(gd::GroupedDataFrame{DataFrame}, args::Union{Regex, AbstractString, Function, Signed, Symbol, Unsigned, Pair, Type, All, Between, Cols, InvertedIndex, AbstractVecOrMat})
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:912
[8] top-level scope
@ In[202]:3
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [January 21, 2024, 6:36pm UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/13 "2024-01-21T18:36:03Z")
That code works fine for me. There must be some other code you are running thats broken.
Bumping the DataFramesMeta.jl solution though (as the mantainer of the package). It’s pretty easy to use imo.
---
### Author:  [@bertschi](https://discourse.julialang.org/u/bertschi)
#### Post date: [January 21, 2024, 9:15pm UTC](https://discourse.julialang.org/t/transform-in-dataframes/109041/14 "2024-01-21T21:15:04Z")
Thanks for posting the details, unfortunately the example runs fine for me as well.
The only way, I can reproduce this error is by shadowing `min`, i.e.,
```julia-repl
julia> let min = (1,2,3)
transform(gdf, [:b, :c] => ((b, c) -> min.(b, c)) => :min_b_c)
end
ERROR: MethodError: objects of type Tuple{Int64, Int64, Int64} are not callable
```
Are you running this line inside a function with an argument `min` or otherwise managed to redefine it?