Transform in DataFrames

General Usage
1

I’m struggling with transformations in DataFrames.

To make it simple, let’s start with a very basic transformation. Suppose that I want to add columns :b and :c in the following DataFrame:

df = DataFrame(a=repeat([1,2], outer=3), b=repeat([1,2,3], outer=2), c=1:6)

6×3 DataFrame

Row a b c
Int64 Int64 Int64
1 1 1 1
2 2 2 2
3 1 3 3
4 2 1 4
5 1 2 5
6 2 3 6 
gdf=groupby(df, :a)
transform(gdf, [:b, :c] => ((b, c) -> b + c) => :b_plus_c)

6×4 DataFrame

Row a b c b_plus_c
Int64 Int64 Int64 Int64
1 1 1 1 2
2 2 2 2 4
3 1 3 3 6
4 2 1 4 5
5 1 2 5 7
6 2 3 6 9

Now, suppose that, instead of adding two columns, I want to multiply them:

transform(gdf, [:b, :c] => ((b, c) -> b * c) => :b_times_c)

Now I get:

MethodError: no method matching *(::SubArray{Int64, 1, Vector{Int64}, Tuple{SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}}, false}, ::SubArray{Int64, 1, Vector{Int64}, Tuple{SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}}, false})

Closest candidates are:
*(::Any, ::Any, ::Any, ::Any…)
@ Base operators.jl:578
*(::AbstractVector, ::LinearAlgebra.AbstractRotation)
@ LinearAlgebra C:\Users\USUARIO\AppData\Local\Programs\Julia-1.9.2\share\julia\stdlib\v1.9\LinearAlgebra\src\givens.jl:19
*(::LinearAlgebra.Diagonal, ::AbstractVector)
@ LinearAlgebra C:\Users\USUARIO\AppData\Local\Programs\Julia-1.9.2\share\julia\stdlib\v1.9\LinearAlgebra\src\diagonal.jl:242
…

If instead of multiplying, I want to calculate the minimum of :b and :c:

transform(gdf, [:b, :c] => ((b, c) -> minimum(b,c)) => :minimum_b_c)

… I get…

MethodError: objects of type SubArray{Int64, 1, Vector{Int64}, Tuple{SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}}, false} are not callable
Use square brackets for indexing an Array.

My conclusion is that transform can only be used to add or subtract two columns, but it is unable to multiply or divide them.
Perhaps I am doing something wrong, but I cannot figure out what it is.

2

Well, the function passed to transform gets handed the full column vectors of your data frame, i.e., in your example b and c will be bound to vectors when executing ((b, c) -> <do something on b and c>). Thus, your function should be designed to operate on vectors and return a vector containing the result of your transformation.

  1. b + c works, because + happens to be defined for vector (as these can be considered as vector spaces where addition makes sense).

  2. b * c is not defined on vectors as there is no unambiguous definition on (finite) vector spaces. To apply * element-wise use explicit broadcasting, i.e., write b .* c

  3. minimum usually takes a single argument and reduces its argument to a single minimal value. To compute the elementwise min, you need to broadcast the min function, i.e., write min.(b, c).

3

Thanks, @bertschi
I have checked your solution on my PC.

b .* c works for me.
but min.(b, c) does not

I get the following error:

transform(gdf, [:b, :c] => ((b, c) -> min.(b, c)) => :minimum_b_c)

MethodError: objects of type Tuple{Int64, Int64, Int64} are not callable

4

Hmm, it does work for me (and I cannot reproduce the exact error when trying some variants).

5

I’m running Julia Ver 1.9.2 and DataFrames Ver 1.6.1

Could this explain why it is not working for me?

6

Maybe, but broadcasting is very deeply engrained into Julia and should work in any version.
Can you post the precise interaction, you typed into the REPL, including the code line and the full stacktrace?

7

You need to look at the ByRow function.

using DataFrames

df = DataFrame(a=repeat([1,2], outer=3), b=repeat([1,2,3], outer=2), c=1:6)

transform(df,["b","c"]=>ByRow(+))

transform(df,["b","c"]=>ByRow(*))

transform(df,["b","c"]=>ByRow(min))
2 Likes
8

Probably easiest is DataFramesMeta:

using DataFrames, DataFramesMeta;

df = DataFrame(a=repeat([1,2], outer=3), b=repeat([1,2,3], outer=2), c=1:6);

gdf = groupby(df, :a);

@rtransform(gdf, :bcmin = min(:b, :c))
3 Likes
9

Have you accidentally done something like this?

# julia> transform(gdf, [:b, :c] => ((b, c) -> (c...,)(b)) => :minimum_b_c)
# ERROR: MethodError: objects of type Tuple{Int64, Int64, Int64} are not callable

In case you need it, know that a function that transforms your input vectors into a scalar is fine too.

transform(gdf, [:b, :c] => ((b, c) -> b'*c) => :b_dot_c)
1 Like
10

In case you’re just starting with Julia DataFrames, I think it’s important to distinguish between the operations you’re trying to perform.

When you’re operating with pairs of columns, like with +, you don’t need to group the data or apply transform. You should simply do:


dff = DataFrame(a=repeat([1,2], outer=3), b=repeat([1,2,3], outer=2), c=1:6)

dff.b_plus_c = dff.b + dff.c

The same holds for min, if what you aimed for was to keep the minimum between pairs of two columns

dff.b_plus_c = min.(dff.b, dff.c)

Instead, use transform when you want to operate with whole columns rather than pairs of columns.

In that case, you could probably want to define what a whole vector should be considered, in the sense of only taking the whole vector by groups.

For example, if you want to take the minimum of c for each specific group of a and return the result, then you need:

gdf = groupby(dff, :a)

transform(gdf, :c => minimum => :min_c)

transform(gdf, :c => (a -> minimum(a)) => :min_c) #equivalent, what you're doing with the previous operation

Also, use transform! if you want to update the original dff, rather than creating a new dataframe. Otherwise, all the results performed with transform will be lost.

1 Like
11

If this syntax is more familiar, the TidierData.jl package makes the broadcasting invisible:

using TidierData
@chain df begin
  @mutate(b_plus_c = b + c, b_times_c = b * c, bc_min = min(b,c))
end
2 Likes
12

Here is what I tried, as per your suggestion:

df = DataFrame(a=repeat([1,2], outer=3), b=repeat([1,2,3], outer=2), c=1:6)
gdf=groupby(df, :a)
transform(gdf, [:b, :c] => ((b, c) -> min.(b, c)) => :min_b_c)

And this this what I’m getting:

MethodError: objects of type Tuple{Int64, Int64, Int64} are not callable

Stacktrace:
[1] _combine(gd::GroupedDataFrame{DataFrame}, cs_norm::Vector{Any}, optional_transform::Vector{Bool}, copycols::Bool, keeprows::Bool, renamecols::Bool, threads::Bool)
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:755
[2] _combine_prepare_norm(gd::GroupedDataFrame{DataFrame}, cs_vec::Vector{Any}, keepkeys::Bool, ungroup::Bool, copycols::Bool, keeprows::Bool, renamecols::Bool, threads::Bool)
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:87
[3] _combine_prepare(gd::GroupedDataFrame{DataFrame}, ::Base.RefValue{Any}; keepkeys::Bool, ungroup::Bool, copycols::Bool, keeprows::Bool, renamecols::Bool, threads::Bool)
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:52
[4] _combine_prepare
@ C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:26 [inlined]
[5] select(::GroupedDataFrame{DataFrame}, ::Union{Regex, AbstractString, Function, Signed, Symbol, Unsigned, Pair, Type, All, Between, Cols, InvertedIndex, AbstractVecOrMat}, ::Vararg{Union{Regex, AbstractString, Function, Signed, Symbol, Unsigned, Pair, Type, All, Between, Cols, InvertedIndex, AbstractVecOrMat}}; copycols::Bool, keepkeys::Bool, ungroup::Bool, renamecols::Bool, threads::Bool)
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:892
[6] transform(gd::GroupedDataFrame{DataFrame}, args::Union{Regex, AbstractString, Function, Signed, Symbol, Unsigned, Pair, Type, All, Between, Cols, InvertedIndex, AbstractVecOrMat}; copycols::Bool, keepkeys::Bool, ungroup::Bool, renamecols::Bool, threads::Bool)
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:917
[7] transform(gd::GroupedDataFrame{DataFrame}, args::Union{Regex, AbstractString, Function, Signed, Symbol, Unsigned, Pair, Type, All, Between, Cols, InvertedIndex, AbstractVecOrMat})
@ DataFrames C:\Users\USUARIO.julia\packages\DataFrames\58MUJ\src\groupeddataframe\splitapplycombine.jl:912
[8] top-level scope
@ In[202]:3

13

That code works fine for me. There must be some other code you are running thats broken.

Bumping the DataFramesMeta.jl solution though (as the mantainer of the package). It’s pretty easy to use imo.

14

Thanks for posting the details, unfortunately the example runs fine for me as well.
The only way, I can reproduce this error is by shadowing min, i.e.,

julia> let min = (1,2,3)
           transform(gdf, [:b, :c] => ((b, c) -> min.(b, c)) => :min_b_c)
       end
ERROR: MethodError: objects of type Tuple{Int64, Int64, Int64} are not callable

Are you running this line inside a function with an argument min or otherwise managed to redefine it?

1 Like