Run multiple instances of transform on specific column combinations of a GroupedDataFrame in DataFrames mini language

New to Julia
,
21

Thank you so much! Sorry I would have liked to mark multiple solutions but wasn’t sure how to do it. If you have a moment could you clarify how the zip and ... commands are working here?

When I add the splat ... operator after the Cols argument in the function declaration does that turn that argument in the function into an array? So when I input Cols... into funnynt it would be in the form [col1,col2.col3]

Also I am not sure what I am getting wrong here. But when I try

df  = DataFrame(id=rand(1:3,10), A=rand(11:13,10),       
              B=rand(101:103,10), C = rand(25:27,10), D = rand(32:34,10))
10×5 DataFrame
 Row │ id     A      B      C      D     
     │ Int64  Int64  Int64  Int64  Int64 
─────┼───────────────────────────────────
   1 │     1     12    102     26     32
   2 │     1     13    101     25     34
   3 │     2     13    102     26     33
   4 │     3     11    102     27     33
   5 │     1     13    102     26     33
   6 │     3     11    101     25     32
   7 │     1     11    103     26     34
   8 │     2     12    101     27     34
   9 │     2     12    102     25     32
  10 │     2     12    101     25     32

gdf = groupby(df,:id)
GroupedDataFrame with 3 groups based on key: id
First Group (4 rows): id = 1
 Row │ id     A      B      C      D     
     │ Int64  Int64  Int64  Int64  Int64 
─────┼───────────────────────────────────
   1 │     1     12    102     26     32
   2 │     1     13    101     25     34
   3 │     1     13    102     26     33
   4 │     1     11    103     26     34
⋮
Last Group (2 rows): id = 3
 Row │ id     A      B      C      D     
     │ Int64  Int64  Int64  Int64  Int64 
─────┼───────────────────────────────────
   1 │     3     11    102     27     33
   2 │     3     11    101     25     32

function funnynt(id,Cols...; p1=1,p2=2,p3=3)
                  m= Matrix{Float64}(undef, length(id),length(Cols))
                  for i in eachindex(Cols)
                      if p3>0
                          m[:,i]= (id .+p1).*(Cols[i].+p2*p3)
                      else
                          m[:,i]= (id .+p1).*(Cols[i].+p2)
                      end
                  end
                  ncols=Symbol.("id_",names(df,Not(:id)))
                  return (;zip(ncols,eachcol(m))...)
              end
funnynt (generic function with 1 method)

transform(gdf,Cols(:) => funnynt => AsTable)

I get

ERROR: NamedTuple names and field types must have matching lengths

Again I apologize if I am overlooking something elementary, but thanks so much for your time and help.

22

Here my log, starting from a clean session.
See if you can see any difference.

julia> using DataFrames

julia> df = DataFrame(id=rand(1:3,10), A=rand(11:13,10),
       B=rand(101:103,10), C = rand(25:27,10), D = rand(32:34,10))
10×5 DataFrame
 Row │ id     A      B      C      D     
     │ Int64  Int64  Int64  Int64  Int64 
─────┼───────────────────────────────────
   1 │     1     13    103     26     32
   2 │     2     13    101     26     33
   3 │     3     13    103     25     32
   4 │     2     11    101     26     32
   5 │     2     12    102     25     34
   6 │     1     11    102     25     34
   7 │     2     11    103     26     32
   8 │     3     13    102     25     32
   9 │     1     12    103     25     33
  10 │     3     11    101     26     32

julia> gdf = groupby(df, :id)
GroupedDataFrame with 3 groups based on key: id
First Group (3 rows): id = 1
 Row │ id     A      B      C      D     
     │ Int64  Int64  Int64  Int64  Int64 
─────┼───────────────────────────────────
   1 │     1     13    103     26     32
   2 │     1     11    102     25     34
   3 │     1     12    103     25     33
⋮
Last Group (3 rows): id = 3
 Row │ id     A      B      C      D     
     │ Int64  Int64  Int64  Int64  Int64 
─────┼───────────────────────────────────
   1 │     3     13    103     25     32
   2 │     3     13    102     25     32
   3 │     3     11    101     26     32

julia> function funnynt(id,Cols...; p1=1,p2=2,p3=3)
           m= Matrix{Float64}(undef, length(id),length(Cols))   
           for i in eachindex(Cols)
               if p3>0
                   m[:,i]= (id .+p1).*(Cols[i].+p2*p3)
               else
                   m[:,i]= (id .+p1).*(Cols[i].+p2)
               end
           end
           #ncols=Symbol.("id_c",1:length(Cols))
           ncols=Symbol.("id_",names(df,Not(:id)))

           return (;zip(ncols,eachcol(m))...)
       end
funnynt (generic function with 1 method)

julia> transform(gdf,Cols(:) => funnynt=>AsTable)
10×9 DataFrame
 Row │ id     A      B      C      D      id_A     id_B     id_C     id_D ⋯
     │ Int64  Int64  Int64  Int64  Int64  Float64  Float64  Float64  Floa ⋯
─────┼─────────────────────────────────────────────────────────────────────
   1 │     1     13    103     26     32     38.0    218.0     64.0     7 ⋯
   2 │     2     13    101     26     33     57.0    321.0     96.0    11  
   3 │     3     13    103     25     32     76.0    436.0    124.0    15  
   4 │     2     11    101     26     32     51.0    321.0     96.0    11  
   5 │     2     12    102     25     34     54.0    324.0     93.0    12 ⋯
   6 │     1     11    102     25     34     34.0    216.0     62.0     8  
   7 │     2     11    103     26     32     51.0    327.0     96.0    11  
   8 │     3     13    102     25     32     76.0    432.0    124.0    15  
   9 │     1     12    103     25     33     36.0    218.0     62.0     7 ⋯
  10 │     3     11    101     26     32     68.0    428.0    128.0    15  
                                                           1 column omitted

julia> transform(gdf,Cols(:) =>((x...)->funnynt(x...,p1=0;p3=0))=>AsTable) 
10×9 DataFrame
 Row │ id     A      B      C      D      id_A     id_B     id_C     id_D ⋯
     │ Int64  Int64  Int64  Int64  Int64  Float64  Float64  Float64  Floa ⋯
─────┼─────────────────────────────────────────────────────────────────────
   1 │     1     13    103     26     32     15.0    105.0     28.0     3 ⋯
   2 │     2     13    101     26     33     30.0    206.0     56.0     7  
   3 │     3     13    103     25     32     45.0    315.0     81.0    10  
   4 │     2     11    101     26     32     26.0    206.0     56.0     6  
   5 │     2     12    102     25     34     28.0    208.0     54.0     7 ⋯
   6 │     1     11    102     25     34     13.0    104.0     27.0     3  
   7 │     2     11    103     26     32     26.0    210.0     56.0     6  
   8 │     3     13    102     25     32     45.0    312.0     81.0    10  
   9 │     1     12    103     25     33     14.0    105.0     27.0     3 ⋯
  10 │     3     11    101     26     32     39.0    309.0     84.0    10  
                                                           1 column omitted

As for

I don’t know if I’m able to clarify what actually happens behind the scene.
I go a bit for analogies in my experience with Julia (I don’t read much documentation, I must confess).


nt=(f1=1,f2=2,f3=3)   # basic definition of a named tuple

nt=(;(f1=1,f2=2,f3=3)...) # the function of the splatting operator is to "open" the collection


# If you need to define a namedtuple dynamically, you need to follow these steps




fields=Symbol.('f',1:3)   # a sequence of symbols

# and

values=[1,2,3] # a sequence of values

# below various ways to input it to the syntax (;kwargs...)
# which should be treated by the parser (?) in the same way as non-positional parameters of functions!?
# But here perhaps it is better for someone who knows about these things to be more specific.
(;zip(fields,values)...)
(;Pair.(fields,values)...)
(;tuple.(fields,values)...)
1 Like
23

To expand a bit on @rocco_sprmnt21’s answer:

It’s a syntax for declaring named tuples. It’s not very well documented so I’ve submitted a PR.

Basically, you can write (; :a => 1, :b => 2) to create the named tuple as if you had written (a=1, b=2). The first form is more flexible: you can generate the name programmatically (as done above with Symbol("E_$suffix")), and if you want a tuple field to have the same name and same value as a variable, you don’t have to repeat yourself: (; a=a) can be shortened to (; a). I used this above in (; E, F, G).

It’s easy to try yourself to see what Julia does, for example:

julia> f(x...) = x;

julia> f(1,2,"text")
(1, 2, "text")

so if a function is declared with x, in the body the value of x is a tuple with all the arguments passed to the function (note that for keyword arguments it’s different, you get a special object that also holds the argument names).

The description above is for the case where ... is used in the function declaration, as in your funnynt(id, Cols...; p1=1, p2=2, p3=3). In the case of (; zip(ncols,eachcol(m))...) it’s different: here the splatting operator is not in a declaration, it’s more like a call. See the FAQ.

2 Likes