Help generating an Expression

General Usage
,
1

I have some wide-formatted data that I need to group and summarize. A MWE is below:

using DataFramesMeta

df = DataFrame(
    id = repeat(1:5, inner=3),
    a_1 = rand(15),
    a_2 = rand(15),
    a_3 = rand(15),
    b_1 = rand(15),
    b_2 = rand(15),
    b_3 = rand(15)
)

summary = @linq df |>
    by(
        :id,
        a_1 = sum(:a_1),
        a_2 = sum(:a_2),
        a_3 = sum(:a_3),
        b_1 = sum(:b_1),
        b_2 = sum(:b_2),
        b_3 = sum(:b_3),
    )

julia> summary
5×7 DataFrame
│ Row │ id    │ a_1     │ a_2      │ a_3      │ b_1      │ b_2     │ b_3     │
│     │ Int64 │ Float64 │ Float64  │ Float64  │ Float64  │ Float64 │ Float64 │
├─────┼───────┼─────────┼──────────┼──────────┼──────────┼─────────┼─────────┤
│ 1   │ 1     │ 2.34505 │ 0.881786 │ 0.969993 │ 0.635778 │ 1.69327 │ 1.22705 │
│ 2   │ 2     │ 1.24684 │ 0.488665 │ 1.65159  │ 1.85681  │ 1.63358 │ 1.05604 │
│ 3   │ 3     │ 2.05449 │ 0.928822 │ 1.53021  │ 1.66948  │ 1.8532  │ 1.69901 │
│ 4   │ 4     │ 1.77823 │ 1.19143  │ 1.36264  │ 1.13891  │ 1.90184 │ 1.46904 │
│ 5   │ 5     │ 1.00121 │ 1.91585  │ 1.11492  │ 1.69419  │ 1.58653 │ 1.98337 │

This is fine but in my real example I have hundreds of columns so I don’t want to write out

    by(
        :id,
        a_1 = sum(:a_1),
        a_2 = sum(:a_2),
        a_3 = sum(:a_3),
        b_1 = sum(:b_1),
        b_2 = sum(:b_2),
        b_3 = sum(:b_3),
    )

this portion of the code. The column names follow a pattern so I was hoping to be able to generate an Expr that I could eval inside of by but I cannot figure out how to do this.

Something like:

julia> for l in (:a,:b), i in 1:3
           println("$(l)_$i = sum(:$(l)_$i),")
       end
a_1 = sum(:a_1),
a_2 = sum(:a_2),
a_3 = sum(:a_3),
b_1 = sum(:b_1),
b_2 = sum(:b_2),
b_3 = sum(:b_3),

but creating an Expr that I can inject into the by function above and eval.

Is such a thing possible or is there a better way to do this? Or am I stuck writing it all out?

2

Don’t use dataframesmeta for this, just use dataframes!

combine(groupby(df, :id), names(df, r"^a_") .=> sum, names(df, r"^b_") .=> sum)
4 Likes
3

I’ve been using DataFramesMeta for so long now that I forget how good regular ol’ DataFrames is :wink:

4

Well, this is (relatively) new functionality.

DataFramesMeta master right now uses transform, combine etc. as the back-end for @transform, @by etc.

So it should be prototype something in DataFramesMeta and then translate the expression to something without any literals and write it in DataFrames.

2 Likes