# DataFramesMeta questions

**URL:** <https://discourse.julialang.org/t/dataframesmeta-questions/69952>\
**Category:** General Usage\
**Tags:** dataframes\
**Created:** [October 17, 2021, 11:58pm UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952 "2021-10-17T23:58:38Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [October 17, 2021, 11:58pm UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/1 "2021-10-17T23:58:38Z")

</div>

Could groupby have a @ macro version, to avoid parenthesis?  
and the ability to define a column name within a groupby statement.  
Also, for a multi-line macro in a @chain block, followed by another macro - would it be possible to drop the begin and end? and define the end as the next macro  
So this

```julia
@chain DataFrame( A = 1:10 ) begin

    @rtransform :Mod3 = :A % 3
    groupby( :Mod3 )
    @combine begin
            :Asum = :A |> sum
            :Amax = :A |> maximum
    end
    @rsubset :Mod3 < 2
end

```

Could by written like this

```julia
@chain DataFrame( A = 1:10 ) begin

    @groupby :Mod3 = :A % 3

    @combine  
        :Asum = :A |> sum
        :Amax = :A |> maximum

    @rsubset :Mod3 < 2
end

```

Also within an @rtransform block. is there any way to reference the previous row?  
something like

```julia
@rtransform :Adiff = :A - prev( :A )

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 18, 2021, 12:22am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/2 "2021-10-18T00:22:32Z")

</div>

The answer to number 2 is no. According to the parser, a new line is a new statement without the begin. It’s just the way macros work.

As for number 1, it’s complicated.

This is something I would like to do, but getting the right intuitive behavior is hard.

Should

```julia
@groupby :Mod3 = :A % 3

```

allocate a new data frame? If so, that would be different from `groupby` in DataFrames.jl, which is a light-weight operation which does not allocate. What should `@groupby :g` do? Would it also allocate a new data frame?

I don’t think it would be intuitive behavior for `@groupby` to allocate but not `groupby`.

Perhaps we should not allocate a fresh data frame, and have it be an alias for

```julia
@rtransform! :Mod3 = :A % 3
groupby(:Mod3)

```

but then should it be called `@groupby!`? Maybe. These are API questions that deserve more discussion.

[Here](https://github.com/JuliaData/DataFramesMeta.jl/issues/302) is some discussion on Discourse that mentions this feature as well as some other ideas for what `@groupby` could do.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 18, 2021, 12:26am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/3 "2021-10-18T00:26:43Z")

</div>

> [@Lincoln\_Hannah](#):
>
> groupby have a @ macro

some other packages like DataFrameMacros.jl do. I think it will be good for DataFramesMeta.jl to have it too.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 18, 2021, 12:28am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/4 "2021-10-18T00:28:54Z")

</div>

It’s worth nothing that `@groupby` makes a copy of the data frame whether a new column is created or not. See [here](https://github.com/jkrumbiegel/DataFrameMacros.jl/blob/6b43ab050b870db9ecb92fa5e42d0ecfd54421cc/src/DataFrameMacros.jl#L36). I would be open to adding this but it does seem like it would make things slow for new users.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 18, 2021, 12:34am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/5 "2021-10-18T00:34:29Z")

</div>

Without reading that note, I suspect you can make `@groupby` compile to a `transform` + `groupby` unless there is a name clash etc

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [October 18, 2021, 1:03am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/6 "2021-10-18T01:03:25Z")

</div>

I added DataFrameMacros. that @groupby seems to work fine alongside DataFramesMeta commands. This even works `@groupby :Mod3 = :A % 3`. Many thanks 🙂

last bit of the question: within @rtransform, is it possible to reference the previous row. eg  
`@rtransform :Adiff = :A - prev( :A ) `

Also, I upgraded to yesterday’s release v0.10.0 but this still gives an error (Column :B not found)

```julia
@chain DataFrame(A=1) begin
       @rtransform begin
                   :B = :A
                   :C = :B
        end
end

```

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 18, 2021, 1:11am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/7 "2021-10-18T01:11:48Z")

</div>

I suspect with row-semantic u might need to create the column using `lag` first

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [October 18, 2021, 1:14am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/8 "2021-10-18T01:14:16Z")

</div>

no worries thanks

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 18, 2021, 1:39am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/9 "2021-10-18T01:39:02Z")

</div>

`transform` makes a copy, though, as well, so that doesn’t get around the problem fully.

Another name might help: `@transformgroup` or something less verbose.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 18, 2021, 1:40am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/10 "2021-10-18T01:40:11Z")

</div>

You need `@astable` there.

```julia
@chain DataFrame(A=1) begin
       @rtransform @astable begin
                   :B = :A
                   :C = :B
        end
end

```

Making interdependent columns is not the default.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 18, 2021, 1:41am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/11 "2021-10-18T01:41:18Z")

</div>

> [@Lincoln\_Hannah](#):
>
> last bit of the question: within @rtransform, is it possible to reference the previous row. eg  
> `@rtransform :Adiff = :A - prev( :A )`

No, to do this I would just use `@transform`, which is column-wise

```julia
using ShiftedArrays
@transform df :Adiff = :A .- lag(:A)

```

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [October 18, 2021, 2:03am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/12 "2021-10-18T02:03:33Z")

</div>

Thanks so much for all this 🙂

DataFramesMeta rocks

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [October 18, 2021, 11:56pm UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/13 "2021-10-18T23:56:48Z")

</div>

Is it possible to use @pipe within @chain ?

Someone gave me code for nested pipes. So \_ references the inner most pipe.

`macro pipe2(x) esc(Pipe.funnel(macroexpand( __module__ , x))) end`

is there an equivalent to allow a pipe within a chain and have \_ reference the previous pipe output

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 19, 2021, 1:52am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/14 "2021-10-19T01:52:33Z")

</div>

I don’t think so… if an outer macro and an inner macro both give `_` a special meaning, it’s hard to distinguish different meanings.

Note that Chain.jl _already_ let’s you use `_` as the previous output.

```julia
foo(x, y) = x + y
z = 1
@chain z begin 
    foo(5, _)
end

```

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [October 19, 2021, 2:02am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/15 "2021-10-19T02:02:30Z")

</div>

yes, a chain can exist within a function within a chain. and `_` references the inner chain.  
though you can’t use a pipe within a chain in the same way

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 19, 2021, 3:05am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/16 "2021-10-19T03:05:42Z")

</div>

No, unfortunately it’s hard to make macros play nice together like that.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [October 19, 2021, 5:37am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/17 "2021-10-19T05:37:28Z")

</div>

Chain inside chain is a special behavior I programmed in, it doesn’t follow from macro behavior. Macros don’t compose that way sadly.

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [October 19, 2021, 5:45am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/18 "2021-10-19T05:45:58Z")

</div>

Well its much appreciated. Thank you 🙂

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [October 25, 2021, 10:52am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/19 "2021-10-25T10:52:00Z")

</div>

How about, instead of having the @chain macro, change base Julia so the previous line’s output is always available, either through “\_” or by emitting the first argument of a function. Maybe two blank lines could be a break in the chain.

I can get similar behaviour by enclosing the whole program within a @chain macro.  
Though it can create problems when the previous line isn’t used. A double blank line indicating a chain break would fix this I think.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 25, 2021, 10:55am UTC](https://discourse.julialang.org/t/dataframesmeta-questions/69952/20 "2021-10-25T10:55:25Z")

</div>

You are aware of the `@aside` macro-flag, right? It does what you want

```julia
@chain df begin
    @rtransform :B = :A * 2
    @aside X = 1
    @rtransform :X = :B + X
end

```

See docs [here](https://github.com/jkrumbiegel/Chain.jl#the-aside-macro).

[Next page](https://discourse.julialang.org/t/dataframesmeta-questions/69952.md?page=2)