# \[ANN-RFC\] DFMacros.jl

**URL:** <https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770>\
**Category:** Package Announcements\
**Tags:** dataframes\
**Created:** [June 11, 2021, 11:11pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770 "2021-06-11T23:11:54Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 11, 2021, 11:11pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/1 "2021-06-11T23:11:54Z")

</div>

Hi there,

I really like using DataFrames, but what I don’t like so much is redundant syntax and creating all these anonymous functions by hand. I’ve used DataFramesMeta for a while and it’s great, but it’s not quite the right fit for me. That’s why I’m making a new package that does things exactly how I like them, in an opinionated way.

> **[GitHub - jkrumbiegel/DataFrameMacros.jl: Macros that simplify working with DataFrames.jl](https://github.com/jkrumbiegel/DataFrameMacros.jl)**
>
> Macros that simplify working with DataFrames.jl

Here’s the current readme. The RFC in the title relates mostly to the name, I’m not really happy with DFMacros 🙂. Maybe you have better ideas!

# DFMacros.jl

The following macros are currently available:

- `@transform`
- `@select`
- `@groupby`
- `@combine`
- `@subset`
- `@sort`

These are the most important opinionated aspects that differ from other packages:

- `@transform`, `@select` and `@subset` work row-wise by default, `@combine` works column-wise by default. This matches the most common modes these functions are used in and reduces friction.
- `@groupby` and `@sort` allow using arbitrary expressions including multiple columns, without having to `@transform` first and repeat the new column names.
- Keyword arguments to the macro-underlying functions work by separating them from column expressions with the `;` character.
- Column expressions are interpolated into the macro with `$`.
- Target column names are written with `:` symbols to avoid visual ambiguity (`:newcol = ...`). This also allows to use `AsTable` as a target like in DataFrames.jl.
- A flag macro (`@c` or `@r`) can be used to switch between row/column-based mode.
- The flag macro can also include the character `m` to switch on automatic `passmissing` in row-wise mode.

## Examples

```julia
using DFMacros
using DataFrames
using Random
using Statistics
Random.seed!(123)

df = DataFrame(
    id = shuffle(1:5),
    group = rand('a':'b', 5),
    weight_kg = randn(5) .* 5 .+ 60,
    height_cm = randn(5) .* 10 .+ 170)

```

```julia-auto
5×4 DataFrame
 Row │ id group weight_kg height_cm
     │ Int64 Char Float64 Float64
─────┼────────────────────────────────────
   1 │ 1 b 64.9048 161.561
   2 │ 4 b 59.6226 161.111
   3 │ 2 a 61.3691 173.272
   4 │ 3 a 59.0289 175.924
   5 │ 5 b 58.3032 173.68

```

### @select

```julia
@select(df, :height_m = :height_cm / 100)

```

```julia-auto
5×1 DataFrame
 Row │ height_m
     │ Float64
─────┼──────────
   1 │ 1.61561
   2 │ 1.61111
   3 │ 1.73272
   4 │ 1.75924
   5 │ 1.7368

```

```julia
@select(df, AsTable = (w = :weight_kg, h = :height_cm))

```

```julia-auto
5×2 DataFrame
 Row │ w h
     │ Float64 Float64
─────┼──────────────────
   1 │ 64.9048 161.561
   2 │ 59.6226 161.111
   3 │ 61.3691 173.272
   4 │ 59.0289 175.924
   5 │ 58.3032 173.68

```

### @transform

```julia
@transform(df, :weight_g = :weight_kg / 1000)

```

```julia-auto
5×5 DataFrame
 Row │ id group weight_kg height_cm weight_g
     │ Int64 Char Float64 Float64 Float64
─────┼───────────────────────────────────────────────
   1 │ 1 b 64.9048 161.561 0.0649048
   2 │ 4 b 59.6226 161.111 0.0596226
   3 │ 2 a 61.3691 173.272 0.0613691
   4 │ 3 a 59.0289 175.924 0.0590289
   5 │ 5 b 58.3032 173.68 0.0583032

```

```julia
@transform(df, :BMI = :weight_kg / (:height_cm / 100) ^ 2)

```

```julia-auto
5×5 DataFrame
 Row │ id group weight_kg height_cm BMI
     │ Int64 Char Float64 Float64 Float64
─────┼─────────────────────────────────────────────
   1 │ 1 b 64.9048 161.561 24.8658
   2 │ 4 b 59.6226 161.111 22.9701
   3 │ 2 a 61.3691 173.272 20.4405
   4 │ 3 a 59.0289 175.924 19.0728
   5 │ 5 b 58.3032 173.68 19.3282

```

#### column flag @c

```julia
@transform(df, :weight_z = @c (:weight_kg .- mean(:weight_kg)) / std(:weight_kg))

```

```julia-auto
5×5 DataFrame
 Row │ id group weight_kg height_cm weight_z
     │ Int64 Char Float64 Float64 Float64
─────┼───────────────────────────────────────────────
   1 │ 1 b 64.9048 161.561 1.61523
   2 │ 4 b 59.6226 161.111 -0.388008
   3 │ 2 a 61.3691 173.272 0.274332
   4 │ 3 a 59.0289 175.924 -0.613175
   5 │ 5 b 58.3032 173.68 -0.888383

```

### @groupby & @combine

```julia
g = @groupby(df, iseven(:id))

```

```julia-auto
GroupedDataFrame with 2 groups based on key: id_iseven
Group 1 (3 rows): id_iseven = false
 Row │ id group weight_kg height_cm id_iseven
     │ Int64 Char Float64 Float64 Bool
─────┼───────────────────────────────────────────────
   1 │ 1 b 64.9048 161.561 false
   2 │ 3 a 59.0289 175.924 false
   3 │ 5 b 58.3032 173.68 false
Group 2 (2 rows): id_iseven = true
 Row │ id group weight_kg height_cm id_iseven
     │ Int64 Char Float64 Float64 Bool
─────┼───────────────────────────────────────────────
   1 │ 4 b 59.6226 161.111 true
   2 │ 2 a 61.3691 173.272 true

```

```julia
@combine(g, :total_weight_kg = sum(:weight_kg))

```

```julia-auto
2×2 DataFrame
 Row │ id_iseven total_weight_kg
     │ Bool Float64
─────┼────────────────────────────
   1 │ false 182.237
   2 │ true 120.992

```

### @sort

```julia
@sort(df, -sqrt(:height_cm))

```

```julia-auto
5×4 DataFrame
 Row │ id group weight_kg height_cm
     │ Int64 Char Float64 Float64
─────┼────────────────────────────────────
   1 │ 3 a 59.0289 175.924
   2 │ 5 b 58.3032 173.68
   3 │ 2 a 61.3691 173.272
   4 │ 1 b 64.9048 161.561
   5 │ 4 b 59.6226 161.111

```

### passmissing flag @m

```julia
df = DataFrame(name = ["joe", "jim", missing, "james"])

@transform(df, :cap_name = @m uppercasefirst(:name))

```

```julia-auto
4×2 DataFrame
 Row │ name cap_name
     │ String? String?
─────┼───────────────────
   1 │ joe Joe
   2 │ jim Jim
   3 │ missing missing
   4 │ james James

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [June 11, 2021, 11:40pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/2 "2021-06-11T23:40:40Z")

</div>

Hey! I’m glad you put development into this.

As the maintainer of DataFramesMeta I have to say I’m a bit bummed you chose to fork the package. In particular, a lot of the things that this package is trying to do I think are just around the corner in DataFramesMeta

1. row-wise by default. We are almost done with the addition of a `@byrow` flag to allow for row-wise operations. See [here](https://github.com/JuliaData/DataFramesMeta.jl/pull/250). In particular I really like your idea for `.=` and it should be easy to add after it’s finished.
2. `groupby` with expressions. I think this is a great idea and should definitely be added to DataFramesMeta.
3. Interpolation with `$`. I think this is probably the direction to go in.
4. `:x` instead of `x`. I think this is probably the move as well. I have procrastinated because it’s a very breaking change.
5. Macro flags were hard with the design, but with the addition of the [block syntax](https://github.com/JuliaData/DataFramesMeta.jl/pull/245) more flags can be added in the future.

In short I think a lot of these should be added to DataFramesMeta, I think it would be a shame to splinter the ecosystem with multiple packages that implement `@transform` etc.

Should we try to agree on a plan to upstream lots of these changes into DataFramesMeta?

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 12, 2021, 5:51am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/3 "2021-06-12T05:51:14Z")

</div>

Yeah it would be nice if some of these things made their way into DataFramesMeta, I contributed some of the code there myself so it’s not that I don’t have any interest in that package. It’s more that such a package is, to me, mostly about convenience, so for example `@byrow` is not really what I want to write all the time, and I’ve noticed in my analysis style I use it almost every line. That’s unlikely to change in DataFramesMeta, right?

About all the other points, note how you said they should or could be added, but were pretty breaking. I’ve followed all the discussions for some months, and it seemed unlikely to me that the project would fully go in this direction, or it would at least take quite long as DataFramesMeta has too many users already which need to be accommodated. To some degree, I need an analysis package for my work now, so I can’t wait for all that to resolve. It’s also not really a fork, I wrote this from scratch to handle some design issues deeper down.

This is more of a “if someone out there happens to have the exact same pain points as me, here could be a solution” thing. It’s not an infrastructure but an end user package, so there’s in my mind no danger of fracturing anything.

So to sum up, dataframesmeta is great, I’m sure the issues mentioned above will resolve at some point one way or another, yet there is also enough space for slightly different approaches I think (the slightly matters to me a lot)

---

<div class="post-metadata">

**Author:** ![eliascarv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eliascarv/32/22152_2.png) [@eliascarv](https://discourse.julialang.org/u/eliascarv)\
**Post date:** [June 12, 2021, 9:16am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/4 "2021-06-12T09:16:52Z")

</div>

Some suggestions for the package name:

DFTools.jl  
DFToolbox.jl  
DFManipulation.jl  
DFManipulations.jl  
DataManipulation.jl

I didn’t find good names either 😅.

But I love your package idea, the syntax is perfect! It was definitely the data manipulation package I was looking for.

Congratulations on your amazing work!

---

<div class="post-metadata">

**Author:** ![EvoArt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/evoart/32/25357_2.png) [@EvoArt](https://discourse.julialang.org/u/EvoArt)\
**Post date:** [June 12, 2021, 10:14am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/5 "2021-06-12T10:14:12Z")

</div>

This looks great. I think this will be more convenient for my analysis work flow than anything else I’ve seen in Julia.

I agree that there should be space for different approaches.

I do have one thought though, re convenience and splintering of ecosystem. Would it be worth having a way to set your default options in dataframesmeta like `dfmoptions(groupby = :row)` or some such? Maybe this already exists?

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 12, 2021, 10:50am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/6 "2021-06-12T10:50:24Z")

</div>

I have also thought about that. Could be an option 🙂 Maybe it’s too complicated if you need to keep this option in mind to understand code

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [June 12, 2021, 11:24am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/7 "2021-06-12T11:24:38Z")

</div>

Amazing @jules ! Any chance the package could be generalized to Tables.jl tables as well? That would really solve the major pain point I have with the alternatives. The only package I can use currently is Query.jl because of its generality but it would be nice to see other approaches that work with Tables.jl and have clean syntax.

---

<div class="post-metadata">

**Author:** ![EvoArt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/evoart/32/25357_2.png) [@EvoArt](https://discourse.julialang.org/u/EvoArt)\
**Post date:** [June 12, 2021, 11:50am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/8 "2021-06-12T11:50:52Z")

</div>

Good point!

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [June 13, 2021, 11:19am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/9 "2021-06-13T11:19:30Z")

</div>

Very nice and well thought out.

> [@pdeffebach](#):
>
> I’m a bit bummed you chose to fork the package

> [@jules](#):
>
> it seemed unlikely to me that the project would fully go in this direction

i can see both sides. But I think a bit more choice is never a bad thing. I will definitely give DFMacros a try. On the surface, I will probably adopt DFMacros for my workflow.

> [@jules](#):
>
> row-wise by default,

Will this have performance implications? Or is it just autobroadcasting and saving of `.` typiing.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 13, 2021, 11:47am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/10 "2021-06-13T11:47:42Z")

</div>

Just auto wrapping in ByRow because I think it’s a better default for everything but combine. Many string logic things are annoying to write in broadcasting style for example, or if you have unusual objects where you need to index or access properties. It also makes missings easier to handle, because you can wrap the auto-byrow function in passmissing with the `@m` flag. Which again pertains mostly to string and object manipulation as number functions such as + and \* in many cases already propagate missings.

To add one more thought, I think the crux is that broadcasting is most useful where different dimensions come together. But in DataFrames, everything is forced to be same-length vectors anyway.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 16, 2021, 9:55pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/11 "2021-06-16T21:55:50Z")

</div>

> [@jules](#):
>
> The RFC in the title relates mostly to the name, I’m not really happy with DFMacros

There is an issue in DataFramesMeta.jl about a possible rename where the name DataFramesMacros.jl was suggested. However, it looks like the maintainers are leaning towards staying with the name DataFramesMeta. So, if they decide to stay with DataFramesMeta, it might be reasonable to rename DFMacros.jl to DataFramesMacros.jl.

> <https://github.com/JuliaData/DataFramesMeta.jl/issues/175>
>
> DataFramesMeta is not a very clear name to new users. "Meta" means metaprogrammi…ng in this context, of course, but it's not clear form the name. 
> 
> Alternatives: DataFramesMacros?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [June 16, 2021, 10:07pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/12 "2021-06-16T22:07:50Z")

</div>

If you are considering Tables.jl tables in the future, maybe a more general name would make sense without DataFrame on it. TableTools.jl or TableMacros.jl, something short to type.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 17, 2021, 6:11am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/13 "2021-06-17T06:11:29Z")

</div>

All the macros forward to dataframes functions, I’m not sure if that can be made generic for all tables.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 17, 2021, 6:15am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/14 "2021-06-17T06:15:16Z")

</div>

I thought about that as well, but then thought it was a bit too similar to DataFramesMeta and could be confusing. I think DF for DataFrame is common enough as an abbreviation no?

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [June 17, 2021, 8:36am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/15 "2021-06-17T08:36:03Z")

</div>

Respectfully … no.  
While DF for DataFrame is common enough to be understood by those who commonly use DF to mean DataFrame, most of the Julia Community members who may utilize DataFrames for some purpose are not them.

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [June 17, 2021, 8:52am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/16 "2021-06-17T08:52:50Z")

</div>

I like the name `DFMacros` 🙂 Someone who has no idea about data frames will have to learn about DataFrames.jl anyway so I don’t see this as an obstacle.

As for the redundancy with DataFramesMeta: I also prefer when there is one standard way to do things in a mature ecosystem. It helps a lot with readability and getting familiar with other people’s code. But maybe it’s too early to say what the “standard way” should be here, so I’m happy to see another package trying different things… Better to try things with complete freedom and gather the best parts in a later package.

> [@EvoArt](#):
>
> Would it be worth having a way to set your default options in dataframesmeta like `dfmoptions(groupby = :row)` or some such?

I’d rather avoid having global state change the meaning of the code. It would be quite bad for readability and sharing code. Imagine if every time you find a solution on Discourse you have to check if it’s valid for your particular defaults…

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 17, 2021, 9:15am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/17 "2021-06-17T09:15:10Z")

</div>

> I also prefer when there is one standard way to do things in a mature ecosystem

I think this is an important point, let me stress that I do not aim for this package to become the “standard way”, I’m just putting this up for like-minded people. If DataFramesMeta maintains the close coupling with DataFrames, for example being mentioned explicitly in the documentation, etc, I don’t think there’s a danger for this package to interfere with that.

I would also say that because such macro packages are quite simple (mine is 260 lines of code), it’s not much wasted effort to make a new one. It would be a different story if I attempted to make a whole new DataFrames.jl with a couple of smaller changes.

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [June 17, 2021, 9:18am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/18 "2021-06-17T09:18:55Z")

</div>

I am someone who is familiar with data frames and their use. `DFMacros` is not transparent to me (DataFlow is a more common DF than DataFrame) – this is an example of using an acronym/abbreviation in package naming that the guidelines prefer to avoid ([5. Creating Packages · Pkg.jl](https://pkgdocs.julialang.org/v1/creating-packages/#Package-naming-guidelines)).

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 17, 2021, 9:21am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/19 "2021-06-17T09:21:40Z")

</div>

I agree in principle, but don’t you think `DataFrameMacros` and `DataFramesMeta` is a bit close? I’m not particularly attached to `DFMacros`

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [June 17, 2021, 9:32am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/20 "2021-06-17T09:32:52Z")

</div>

@JeffreySarnoff good point regarding the guidelines! Though maybe that one makes less sense for a package that builds on another package, as we have here… Also I care a lot about clarity in the API (e.g. function names) but I don’t think package names need to be transparent.

Many software projects don’t have transparent names, they just have a name, which is part of their identity. A few random examples: Gurobi, Stan, Ansys, Gumbo, Pango, GTK, Qt, TensorFlow.

Or some more Julian examples: Turing, Gen, Makie, Zygote, Flux. I don’t think there’s anything wrong with these names.

@jules just in case it was not clear, I think it’s good to have this new package now, and maybe consolidate in the future.

[Next page](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770.md?page=2)