# \[ANN-RFC\] DFMacros.jl

**URL:** <https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770>\
**Category:** Package Announcements\
**Tags:** dataframes\
**Created:** [June 11, 2021, 11:11pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770 "2021-06-11T23:11:54Z")\
**Posts on this page:** 11\
**Page:** 2

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [June 17, 2021, 10:14am UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/21 "2021-06-17T10:14:11Z")

</div>

Note that many table types already overload common Julia functions such as `map`. This single API makes it very convenient and consistent to work with them. Add `DataPipes.jl` _(disclaimer: my package)_ on top to reduce syntactic boilerplate - and turns out that basically all operations from the first post are easy to write for a wide variety of tables!  
My translation of those examples to `TypedTables.Table` is below. It also works as-is with other implementations such as `Tables.rowtable` (vector of namedtuples). As a bonus, all operations are easily “pipeable” (:

```julia
julia> using Random
julia> using SplitApplyCombine
julia> using Tables
julia> using TypedTables
julia> using DataPipes
julia> using Missings

julia> table = Table(
           id = shuffle(1:5),
           group = rand('a':'b', 5),
           weight_kg = randn(5) .* 5 .+ 60,
           height_cm = randn(5) .* 10 .+ 170
       )
Table with 4 columns and 5 rows:
     id group weight_kg height_cm
   ┌────────────────────────────────
 1 │ 1 a 55.1194 150.885
 2 │ 4 b 55.2233 164.44
 3 │ 3 a 51.9789 178.343
 4 │ 5 a 59.483 166.938
 5 │ 2 b 53.3129 179.829

julia> @p table |> map((height_m = _.height_cm / 100,))
Table with 1 column and 5 rows:
     height_m
   ┌─────────
 1 │ 1.50885
 2 │ 1.6444
 3 │ 1.78343
 4 │ 1.66938
 5 │ 1.79829

julia> @p table |> map((w = _.weight_kg, h = _.height_cm))
Table with 2 columns and 5 rows:
     w h
   ┌─────────────────
 1 │ 55.1194 150.885
 2 │ 55.2233 164.44
 3 │ 51.9789 178.343
 4 │ 59.483 166.938
 5 │ 53.3129 179.829

julia> @p table |> mutate(weight_g = _.weight_kg / 1000)
Table with 5 columns and 5 rows:
     id group weight_kg height_cm weight_g
   ┌───────────────────────────────────────────
 1 │ 1 a 55.1194 150.885 0.0551194
 2 │ 4 b 55.2233 164.44 0.0552233
 3 │ 3 a 51.9789 178.343 0.0519789
 4 │ 5 a 59.483 166.938 0.059483
 5 │ 2 b 53.3129 179.829 0.0533129

julia> @p table |> mutate(BMI = _.weight_kg / (_.height_cm / 100)^2)
Table with 5 columns and 5 rows:
     id group weight_kg height_cm BMI
   ┌─────────────────────────────────────────
 1 │ 1 a 55.1194 150.885 24.2111
 2 │ 4 b 55.2233 164.44 20.4223
 3 │ 3 a 51.9789 178.343 16.3424
 4 │ 5 a 59.483 166.938 21.3444
 5 │ 2 b 53.3129 179.829 16.486

julia> g = @p table |> group(iseven(_.id))
2-element Dictionaries.Dictionary{Bool, Table{NamedTuple{(:id, :group, :weight_kg, :height_cm), Tuple{Int64, Char, Float64, Float64}}, 1, NamedTuple{(:id, :group, :weight_kg, :height_cm), Tuple{Vector{Int64}, Vector{Char}, Vector{Float64}, Vector{Float64}}}}}
 false │ Table with 4 columns and 3 rows:
     id group weight_kg height_cm
   ┌────────────────────────────────
 1 │ 1 a 55.1194 150.885
 2 │ 3 a 51.9789 178.343
 3 │ 5 a 59.483 166.938
  true │ Table with 4 columns and 2 rows:
     id group weight_kg height_cm
   ┌────────────────────────────────
 1 │ 4 b 55.2233 164.44
 2 │ 2 b 53.3129 179.829

julia> @p g |> map(@p(sum(_.weight_kg, _1)))
2-element Dictionaries.Dictionary{Bool, Float64}
 false │ 166.58126233561455
  true │ 108.53621402076169

julia> @p table |> sort(by=-sqrt(_.height_cm))
Table with 4 columns and 5 rows:
     id group weight_kg height_cm
   ┌────────────────────────────────
 1 │ 2 b 53.3129 179.829
 2 │ 3 a 51.9789 178.343
 3 │ 5 a 59.483 166.938
 4 │ 4 b 55.2233 164.44
 5 │ 1 a 55.1194 150.885

```

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 17, 2021, 12:27pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/22 "2021-06-17T12:27:21Z")

</div>

DF also sometimes stands for “degrees of freedom”.

> [@jules](#):
>
> don’t you think `DataFrameMacros` and `DataFramesMeta` is a bit close?

I think DataFrame[s]Macros is different enough from DataFramesMeta. At any rate, it’s likely that other DataFrame\_\_\_.jl packages will be added to the general registry in the future.

Regarding whether the [s] should be included… I suppose it makes more sense grammatically to omit the [s]. 🙂

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [June 17, 2021, 12:48pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/23 "2021-06-17T12:48:25Z")

</div>

I don’t think it’s a great idea to have two packages `DataFramesMeta` and `DataFramesMacros` that both export `@transform`, `@select` etc. I think the names are too similar.

Given that the key difference between the two is that in DFMacros, operations are by row by default, perhaps some reference to this could be in the name for now?

Another alternative would be to rename some macros. As far as I know, Stata-based puns are open for the taking, i.e. `@generate` to make new columns, `@keep` to keep columns, etc. This would lead to less confusion among new users.

---

<div class="post-metadata">

**Author:** ![Satvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/satvik/32/20486_2.png) [@Satvik](https://discourse.julialang.org/u/Satvik)\
**Post date:** [June 17, 2021, 2:00pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/24 "2021-06-17T14:00:32Z")

</div>

These are pretty standard names within other DataFrames ecosystems (R, Python) and IIUC these macros are doing the same things, but with different defaults / implementation. So having the same names seems correct to me.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 17, 2021, 2:35pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/25 "2021-06-17T14:35:57Z")

</div>

Yeah, I don’t think we would want to break the direct correspondences ` transfrom -> @transform`, `select -> @select`, etc.

---

<div class="post-metadata">

**Author:** ![ggggggggg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ggggggggg/32/265_2.png) [@ggggggggg](https://discourse.julialang.org/u/ggggggggg)\
**Post date:** [June 17, 2021, 2:39pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/26 "2021-06-17T14:39:01Z")

</div>

What about having two sets of macros. ‘@select’ and ‘@Cselect’ for by row and by column respectively?

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 19, 2021, 2:07pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/27 "2021-06-19T14:07:03Z")

</div>

Then you can’t pass different modes to one transform call. I considered that and found it too unflexible.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 19, 2021, 2:08pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/28 "2021-06-19T14:08:12Z")

</div>

I have still not registered the package due to the name issue, but I’ve added block syntax and an auto-table mode which might be useful for some scenarios. Here’s an excerpt from the Readme again:

### `@t` flag macro for automatic `AsTable`

To use `AsTable` as a target, you usually have to construct a NamedTuple in the passed function.  
You can avoid both passing `AsTable` explicitly and constructing the NamedTuple by using the `@t` flag macro.  
All expressions of the type `:symbol = expression` are collected, the `:symbol`s are replaced with anonymous variables, and these variables are collected in a NamedTuple as the return value automatically.

```julia
df = DataFrame(a = 1:3, b = 4:6)
df2 = @transform df @t begin
    x = :a + :b
    :y = x * 2
    :z = x + 4
end

```

```julia-auto
3×4 DataFrame
 Row │ a b y z
     │ Int64 Int64 Int64 Int64
─────┼────────────────────────────
   1 │ 1 4 10 9
   2 │ 2 5 14 11
   3 │ 3 6 18 13

```

---

<div class="post-metadata">

**Author:** ![eliascarv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eliascarv/32/22152_2.png) [@eliascarv](https://discourse.julialang.org/u/eliascarv)\
**Post date:** [June 19, 2021, 4:30pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/29 "2021-06-19T16:30:08Z")

</div>

Opinion: You could add Chain.jl as a dependency of DataFrameMacros.jl and re-export the `@chain` macro.

What do you think @jules? It would be a good idea?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 19, 2021, 4:43pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/30 "2021-06-19T16:43:01Z")

</div>

Wait, why does a DiffEqFlux macros package have a bunch of tables stuff in it?

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 19, 2021, 6:02pm UTC](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770/31 "2021-06-19T18:02:08Z")

</div>

I’ve decided to register as DataFrameMacros, DF is not clear enough and that’s just what it is, macros for DataFrames.

[Previous page](https://discourse.julialang.org/t/ann-rfc-dfmacros-jl/62770.md?page=1)
