# Apply function By Row without re-stating column names

**URL:** <https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765>\
**Category:** General Usage\
**Tags:** dataframes, functions\
**Created:** [August 4, 2021, 12:13am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765 "2021-08-04T00:13:53Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [August 4, 2021, 12:13am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/1 "2021-08-04T00:13:53Z")

</div>

I’d like to apply a function to each row of a DataFrame and have it add columns based on the output names. I would use transform-ByRow but I’d like to avoid restating the column names on input and output. Is there anything more elegant than the below?

```julia
function f(; a=0, b=0, _... ) 
    x = a + b    
    y = a - b
    (; x, y)
end

D = DataFrame( a=1:5, b=11:15, C = 1:5 )

@pipe eachrow(D) .|> (; _..., f(; _... )... ) |> DataFrame

```

---

<div class="post-metadata">

**Author:** ![DataFrames](https://avatars.discourse-cdn.com/v4/letter/d/e19b73/32.png) [@DataFrames](https://discourse.julialang.org/u/DataFrames)\
**Post date:** [August 4, 2021, 1:13am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/2 "2021-08-04T01:13:33Z")

</div>

```julia
transform(D, [:a, :b] => ByRow((x,y) -> f(a=x, b=y)) => AsTable)

```

or change your function signature to `function f(a,b)` and then use

```julia
transform(D, [:a, :b] => ByRow(f) => AsTable)

```

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [August 4, 2021, 1:34am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/3 "2021-08-04T01:34:20Z")

</div>

“AsTable” is really cool thank you 🙂  
It automatically maps the output column names.

This example still restates the input columns though. In practice I have a CSV file or DB table with 30 columns. My function uses 10 of them (by-name) and adds another 10 to the output table.  
I’d like to avoid naming the 10 columns - provided the function parameter names match the table field names I’d like it to match them automatically.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [August 4, 2021, 2:01am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/4 "2021-08-04T02:01:30Z")

</div>

This is really cool. I don’t think it’s possible with DataFrameMacros.jl just yet

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2021, 2:03am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/5 "2021-08-04T02:03:23Z")

</div>

```julia
Byrow(x-> f(x...))

```

??

Just a guess, or something along those lines?

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [August 4, 2021, 2:10am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/6 "2021-08-04T02:10:37Z")

</div>

Every ByRow example I’ve seen requires the input columns to be specified

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [August 4, 2021, 2:15am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/7 "2021-08-04T02:15:08Z")

</div>

> [@Lincoln\_Hannah](#):
>
> avoid restating the column names on input and output

I am thinking really hard about how to avoid input names. But it’s hard cos you can have multiple definitions of `f` possibly with different argument names! So on the surface, I don’t think it’s generally possible nor a good idea.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [August 4, 2021, 2:21am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/8 "2021-08-04T02:21:12Z")

</div>

You can do `AsTable(:) => ByRow(f) => AsTable`

But this will be awkward (large compile times) with many many columns. 30 is doable, though.

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [August 4, 2021, 2:44am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/9 "2021-08-04T02:44:21Z")

</div>

Do you have a full example?  
This doesn’t work for me.  
The closest thing is

`transform(D, AsTable(:) => ByRow( x-> f(x...)) => AsTable)`

But this requires positional arguments `function f(a,b)`  
I’d like named arguments `function f(; a, b, _...)`  
Within the 30 column table, the function might use columns 1,3,7,10,11,19,20,24,29  
I’d like these automatically matched on name.

Though I take xiaodai’s point. If you have multiple versions of a function with different arguments it would be confusing.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [August 4, 2021, 6:04am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/10 "2021-08-04T06:04:45Z")

</div>

> [@Lincoln\_Hannah](#):
>
> I’d like these automatically matched on name.

is it not possible to make your function accept a dataframe instead?

e.g.

```julia
using DataFrames
df = DataFrame(a = 1:10, b = 11:20)

function f(df)
  (x = df.a .+ df.b, )
end

transform(df, AsTable(:) => f => AsTable)

```

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [August 4, 2021, 9:15am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/11 "2021-08-04T09:15:18Z")

</div>

The function might have 100 lines.  
`a + b` is much clearer than `df.a .+ df.b` especially when there are 10 terms in the equation.

Within the function I’d like to think purely in terms of the variables (not the data-frame columns)  
I’d like to be able to call the function directly as well as apply it to the DataFrame.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [August 4, 2021, 10:00am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/12 "2021-08-04T10:00:42Z")

</div>

> [@xiaodai](#):
>
> cos you can have multiple definitions of `f` possibly with different argument names!

Ok. then for this reason, I think it will be hard to achieve.

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [August 4, 2021, 10:13am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/13 "2021-08-04T10:13:51Z")

</div>

Thanks xiaodai.  
Personally, I don’t create multiple versions of the same function. But I know its a much more important feature than what I’m suggesting. In any case, I can achieve the result with:

`@pipe eachrow(D) .|> (; _..., f(; _... )... ) |> DataFrame`

Its just not readable.

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [August 4, 2021, 12:02pm UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/14 "2021-08-04T12:02:30Z")

</div>

what if it just gave an error if multiple versions of `f` are defined.  
So to use the feature the programmer restricts themselves to a single version - not a bad trade off.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [August 4, 2021, 12:48pm UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/15 "2021-08-04T12:48:59Z")

</div>

You were really close there, just missing a semi-colon

```julia
julia> using DataFrames

julia> function foo(;a = 1, b=1, kwargs...)
           (c = a + b, d = a * b)
       end;

julia> df = DataFrame(a = [5], b = [6]);

julia> transform(df, AsTable(:) => ByRow(t -> foo(; t...)) => AsTable)
1×4 DataFrame
 Row │ a b c d     
     │ Int64 Int64 Int64 Int64 
─────┼────────────────────────────
   1 │ 5 6 11 30

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [August 4, 2021, 12:58pm UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/16 "2021-08-04T12:58:03Z")

</div>

> [@xiaodai](#):
>
> is it not possible to make your function accept a dataframe instead?

Just to note with the syntax, `AsTable(:) => fun => ...`, `fun` does _not_ get a `AbstractDataFrame`. Rather, it gets a `NamedTuple` of vectors. And in `AsTable(:) => ByRow(fun) => ...`, `fun` gets a named tuple.

The reason is because we want `src => fun => dest` to be very performant and type stable, which we can’t be if we just pass a `DataFrame` to `fun`.

> [@Lincoln\_Hannah](#):
>
> `@pipe eachrow(D) .|> (; _..., f(; _... )... ) |> DataFrame`

This is definitely confusing, but there’s always the ability to define your own function

```julia
julia> df = DataFrame(a = [5, 6], b = [7, 8]);

julia> function maprowsdf(f, df)
           map(eachrow(df)) do r
               nt = NamedTuple(r)
               res = f(; nt...)
               merge(nt, res)
           end |> DataFrame
       end;

julia> maprowsdf(foo, df)
2×4 DataFrame
 Row │ a b c d     
     │ Int64 Int64 Int64 Int64 
─────┼────────────────────────────
   1 │ 5 7 12 35
   2 │ 6 8 14 48

```

Though I think `transform` with `ByRow` is still better.

---

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [August 4, 2021, 4:16pm UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/17 "2021-08-04T16:16:48Z")

</div>

When do I want `AsTable` vs `AsTable(:)`? Is there documentation on this?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [August 4, 2021, 4:21pm UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/18 "2021-08-04T16:21:20Z")

</div>

The most complete docs are probably [here](https://dataframes.juliadata.org/stable/man/split_apply_combine/). But in general you want `AsTable(:)` in the `src` and `AsTable` in the `dest` for a `src => fun => dest` command.

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [August 4, 2021, 11:59pm UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/19 "2021-08-04T23:59:55Z")

</div>

Thanks so much for your help Peter. Underscores package helps with readability as well.  
I like both of these options.

`transform(D, @_ AsTable(:) => f(;_...) |> ByRow => AsTable)`

`@_ eachrow(D) .|> merge(_,f(;_...)) |> DataFrame`

---

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [February 10, 2022, 5:00am UTC](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765/20 "2022-02-10T05:00:48Z")

</div>

Can AsTable( : ) work within the @rtransform macro ?  
Something like ?

```julia
myfunc(; A, B) = A + B

@chain begin
    DataFrame(A=1:2,B=2:3)
    @rtransform :X = myfunc( AsTable(:)... )
end

```

[Next page](https://discourse.julialang.org/t/apply-function-by-row-without-re-stating-column-names/65765.md?page=2)