# Dataframes transform!

**URL:** <https://discourse.julialang.org/t/dataframes-transform/84041>\
**Category:** General Usage\
**Tags:** dataframes\
**Created:** [July 11, 2022, 9:23am UTC](https://discourse.julialang.org/t/dataframes-transform/84041 "2022-07-11T09:23:06Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![jerry\_ji](https://avatars.discourse-cdn.com/v4/letter/j/e36b37/32.png) [@jerry\_ji](https://discourse.julialang.org/u/jerry_ji)\
**Post date:** [July 11, 2022, 9:23am UTC](https://discourse.julialang.org/t/dataframes-transform/84041/1 "2022-07-11T09:23:06Z")

</div>

Hello!

What is right way to transform an dataframe in place with a function which return multiple columns?  
I want to write it like  
transform!(df, :A=\>(x-\>f(x))=\>[:B,:C])  
but it doesnt work.

---

<div class="post-metadata">

**Author:** ![skleinbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skleinbo/32/36080_2.png) [@skleinbo](https://discourse.julialang.org/u/skleinbo)\
**Post date:** [July 11, 2022, 9:38am UTC](https://discourse.julialang.org/t/dataframes-transform/84041/2 "2022-07-11T09:38:06Z")

</div>

**Edit** If the function returns a vector of vectors, it is interpreted as a vector of _rows_, not columns. One solution is to return a matrix instead. Notice the space instead of a comma in `x->[x.+1 x.^2]`

```julia
julia> df
3×1 DataFrame
 Row │ x
     │ Int64
─────┼───────
   1 │ 1
   2 │ 2
   3 │ 3

julia> transform!(df, :x=>(x->[x.+1 x.^2]) => [:y, :z])
3×3 DataFrame
 Row │ x y z
     │ Int64 Int64 Int64
─────┼─────────────────────
   1 │ 1 2 1
   2 │ 2 3 4
   3 │ 3 4 9

```

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [July 11, 2022, 10:17am UTC](https://discourse.julialang.org/t/dataframes-transform/84041/3 "2022-07-11T10:17:38Z")

</div>

Probably the one proposed by @skleinbo is the most typical way, but there are other ways

```julia

    transform!(df, :x=>ByRow(r->(y=r+1, z=r^2))=>AsTable)

    hcat(df,DataFrame(y=df.x.+1, z=df.x.^2))
    
    insertcols!(df,2, :y=>df.x.+1, :z=>df.x.^2)

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 11, 2022, 10:31am UTC](https://discourse.julialang.org/t/dataframes-transform/84041/4 "2022-07-11T10:31:28Z")

</div>

> [@skleinbo](#):
>
> not a vector of vectors

The function can return “vector of vectors”, but they are interpreted as rows:

```julia
julia> df = DataFrame(A=1:3)
3×1 DataFrame
 Row │ A
     │ Int64
─────┼───────
   1 │ 1
   2 │ 2
   3 │ 3

julia> transform!(df, :A => (x -> [[v+1, v+2] for v in x]) => [:B, :C])
3×3 DataFrame
 Row │ A B C
     │ Int64 Int64 Int64
─────┼─────────────────────
   1 │ 1 2 3
   2 │ 2 3 4
   3 │ 3 4 5

```

The general format of expected output with multiple output columns is:

1. If function returns one of AbstractDataFrame, NamedTuple, DataFrameRow, AbstractMatrix then columns are taken from the output columns.
2. If function returns an AbstractVector then each element of this vector must support the keys function, which must return a collection of Symbols, strings or integers; the return value of keys must be identical for all elements. Then as many columns are created as there are elements in the return value of the keys function.
3. If fun returns a value of any other type then it is assumed that it is a table conforming to the Tables.jl API and the Tables.columntable function is called on it.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [July 11, 2022, 10:59am UTC](https://discourse.julialang.org/t/dataframes-transform/84041/5 "2022-07-11T10:59:51Z")

</div>

I take this opportunity to ask for some more details on the choices made regarding the possible outputs of “fun”.

```julia
#this 

transform!(df, :x => (x -> [(v+1, v+2) for v in x]) => [:B, :C])

# is equivalent to this

transform!(df, :x => (x -> [(B=v+1, C=v+2) for v in x]) => AsTable)

```

my question is why this (array of namedtuples) works

```julia
transform!(df, :x=>ByRow(r->(y=r+1, z=r^2))=>AsTable)

```

and this (namedtuple of arrays) not

```julia
transform!(df, :x=>r->(y = r.+1, z = r.^2)=>AsTable)

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 11, 2022, 11:09am UTC](https://discourse.julialang.org/t/dataframes-transform/84041/6 "2022-07-11T11:09:12Z")

</div>

> [@rocco\_sprmnt21](#):
>
> this (namedtuple of arrays) not

It works, you just have forgotten parentheses:

```julia
julia> df = DataFrame(x=1:3)
3×1 DataFrame
 Row │ x
     │ Int64
─────┼───────
   1 │ 1
   2 │ 2
   3 │ 3

julia> transform!(df, :x => (r->(y = r.+1, z = r.^2)) => AsTable)
3×3 DataFrame
 Row │ x y z
     │ Int64 Int64 Int64
─────┼─────────────────────
   1 │ 1 2 1
   2 │ 2 3 4
   3 │ 3 4 9

```

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [July 11, 2022, 11:20am UTC](https://discourse.julialang.org/t/dataframes-transform/84041/7 "2022-07-11T11:20:09Z")

</div>

I tried to compare the various ways, but they all seem equivalent.  
I couldn’t figure out how insertcols performs, as @btime fails on the second pass because it already finds the columns with the same name.  
Has the option in the insertcols function been evaluated to overwrite an existing column?  
If so, why was it discarded?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 11, 2022, 11:26am UTC](https://discourse.julialang.org/t/dataframes-transform/84041/8 "2022-07-11T11:26:56Z")

</div>

> [@rocco\_sprmnt21](#):
>
> I couldn’t figure out how insertcols performs, as @btime fails on the second pass because it already finds the columns with the same name.

use initialization code for `@btime`.

> [@rocco\_sprmnt21](#):
>
> Has the option in the insertcols function been evaluated to overwrite an existing column?

This is the point of `insertcols!` that it should error in this case. If you want to overwrite an existing column use `setindex!` or `setproperty!` (i.e. just write `df.col = vector` or `df[!, col] = vector`).

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 11, 2022, 11:28am UTC](https://discourse.julialang.org/t/dataframes-transform/84041/9 "2022-07-11T11:28:40Z")

</div>

or pass `makeunique=true` in `insertcols!`.