# Adding multiple new columns to dataframe?

**URL:** <https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997>\
**Category:** New to Julia\
**Tags:** question, dataframes\
**Created:** [November 30, 2020, 2:55pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997 "2020-11-30T14:55:21Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![LasseKatten](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lassekatten/32/8145_2.png) [@LasseKatten](https://discourse.julialang.org/u/LasseKatten)\
**Post date:** [November 30, 2020, 2:55pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/1 "2020-11-30T14:55:21Z")

</div>

Hi

Previously, one could do the following to add new columns to a dataframe:

```julia
df = Dataframe()
df[[:col1, :col2, :col3]] = []

```

But nowadays, this yields the following error message:

```julia
ERROR: MethodError: no method matching setindex!(::DataFrame, ::Array{Any,1}, ::Array{Symbol,1})

```

What is the new way to add new columns to a dataframe?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 30, 2020, 3:00pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/2 "2020-11-30T15:00:50Z")

</div>

Hmm I’m not sure that ever worked. What did you expend `df.col1` to return after you ran that code?

Do you want a `for` loop?

```julia
df = DataFrame()
for c in [:col1, :col2, :col3]
    df[:, c] = []
end

```

Note the `:`. In more recent version of DataFrames you need to specify _both_ dimensions when indexing a data frame. `df[:col]` and `df[[:col1, :col2]]` are both deprecated

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 30, 2020, 3:03pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/3 "2020-11-30T15:03:52Z")

</div>

The closest to the syntax you want is:

```julia
julia> df = DataFrame()
0×0 DataFrame

julia> insertcols!(df, ([:col1, :col2, :col3] .=> Ref([]))...)
0×3 DataFrame

julia> describe(df)
3×7 DataFrame
 Row │ variable mean min median max nmissing eltype
     │ Symbol Nothing Nothing Nothing Nothing Int64 DataType
─────┼──────────────────────────────────────────────────────────────────
   1 │ col1 0 Any
   2 │ col2 0 Any
   3 │ col3 0 Any

```

---

<div class="post-metadata">

**Author:** ![LasseKatten](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lassekatten/32/8145_2.png) [@LasseKatten](https://discourse.julialang.org/u/LasseKatten)\
**Post date:** [November 30, 2020, 3:12pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/4 "2020-11-30T15:12:16Z")

</div>

> [@pdeffebach](#):
>
> Hmm I’m not sure that ever worked.

I’m very certain it worked prior to the new pkg update.  
Right now, I also have to deal with changing from `df[:col]` and `df[[:col1, :col2]]` to `df[!, :col]` and `df[!, [:col1, :col2]]` in my code.

> [@pdeffebach](#):
>
> What did you expend `df.col1` to return after you ran that code?

I expect it to return an empty dataframe with the columns I specified:

```julia
0×3 DataFrame

```

I intend to populate that dataframe row by row in the next step, hence why I need an empty dataframe with specified columns.

> [@pdeffebach](#):
>
> Do you want a `for` loop?

For the record, your for-loop solution works, but I prefer @bkamins one liner.

Thank you!

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 30, 2020, 3:22pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/5 "2020-11-30T15:22:18Z")

</div>

> [@LasseKatten](#):
>
> I intend to populate that dataframe row by row in the next step, hence why I need an empty dataframe with specified columns.

1. You do not need to know the list of columns upfront as `push!` has `cols=:union` option that will handle this
2. If you know the list of columns upfront then it is easier to write:

```julia
DataFrame([:c1, :c2, :c3] .=> Ref([]))

```

or

```julia
DataFrame(fill([], 3), [:c1, :c2, :c3])

```

(in my original answer I thought you do not know the list of columns when creating a data frame)

---

<div class="post-metadata">

**Author:** ![LasseKatten](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lassekatten/32/8145_2.png) [@LasseKatten](https://discourse.julialang.org/u/LasseKatten)\
**Post date:** [November 30, 2020, 3:39pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/6 "2020-11-30T15:39:25Z")

</div>

Very useful suggestions!

1. Currently, I’m pushing a `1xn Array{Any, 2}` row-by-row, but I’m going to switch it to a `Dict` instead, which I think is much safer approach, now that I know `cols=:union` exists.

2. I actually have both cases, one where `DataFrame([:c1, :c2, :c3] .=> Ref([]))` is most useful and another where `insertcols!(df, ([:col1, :col2, :col3] .=> Ref([]))...)` is most suitable.

Thank you!

---

<div class="post-metadata">

**Author:** ![purplishrock](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/purplishrock/32/13451_2.png) [@purplishrock](https://discourse.julialang.org/u/purplishrock)\
**Post date:** [November 30, 2020, 3:40pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/7 "2020-11-30T15:40:21Z")

</div>

> [@bkamins](#):
>
> ```julia
> julia> insertcols!(df, ([:col1, :col2, :col3] .=> Ref([]))...)
> 
> ```

what does the ‘=\>’ operator do ? Is that DataFrames specific or a Julia operator ?  
p.s. is Ref() necessary to make sure that each new column doesn’t get the same empty list ?

---

<div class="post-metadata">

**Author:** ![LasseKatten](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lassekatten/32/8145_2.png) [@LasseKatten](https://discourse.julialang.org/u/LasseKatten)\
**Post date:** [November 30, 2020, 3:42pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/8 "2020-11-30T15:42:53Z")

</div>

This is a Julia operator used to create the data type `Pair`. Example:

```julia
julia> ([:a, :b, :c] .=> Ref([]))
3-element Array{Pair{Symbol,Array{Any,1}},1}:
 :a => []
 :b => []
 :c => []

```

I can confirm that you need the `Ref([])`, otherwise you get a `DimensionMismatch`:

```julia
julia> ([:a, :b, :c] .=> [])
ERROR: DimensionMismatch("arrays could not be broadcast to a common size; got a dimension with lengths 3 and 0")

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 30, 2020, 3:44pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/9 "2020-11-30T15:44:28Z")

</div>

> what does the ‘=\>’ operator do ? Is that DataFrames specific or a Julia operator ?

It is `Pair` syntax from Julia Base. Here you have a list of its usages in DataFrames.jl: [How is =\> used in DataFrames.jl? | Blog by Bogumił Kamiński](https://bkamins.github.io/julialang/2020/07/17/pair.html)

> is Ref() necessary to make sure that each new column doesn’t get the same empty list ?

`Ref` is necessary for the broadcasting to work. DataFrames.jl automatically takes care that the column `[]` is copied and not reused (you cound turn it off with `copycols=false` but in your case do not do this).

---

<div class="post-metadata">

**Author:** ![purplishrock](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/purplishrock/32/13451_2.png) [@purplishrock](https://discourse.julialang.org/u/purplishrock)\
**Post date:** [November 30, 2020, 3:44pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/10 "2020-11-30T15:44:46Z")

</div>

That’s very funny. I’m not sure I’ve ever used ‘=\>’. Is that possible ?! lol.

I do remember the `Ref([])` operation now, I had a similar problem where a broadcasted assignment didn’t work without.

Thank you.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 30, 2020, 3:45pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/11 "2020-11-30T15:45:45Z")

</div>

The original usage of `=>` is to create `Dict`s:

```julia
julia> Dict(1=>2, 3=>4)
Dict{Int64,Int64} with 2 entries:
  3 => 4
  1 => 2

```

---

<div class="post-metadata">

**Author:** ![purplishrock](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/purplishrock/32/13451_2.png) [@purplishrock](https://discourse.julialang.org/u/purplishrock)\
**Post date:** [November 30, 2020, 3:53pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/12 "2020-11-30T15:53:47Z")

</div>

> [@bkamins](#):
>
> It is `Pair` syntax from Julia Base. Here you have a list of its usages in DataFrames.jl

Great, thank you for taking the time to write that, it’s really useful (and thank you also for, you know, taking the time to write DataFrames !)

Also. shame on me. Julia’s new and improved help mode can tell me what ‘=\>’ is (the online documentation doesn’t work well for operators).

```julia
help?> =>
search: =>

  Pair(x, y)
  x => y

  Construct a Pair object with type Pair{typeof(x), typeof(y)}. The elements
  are stored in the fields first and second. They can also be accessed via
  iteration (but a Pair is treated as a single "scalar" for broadcasting
  operations).
...

```

---

<div class="post-metadata">

**Author:** ![LasseKatten](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lassekatten/32/8145_2.png) [@LasseKatten](https://discourse.julialang.org/u/LasseKatten)\
**Post date:** [November 30, 2020, 4:03pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/13 "2020-11-30T16:03:37Z")

</div>

Very nice blog btw. A lot of useful tips!

If you need any ideas for future blog post, may I suggest writing about how to optimize performance when using DataFrames?

For example, I noted in the previous pkg version that the difference in computation time performance between `filter(row -> row.col1 == x, df)` and `df[df[:col1] .== x, :]`

was very significant. For a particular `DataFrame`, I measured the average time over 1000 runs, and the results were:

- `filter(row -> row.col1 == x, df)` → 9.1429996 ms
- `df[df[:col1] .== x, :]` → 0.0309999 ms

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 30, 2020, 4:18pm UTC](https://discourse.julialang.org/t/adding-multiple-new-columns-to-dataframe/50997/14 "2020-11-30T16:18:01Z")

</div>

> [@LasseKatten](#):
>
> filter(row → row.col1 == x, df)

This performance difference is documented in `filter` docstring. The performant syntax is:

```julia
filter(:col1 => ==(x), df)

```