# Best way to iteratively add to a DataFrame?

**URL:** <https://discourse.julialang.org/t/best-way-to-iteratively-add-to-a-dataframe/21084>\
**Category:** General Usage\
**Tags:** dataframes\
**Created:** [February 22, 2019, 3:56pm UTC](https://discourse.julialang.org/t/best-way-to-iteratively-add-to-a-dataframe/21084 "2019-02-22T15:56:08Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![torgo](https://avatars.discourse-cdn.com/v4/letter/t/94ad74/32.png) [@torgo](https://discourse.julialang.org/u/torgo)\
**Post date:** [February 22, 2019, 3:56pm UTC](https://discourse.julialang.org/t/best-way-to-iteratively-add-to-a-dataframe/21084/1 "2019-02-22T15:56:08Z")

</div>

I would like to do something like this:

```julia
using DataFrames
dflong = DataFrame()
for i = 1:3
    df = DataFrame(a = rand(i))
    vcat(dflong, df)
end

```

I understand that this doesn’t work for two reasons:

1. `dflong` cannot be modified inside the local `for` scope
2. Even if it could, `dflong` and `df` have a different number of columns.

I have devised a solution that works, but seems very ugly, inelegant, and perhaps inefficient:

```julia
using DataFrames

dflong = DataFrame()
first = true

for i = 1:3
    df = DataFrame(a = rand(i))

    global dflong
    global first

    if first
        dflong = similar(df, 0)
        first = false
    else
        dflong = vcat(dflong, df)
    end
end

```

Can you suggest a better way to do this?  
I am new to Julia so probably just not getting something basic here about the proper way to adapt to for loops with local scope.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [February 22, 2019, 4:39pm UTC](https://discourse.julialang.org/t/best-way-to-iteratively-add-to-a-dataframe/21084/2 "2019-02-22T16:39:43Z")

</div>

```julia
> reduce(vcat, [DataFrame(a = rand(i)) for i in 1:5])
> 15×1 DataFrame
│ Row │ a │
│ │ Float64 │
├─────┼───────────┤
│ 1 │ 0.0250787 │
│ 2 │ 0.144394 │
│ 3 │ 0.216657 │
│ 4 │ 0.761747 │
│ 5 │ 0.351675 │
│ 6 │ 0.284681 │
│ 7 │ 0.106181 │
│ 8 │ 0.551472 │
│ 9 │ 0.523894 │
│ 10 │ 0.51445 │
│ 11 │ 0.587754 │
│ 12 │ 0.878151 │
│ 13 │ 0.985698 │
│ 14 │ 0.504822 │
│ 15 │ 0.788035 │

```

---

<div class="post-metadata">

**Author:** ![torgo](https://avatars.discourse-cdn.com/v4/letter/t/94ad74/32.png) [@torgo](https://discourse.julialang.org/u/torgo)\
**Post date:** [February 22, 2019, 4:50pm UTC](https://discourse.julialang.org/t/best-way-to-iteratively-add-to-a-dataframe/21084/3 "2019-02-22T16:50:34Z")

</div>

Thanks this is helpful.  
In practice (outside of my simple example) I would like to do many operations inside the `for` loop before concatenating the data frame, so that I can’t use a constructor.  
What’s a good solution for those types of situations?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [February 22, 2019, 4:52pm UTC](https://discourse.julialang.org/t/best-way-to-iteratively-add-to-a-dataframe/21084/4 "2019-02-22T16:52:21Z")

</div>

The solution is correct, but I have some minor additional notes.

`reduce(vcat, [DataFrame(a = rand(i)) for i in 1:5])`

is only minimally faster than

`vcat([DataFrame(a = rand(i)) for i in 1:5]...)`

(the change was merged yesterday to master and has not been released yet (earlier splatting was the recommended approach).

Also creating intermediate data frames is not efficient. The recommended way to add rows to a data frame is:

```julia
using DataFrames
dflong = DataFrame(a=Float64[])
for i = 1:3
    push!(dflong, (rand(i),))
end

```

(you can read the documentation of `push!` to find the accepted types of rows, in particular you can `push!` a `NamedTuple`, a dictionary, a vector or a tuple)

If you really have to create intermediate `DataFrame`s then you can also do it with `append!` which will also be relatively fast (and you do not have to store all the data frames in the memory before `vcat`-ing):

```julia
using DataFrames
dflong = DataFrame(a=Float64[])
for i = 1:3
    append!(dflong, DataFrame(a=rand(i)))
end

```

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [February 22, 2019, 7:20pm UTC](https://discourse.julialang.org/t/best-way-to-iteratively-add-to-a-dataframe/21084/5 "2019-02-22T19:20:55Z")

</div>

This is a situation where allowing `push!` and `append!` to add new columns if the data frame has zero columns would be convenient. Not sure whether that justifies this exception.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [February 23, 2019, 12:44am UTC](https://discourse.julialang.org/t/best-way-to-iteratively-add-to-a-dataframe/21084/6 "2019-02-23T00:44:53Z")

</div>

You should also be able to `vcat` a DataFrame with a `Dict` provided the symbols are the same as the DataFrame’s columns. Since a `Dict` is lighter weight (I think) this might be a solution depending on the details of your problem.

---

<div class="post-metadata">

**Author:** ![bjarthur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bjarthur/32/9638_2.png) [@bjarthur](https://discourse.julialang.org/u/bjarthur)\
**Post date:** [February 23, 2019, 2:22am UTC](https://discourse.julialang.org/t/best-way-to-iteratively-add-to-a-dataframe/21084/7 "2019-02-23T02:22:00Z")

</div>

`push!`ing onto [TypedTables](https://github.com/FugroRoames/TypedTables.jl) is possible with an [issue](https://github.com/FugroRoames/TypedTables.jl/issues/33) i created.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [February 23, 2019, 6:04am UTC](https://discourse.julialang.org/t/best-way-to-iteratively-add-to-a-dataframe/21084/8 "2019-02-23T06:04:04Z")

</div>

> [@nalimilan](#):
>
> Not sure whether that justifies this exception.

`append!` should be OK, but `push!` is problematic, because:

1. if what we push is a vector/tuple we do not have column names
2. if what we push is a dict/named tuple the current behavior of `push!` is to add only a selection of columns that already exist in a `DataFrame`, so we would add no columns.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [February 23, 2019, 10:03am UTC](https://discourse.julialang.org/t/best-way-to-iteratively-add-to-a-dataframe/21084/9 "2019-02-23T10:03:38Z")

</div>

Yeah, that would only work when pushing a named tuple or `DataFrameRow`…