# Efficiently creating a data frame that is made up of smaller data frames
**URL:** https://discourse.julialang.org/t/efficiently-creating-a-data-frame-that-is-made-up-of-smaller-data-frames/87055
**Category:** Modelling & Simulations
**Tags:** dataframes, for-loop
**Created:** [September 10, 2022, 10:54pm UTC](https://discourse.julialang.org/t/efficiently-creating-a-data-frame-that-is-made-up-of-smaller-data-frames/87055 "2022-09-10T22:54:13Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@erickawaguchi](https://discourse.julialang.org/u/erickawaguchi)
#### Post date: [September 10, 2022, 10:54pm UTC](https://discourse.julialang.org/t/efficiently-creating-a-data-frame-that-is-made-up-of-smaller-data-frames/87055/1 "2022-09-10T22:54:13Z")
Hi all,
I’m trying to create a “long” version data frame (e.g. several observations per individual as in a longitudinal study). However, the data frame for each individual will depend on certain parameters - not the same operation for all N individuals; some will have more rows, some fewer. I was able to do it by creating a blank data frame and then continually updating it within a for loop using vcat.
In terms of pseudocode, this is how it currently looks:
```julia
df = DataFrame(ID = Int64[], nᵢ = Int64[], ...) # Define the actual data frame I want
for i in 1:N
dftmp = DataFrame(ID = i, nᵢ = Int64[], ...) # Create a temporary data frame (same args as df)
.
.
.
df = vcat(df, dftmp) # Combine dftmp with df (i.e. Update df)
end
```
However, I feel like there must be a more efficient way of doing this and wanted guidance. I come from an R background and have manipulated data frames using tidyverse.
Thanks in advance.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [September 11, 2022, 3:30am UTC](https://discourse.julialang.org/t/efficiently-creating-a-data-frame-that-is-made-up-of-smaller-data-frames/87055/2 "2022-09-11T03:30:56Z")
What you do is going to be slow.
Instead use almost the same (essentially: create a vector of data frames and then `vcat` them in one shot next):
```julia
reduce(vcat, [DataFrame(ID = i , nᵢ = ...) for i in 1:N])
```
If your source data frames have different sets of columns please read `vcat` documentation about options to handle this case.
---
### Author:  [@erickawaguchi](https://discourse.julialang.org/u/erickawaguchi)
#### Post date: [September 11, 2022, 6:40am UTC](https://discourse.julialang.org/t/efficiently-creating-a-data-frame-that-is-made-up-of-smaller-data-frames/87055/3 "2022-09-11T06:40:52Z")
Thank you for the response.
However, I am not sure how the command below works:
```julia
[DataFrame(ID = i], , nᵢ = ...) for i in 1:N])
```
In addition, the source data frames will have the same rows but different sets of columns.
Pseudocode:
```julia
for i in 1:N
dftmp = DataFrame(ID = i, nᵢ = Int64[], ...) # Create a temporary data frame (same args as df)
.
if (arg is true)
dftmp = ...
else
dftmp = ...
end
df = vcat(df, dftmp) # Combine dftmp with df (i.e. Update df)
end
```
Is there a better way to do the inside `ifelse` loop?
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [September 11, 2022, 9:47am UTC](https://discourse.julialang.org/t/efficiently-creating-a-data-frame-that-is-made-up-of-smaller-data-frames/87055/5 "2022-09-11T09:47:15Z")
> [@SteffenPL](#):
>
> Probably it was just a small typo and it should be
OP used `...` as a placeholder for some other operations so I re-used it. The code was not runnable clearly.
> the source data frames will have the same rows but different sets of columns.
As I have commented above please read `vcat` documentation. I am copying part of the documentation that is relevant:
> The cols keyword argument determines the columns of the returned data frame:
> • :setequal: require all data frames to have the same column names disregarding order. If they
> appear in different orders, the order of the first provided data frame is used.
> • :orderequal: require all data frames to have the same column names and in the same order.
> • :intersect: only the columns present in all provided data frames are kept. If the
> intersection is empty, an empty data frame is returned.
> • :union: columns present in at least one of the provided data frames are kept. Columns not
> present in some data frames are filled with missing where necessary.
> • A vector of Symbols or strings: only listed columns are kept. Columns not present in some
> data frames are filled with missing where necessary.
Now regarding your question:
> Pseudocode:
Please share full code if you want a full working code in response.
`[DataFrame(ID = i], , nᵢ = ...) for i in 1:N]` is a comprehension.
Alternatively you can define e.g. a function taking one argument (i):
```julia
function gen_df(i)
...
end
```
and then use broadcasting like this:
```julia
reduce(vcat, gen_df.(1:N), cols=:union) # I use :union as I understand you want union of the columns
```
---
### Author:  [@SteffenPL](https://discourse.julialang.org/u/SteffenPL)
#### Post date: [September 11, 2022, 9:52am UTC](https://discourse.julialang.org/t/efficiently-creating-a-data-frame-that-is-made-up-of-smaller-data-frames/87055/6 "2022-09-11T09:52:21Z")
Just a quick note, I was referring to the part " `ID = i],` " which seems to be a `],` too much. (Anyway, not really relevant ;))
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [September 11, 2022, 10:25am UTC](https://discourse.julialang.org/t/efficiently-creating-a-data-frame-that-is-made-up-of-smaller-data-frames/87055/7 "2022-09-11T10:25:43Z")
Ah - OK. Fixed