# How to best preallocate nested tuples of dataframes to store simulation results?
**URL:** https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543
**Category:** New to Julia
**Tags:** tuple, dataframes, data\_structures
**Created:** [December 13, 2023, 2:09am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543 "2023-12-13T02:09:12Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)
#### Post date: [December 13, 2023, 2:09am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/1 "2023-12-13T02:09:12Z")
I’m trying to run some sport game simulations and programmatically store the results in the most efficient manner. Even if not strictly necessary for my purpose, I am trying to learn how to do right.
Right now my order of concern here is: `Memory > Readability > Speed`. I came up with this, which is roughly how I would start out in R using a nested list:
```julia
###########################
#### Desired Structure ####
###########################
# -sim1
# -metrics = {repeat(Float32, 3), Bool}(1x4)
# -lgstats
# -rep1 = {Int16, String15, repeat(Int32, 8)}(4x10)
# -rep2 = {Int16, String15, repeat(Int32, 8)}(4x10)
# -[...]
# -team1
# -pinfo = {String15, Int8, String3, String3}(30x4)
# -params = {Int8}(30x6)
# -pstats
# -rep1 = {Int16}(30x7)
# -rep2 = {Int16}(30x7)
# -[...]
# -team2
# -[...]
# -team3
# -[...]
# -team4
# -[...]
# -sim2
# -[...]
# -sim3
# -[...]
```
Each simulation (`sim`) uses a unique set of player parameters/ratings (`params`), and is replicated (`rep`) multiple times per sim step (the output is stochastic).
You can see there are various `metrics` (rmse, etc) and league-level stats (`lgstats`). Then for each `team` there is player info (`pinfo`) like name/etc, along with the player `params` and stats (`pstats`).
After much messing around, I managed to create this:
```julia
using DataFrames, InlineStrings
# Variable
nsim = 3
nrep = 2
nteam = 4
# Constant (see desired structure)
nmetrics = 4
nplayer = 30
nlgstats = 10
nparams = 6
npstats = 7
# Team-level Tuple
teamres = Tuple{DataFrame, # pinfo
DataFrame, # params
NTuple{nrep, DataFrame} # pstats, rep[1:nrep]
} # team1
# Simulation-level Tuple
simres = Tuple{DataFrame, # nmetrics
NTuple{nrep, DataFrame}, # lgstats, rep[1:nrep]
NTuple{nteam, teamres} # teams, team[1:nteam]
} # sim1
# Final Result
allres = NTuple{nsim, simres}
```
When run, it seems to work:
> **Summary**
>
> ```julia
> julia> # Team-level Tuple
> teamres = Tuple{DataFrame, # pinfo
> DataFrame, # params
> NTuple{nrep, DataFrame} # pstats, rep[1:nrep]
> } # team1
> Tuple{DataFrame, DataFrame, Tuple{DataFrame, DataFrame}}
>
>
> julia> # Simulation-level Tuple
> simres = Tuple{DataFrame, # nmetrics
> NTuple{nrep, DataFrame}, # lgstats, rep[1:nrep]
> NTuple{nteam, teamres} # teams, team[1:nteam]
> } # sim1
> Tuple{DataFrame, Tuple{DataFrame, DataFrame}, NTuple{4, Tuple{DataFrame, DataFrame, Tuple{DataFrame, DataFrame}}}}
>
>
> julia> # Final Result
> allres = NTuple{nsim, simres}
> Tuple{Tuple{DataFrame, Tuple{DataFrame, DataFrame}, NTuple{4, Tuple{DataFrame, DataFrame, Tuple{DataFrame, DataFrame}}}}, Tuple{DataFrame, Tuple{DataFrame, DataFrame}, NTuple{4, Tuple{DataFrame, DataFrame, Tuple{DataFrame, DataFrame}}}}, Tuple{DataFrame, Tuple{DataFrame, DataFrame}, NTuple{4, Tuple{DataFrame, DataFrame, Tuple{DataFrame, DataFrame}}}}}
>
> ```
But I cannot figure out how to:
1. Name the elements of these tuples (I could not get NamedTuple{} to work here).
2. Preallocate dataframes of the desired sizes and types.
3. Programmatically add my data to this DataType I created.
And perhaps this is the totally wrong way to go about it. Please tell me if so, because I don’t know what I’m doing here. But even then, I would be interested in knowing how to make this method work (or why it won’t).
---
### Author:  [@juliohm](https://discourse.julialang.org/u/juliohm)
#### Post date: [December 13, 2023, 3:13am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/2 "2023-12-13T03:13:14Z")
I would try to adapt to the DrWatson.jl workflow, they sorted out all these details about saving simulation results, parameters, etc:
[https://juliadynamics.github.io/DrWatson.jl/dev](https://juliadynamics.github.io/DrWatson.jl/dev)
---
### Author:  [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)
#### Post date: [December 13, 2023, 3:22am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/3 "2023-12-13T03:22:51Z")
Thanks, I am currently trying out Dr. Watson for this project and plan to do exactly that. It seems very well thought out.
I guess I should clarify that these _are_ the simulation results I will save. Each of these simulation steps takes \<1s so saving after each step would generate an insane number of files and/or be a bottleneck.
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [December 13, 2023, 3:37am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/4 "2023-12-13T03:37:20Z")
If you want high speed you could have a look at [Arrow.jl](https://arrow.apache.org/julia/stable/) and [StructArrays.jl](https://github.com/JuliaArrays/StructArrays.jl)… I use them together for logging of flight data 20 times per second…
They use memory mapped arrays… Much faster than DataFrames, but less convenient to use…
Well, if speed is not your major concern you could use Arrow also together with DataFrames.jl .
Arrow stores the data in binary form on disk (much more compact than CSV) and in the same form in memory, so no conversion needed when loading or saving…
---
### Author:  [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)
#### Post date: [December 13, 2023, 5:07am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/5 "2023-12-13T05:07:47Z")
Both look very interesting, thanks. I’ll have to play around with them. But this feels like getting ahead of myself.
I guess you are saying try a structarray of nested namedtuples containing dataframes instead of what I did? I am not manipulating this data yet, only generating it and storing it in memory. Will I still benefit from this and Arrow.jl?
The thing is I couldn’t even figure out how to do normal nested namedtuples of dataframes. I think you are assuming I know way more Julia than I do… My problem is much more basic.
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [December 13, 2023, 5:12am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/6 "2023-12-13T05:12:21Z")
Why would you want to use nested tuples of DataFrames? A DataFrame is like a database. Normally you have only one of them where you store everything. For nested structures [HDF5](https://juliaio.github.io/HDF5.jl/stable/) might be a better choice.
---
### Author:  [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)
#### Post date: [December 13, 2023, 5:19am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/7 "2023-12-13T05:19:14Z")
My data has a hierarchical structure with multiple types. So tuples of DataFrames fit the bill for an easily understandable structure to me (if I could get them named).
To be clear, I have no need to read/write anything from/to disk efficiently. I am just trying to organize my data in an understandable way and keep it in memory while not wasting ram.
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [December 13, 2023, 5:30am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/8 "2023-12-13T05:30:15Z")
> [@Tetrakai](#):
>
> I guess you are saying try a structarray of nested namedtuples containing dataframes
No. Arrow would allow you to store one 2D table, which could be one DataFrame. You can use StructArrays instead of DataFrames.
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [December 13, 2023, 5:34am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/9 "2023-12-13T05:34:26Z")
I think you cannot pre-allocate the data structure you currently have in mind. You can pre-allocate arrays or tables.
---
### Author:  [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)
#### Post date: [December 13, 2023, 5:39am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/10 "2023-12-13T05:39:56Z")
> Arrow would allow you to store one 2D table, which could be one DataFrame.
It would have to be multiple tables, and (I think) would require way more effort to get out what I want later. The way I wrote it, this would be very easy. However, I would want to try that if it used way less memory or was much faster.
> I think you cannot pre-allocate the data structure you currently have in mind. You can pre-allocate arrays or tables.
Possibly, I have no idea. I was able to make _some_ kind of DataType that looks like what I want though. Is there a reason why Julia doesn’t allow this?
---
### Author:  [@Emmanuel-R8](https://discourse.julialang.org/u/Emmanuel-R8)
#### Post date: [December 13, 2023, 5:40am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/11 "2023-12-13T05:40:11Z")
Tuples are immutable. You can preallocate the first initialisation, but after that you will need brand new allocations.
For preallocations, I use closures but there might be better alternatives.
---
### Author:  [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)
#### Post date: [December 13, 2023, 5:43am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/12 "2023-12-13T05:43:38Z")
That was my understanding. I don’t plan on modifying this data later. But I can still iteratively fill in the tuples right? Just not change the values later.
---
### Author:  [@Emmanuel-R8](https://discourse.julialang.org/u/Emmanuel-R8)
#### Post date: [December 13, 2023, 5:48am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/13 "2023-12-13T05:48:08Z")
Of the top of my head, no certain. But I think not.
Why not use a normal struct?
---
### Author:  [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)
#### Post date: [December 13, 2023, 5:50am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/14 "2023-12-13T05:50:12Z")
> [@Emmanuel-R8](#):
>
> Why not use a normal struct?
Because I have no idea what I am doing. That is my question. How would I create a struct to hold data organized in the way I laid out above, if that is possible?
**Edit:**
Alternatively, why wouldn’t I want to create a struct that looks like that?
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [December 13, 2023, 7:41am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/15 "2023-12-13T07:41:20Z")
> [@Tetrakai](#):
>
> But I can still iteratively fill in the tuples right?
No, you cannot. The content of a tuple has to be known when it is created. That is the meaning of “immutable”. There might be work-arounds, but I doubt that they work for DataFrames…
Tuples are intended to be used with small, basic data types, not with large structures like DataFrames…
> Tuples in Julia are an immutable collection of distinct values of same or different datatypes separated by commas. Tuples are more like arrays in Julia except that arrays only take values of similar datatypes. The values of a tuple can not be changed because tuples are immutable. Tuples are a heterogeneous collection of values.
---
### Author:  [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)
#### Post date: [December 13, 2023, 2:26pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/16 "2023-12-13T14:26:45Z")
Yea, after playing with it more I saw I would need to have the parent type be a StructArray or something. Then the simres tuple could be filled all at once at the end of each step. I think.
Anyway from the responses it is clear I am doing something odd here. But I also find it hard to believe I am the first person to use julia who wants to store data in memory organized something like this.
I mean it is such a standard, logical thing to do in R you’ll find R 101 examples of it all over. Eg: [https://statisticsglobe.com/create-nested-list-in-r](https://statisticsglobe.com/create-nested-list-in-r)
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [December 13, 2023, 2:32pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/17 "2023-12-13T14:32:06Z")
Here’s what that first example could look like in Julia:
```julia
julia> list_1 = [12:20, ('a':'z')[16:-1:11], ["yyyy"]]
3-element Vector{AbstractVector}:
12:20
'p':-1:'k'
["yyyy"]
julia> list_2 = [4:8, ('a':'z')[7:-1:1], ["xxx"]]
3-element Vector{AbstractVector}:
4:8
'g':-1:'a'
["xxx"]
julia> list_3 = [["Another"], ["list (that's actually a vector of vectors)"], ["in Julia"]]
3-element Vector{Vector{String}}:
["Another"]
["list (that's actually a vector of vectors)"]
["in Julia"]
julia> my_nested_list_1 = [list_1, list_2, list_3]
3-element Vector{Vector{AbstractVector}}:
[12:20, 'p':-1:'k', ["yyyy"]]
[4:8, 'g':-1:'a', ["xxx"]]
[["Another"], ["list (that's actually a vector of vectors)"], ["in Julia"]]
julia> my_nested_list_1[2][2]
'g':-1:'a'
```
---
### Author:  [@Nathan\_Boyer](https://discourse.julialang.org/u/Nathan_Boyer)
#### Post date: [December 13, 2023, 2:49pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/18 "2023-12-13T14:49:38Z")
I think you should compose [structs](https://docs.julialang.org/en/v1/manual/types/#Composite-Types) of simpler types. I’m not following exactly your objective, but something like below:
```julia
struct Sim
metrics::Vector{Metric}
lgstats::Vector{Rep}
teams::Vector{Team}
end
struct Team
pinfo::PInfo
params::Params
pstats::Vector{Rep}
end
struct PInfo
a::String
b::Int64
c::String
d::String
end
```
Your collections should be `Vector` if you need to mutate (update values or add to the collection). Otherwise `Tuple` collections will be faster.
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [December 13, 2023, 3:14pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/19 "2023-12-13T15:14:02Z")
> [@Nathan\_Boyer](#):
>
> Your collections should be `Vector` if you need to mutate (update values or add to the collection).
If you have scalar fields that you want to mutate, then use a `mutable struct`. Just add the keyword `mutable` in front of your struct.
---
### Author:  [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)
#### Post date: [December 13, 2023, 4:16pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/20 "2023-12-13T16:16:04Z")
Thanks, you seem to have understood.
So when I tried playing with structs, I couldn’t figure out how to avoid writing out every name and type. Eg,
```julia
namevec = [a, b, c, d]
typevec = [String; Int64; repeat([String], 2)]
# Is there a function like this?
PInfo = create_struct(namevec, typevec)
```
And I’m guessing if I want to preallocate for eg 4 teams then I would use:
```julia
struct Sim
metrics::Vector{Metric}
lgstats::Vector{Rep}
teams::Vector{Team}(undef, 4)
end
```
I will play around with it but this seems to be what I want.
[Next page](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543.md?page=2)