# How to best preallocate nested tuples of dataframes to store simulation results?

**URL:** <https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543>\
**Category:** New to Julia\
**Tags:** tuple, dataframes, data\_structures\
**Created:** [December 13, 2023, 2:09am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543 "2023-12-13T02:09:12Z")\
**Posts on this page:** 12\
**Page:** 2

<div class="post-metadata">

**Author:** ![Nathan\_Boyer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nathan_boyer/32/14825_2.png) [@Nathan\_Boyer](https://discourse.julialang.org/u/Nathan_Boyer)\
**Post date:** [December 13, 2023, 4:51pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/21 "2023-12-13T16:51:43Z")

</div>

> [@Tetrakai](#):
>
> `# Is there a function like this?`

I doubt it, but maybe there are macros for this. It seems unlikely, since a dynamic type like you propose would not allow the type to be compiled ahead of time for dispatch.

In general, you need to define the types (`struct`s) explicitly first. Then create variables as instances of those types.

```julia
julia> myinfo = PInfo("Hello", 2, "John", "Snow")
PInfo("Hello", 2, "John", "Snow")

julia> typeof(myinfo)
PInfo

```

> [@Tetrakai](#):
>
> And I’m guessing if I want to preallocate for eg 4 teams then I would use:
> 
> ```julia
> struct Sim
> metrics::Vector{Metric}
> lgstats::Vector{Rep}
> teams::Vector{Team}(undef, 4)
> end
> 
> ```

No, again the allocation will occur once the type is invoked not in the definition of the type.

Define a simple type `Test`:

```julia
struct Test
    v::Vector{Int64}
end

```

It is possible to create an `undef` instance as you propose, but it is verbose and the data will be junk (not necessarily zeros):

```julia-repl
julia> t = Test(Vector{Int64}(undef, 4))
Test([0, 0, 0, 0])

```

I would probably just:

```julia-repl
julia> t = Test(zeros(4))
Test([0, 0, 0, 0])

```

Then, either way, you can start filling the values.

```julia-repl
julia> t.v[3] = 5
5

julia> t
Test([0, 0, 5, 0])

```

---

<div class="post-metadata">

**Author:** ![Tetrakai](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)\
**Post date:** [December 13, 2023, 5:07pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/22 "2023-12-13T17:07:43Z")

</div>

> [@Nathan\_Boyer](#):
>
> It seems unlikely, since a dynamic type like you propose would not allow the type to be compiled ahead of time for dispatch.

Well in principle the function could just create a string, right?

```julia
PInfo_str = create_struct_string(structname, namevec, typevec)
"Pinfo = struct PInfo
           a::String
           b::Int64
           c::String
           d::String
         end"

```

Then write it (without quotes) to structs.jl, which then gets included. I’d guess I should also look into the eval function, but this seems like it would be a common annoyance. So once again, if it doesn’t already exist, I am probably trying to misuse structs in some way.

---

<div class="post-metadata">

**Author:** ![Tetrakai](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)\
**Post date:** [December 13, 2023, 6:07pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/23 "2023-12-13T18:07:05Z")

</div>

I mean I just made this one using some copypasting and regex in sublime text:

```julia
struct lgtble
    Pl ::Int16
    Team::String15 
    P ::Int32 
    W ::Int32 
    D ::Int32 
    L ::Int32 
    GF ::Int32 
    GA ::Int32 
    GD ::Int32 
    Pts ::Int32
end

```

Julia users don’t see the need for a more convenient way than fiddling around in the editor or writing `::Int32` over and over? That makes me think I’m trying to misuse structs somehow.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [December 13, 2023, 6:16pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/24 "2023-12-13T18:16:17Z")

</div>

Well, if you use the package [GitHub - mauro3/Parameters.jl: Types with default field values, keyword constructors and (un-)pack macros](https://github.com/mauro3/Parameters.jl) you can write:

```julia
@with_kw mutable struct Lg_table @deftype Int32
    Pl ::Int16
    Team::String15 
    P
    W
    D
    L
    GF
    GA
    GD 
    Pts
end

```

But please use upper case names for types and lower case names for variables… And names with a lenght of one character are most of the time not a good idea, if you look at your code in a year you don’t remember what they mean…

---

<div class="post-metadata">

**Author:** ![Nathan\_Boyer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nathan_boyer/32/14825_2.png) [@Nathan\_Boyer](https://discourse.julialang.org/u/Nathan_Boyer)\
**Post date:** [December 13, 2023, 7:38pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/25 "2023-12-13T19:38:24Z")

</div>

There are [parametric types](https://docs.julialang.org/en/v1/manual/types/#man-parametric-composite-types).

```julia
struct lgtble{T}
    Pl ::Int16
    Team::String15 
    P ::T
    W ::T 
    D ::T 
    L ::T 
    GF ::T 
    GA ::T 
    GD ::T 
    Pts ::T
end

```

`NamedTuple`s are also a good choice if you want something you can define quickly. I would still use a more formal `struct` for your outer groupings like `Sim`, but some of your inner collections like this `lgtble` are probably more convenient as a `NamedTuple`.

```julia-repl
julia> nt = (Pl=Int16(3), Team="Eagles", Pts=10)
(Pl = 3, Team = "Eagles", Pts = 10)

julia> typeof(nt)
@NamedTuple{Pl::Int16, Team::String, Pts::Int64}

```

You can dispatch a function on a named tuple by defining `f(x::NamedTuple)`, but that doesn’t give you the same level of flexibility down the line as defining your own custom types up front like `f(x::Sim)`, `f(x::Team)`, etc. I have seen on this forum that often people prototype with `NamedTuple`s and then formalize those into composite types (`struct`s) once they settle on their design. (This is largely because Revise.jl does not work on structs though.)

* * *

I’m sure there are fancier ways to define structs with macros, `eval`, and automatic code generation, but mine were never complicated enough to reach for those things.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [December 13, 2023, 8:24pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/26 "2023-12-13T20:24:20Z")

</div>

> [@Nathan\_Boyer](#):
>
> `NamedTuple`s are also a good choice if you want something you can define quickly

But not if you want to pre-allocate memory… Then, if you want to save a data item once per second, why do you want to pre-allocate it… Normally pre-allocation only matters for real-time systems…

---

<div class="post-metadata">

**Author:** ![Nathan\_Boyer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nathan_boyer/32/14825_2.png) [@Nathan\_Boyer](https://discourse.julialang.org/u/Nathan_Boyer)\
**Post date:** [December 13, 2023, 8:38pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/27 "2023-12-13T20:38:35Z")

</div>

So there are two separate questions:

1. What is the most logical way to organize my data?
2. What can I do to make simulations on this data fast?

Start with #1 and organize in whatever way makes the most logical sense. Julia has a lot of flexibility here both with its native types and with the larger package ecosystem that has more specialized types. If everything is a `Tuple` or `DataFrame` as you have it now, then it is hard to tell what level of nesting you are in and what the different data frames all are. As a start, I suggest:

- If your data is a table, put it in a `DataFrame`.
- If your data is a set of parameters, put them in a `NamedTuple`  
(or `Dict` if you need mutable).
- If your data is a collection of the same element type, put them in a `Vector`.
- If your data is a collection of mismatched types, put them in a custom `struct`.

Then once you have a runnable example, you can ask #2. It is not too hard to swap types for performance once you have something in place.

* * *

One of my custom types for inspiration:

```julia
struct LocalFailureResults
    result::String
    maximum_damage::String
    table::DataFrame
    material_data::OrderedDict
end

```

```julia-repl
julia> r = local_failure()
Material Inputs
Select the .xlsx file containing temperature-dependent data for the chosen material.
Selected Material: SA-723-3-2
Enter the average temperature in the material during operation in degrees Fahrenheit: (Default: 400) 

Local Failure Inputs
Select the .xls file containing the local failure principal stress vectors for the entire model.
Select the .xls file containing the local failure equivalent plastic strain for the entire model.

Local Failure Result Fields: result, maximum_damage, table, material_data

julia> r.result
"PASS"

julia> r.maximum_damage
"2.5%"

julia> r.table
36126×8 DataFrame
   Row │ Node Number Maximum Principal Stress (psi) Middle Prin ⋯
       │ Int64 Float64 Float64 ⋯
───────┼───────────────────────────────────────────────────────────
     1 │ 78415 73795.0 ⋯
     2 │ 190016 74811.0
     3 │ 138631 43446.0
   ⋮ │ ⋮ ⋮ ⋱
 36124 │ 159857 16592.0
 36125 │ 241883 16644.0 ⋯
 36126 │ 241889 14510.0
                                   6 columns and 36080 rows omitted

julia> r.material_data
OrderedCollections.OrderedDict{String, Any} with 20 entries:
  "Material" => "SA-723-3-2"
  "Minimum Hardness (BHN)" => missing
  "Yield Strength at Temperature (ksi)" => 110.3
  "Tensile Strength at Temperature (ksi)" => 135.0
  "R" => 0.817037
  "ϵₚ" => 2.0e-5

```

---

<div class="post-metadata">

**Author:** ![Tetrakai](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)\
**Post date:** [December 13, 2023, 9:19pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/28 "2023-12-13T21:19:13Z")

</div>

There is no saving, just storing the results so I can inspect them at the end of the simulation. Maybe this will clarify:

Right now I am doing:

```julia
allres = Vector{Vector{DataFrame}}(undef, nsim)
for i in 1:nsim
   # set new parameter values
   simres = Vector{NamedTuple}(undef, nrep)
   for j in 1:nrep
       # Run nrep simulations that generate tuples of dataframes called lgtble, etc
       simres[j] = (lgtble, ...)
   end
    # calculate metrics and store in dataframe
    allres[i] = [simres, metrics]
end

```

Maybe that is just the best way to do it?

But would like to have a more organized (named and nested) structure within that `Vector{Vector{DataFrame}}` instead of a vector of dataframes. And I figured if I could tell it the types of the dataframe columns beforehand this would be more efficient.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [December 14, 2023, 3:32am UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/30 "2023-12-14T03:32:26Z")

</div>

> [@Tetrakai](#):
>
> Maybe that is just the best way to do it?

You should understand that you are NOT pre-allocating the DataFrames. You can only pre-allocate variables with a fixed size. What is stored in this type are only pointers to the DataFrames. the DataFrames themselves get their memory from the heap when you create them with concrete data. In other words, you cannot pre-allocate DataFrames.

---

<div class="post-metadata">

**Author:** ![Tetrakai](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)\
**Post date:** [December 14, 2023, 10:14pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/31 "2023-12-14T22:14:20Z")

</div>

So after realizing I failed to even ask a good question, I started trying to make a MWE for this.

But it ended up as more an intermediate working example (IWE) of ~150 lines of utility/helper functions and about 50 lines in the main loop.

Basically my first julia was about 5 weeks ago and I have seen similar questions on here that elicited similar responses (“I’m not sure what you are trying to do”), so it may be helpful to others to have a more complex example of the julian way to store the results of these kinds of nested simulations.

I got hesitant to post it since it ended up so long, but if anyone is interesting/willing to critique it let me know. I would want to know _everything_ I am doing wrong.

---

<div class="post-metadata">

**Author:** ![rdavis120](https://avatars.discourse-cdn.com/v4/letter/r/b5a626/32.png) [@rdavis120](https://discourse.julialang.org/u/rdavis120)\
**Post date:** [December 14, 2023, 10:48pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/32 "2023-12-14T22:48:48Z")

</div>

If you want to stick to your own construction - rather than something like DrWatson or mlflow - you might be better off starting by storing the data frames in an embedded database like sqlite or duckdb, and using a primary key to join the tables by the id of the simulations. It would push the storage out to disk and you indicated that reducing memory was a concern.

---

<div class="post-metadata">

**Author:** ![Tetrakai](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@Tetrakai](https://discourse.julialang.org/u/Tetrakai)\
**Post date:** [December 14, 2023, 10:56pm UTC](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543/33 "2023-12-14T22:56:13Z")

</div>

Thanks, but once again your response is due to a bad presentation of the question, apparently. What I am concerned with would all be contained within this:

```julia
MLflow.start_run()

```

> **[MLflow Julia Integration Guide](https://www.restack.io/docs/mlflow-knowledge-mlflow-julia-integration)**
>
> Explore how to integrate MLflow with Julia for efficient machine learning workflows and experiment tracking.

So unfortunately that doesn’t look like what I am looking for.

[Previous page](https://discourse.julialang.org/t/how-to-best-preallocate-nested-tuples-of-dataframes-to-store-simulation-results/107543.md?page=1)