# Simulation Framework with Logging to Tabular Data

**URL:** <https://discourse.julialang.org/t/simulation-framework-with-logging-to-tabular-data/5205>\
**Category:** General Usage\
**Tags:** question\
**Created:** [August 3, 2017, 4:07pm UTC](https://discourse.julialang.org/t/simulation-framework-with-logging-to-tabular-data/5205 "2017-08-03T16:07:07Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![philip](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/philip/32/17191_2.png) [@philip](https://discourse.julialang.org/u/philip)\
**Post date:** [February 12, 2018, 5:59pm UTC](https://discourse.julialang.org/t/simulation-framework-with-logging-to-tabular-data/5205/21 "2018-02-12T17:59:29Z")

</div>

I think this [technique](https://discourse.julialang.org/t/get-fieldnames-and-values-of-struct-as-namedtuple/8991/3) is interesting. Perhaps if someone can edit the `to_named_tuple` piece this could work:

```julia

using DataFrames, NamedTuples

#gentup(struct_T) = NamedTuple{( fieldnames(struct_T)...,),
# Tuple{(fieldtype(struct_T,i) for i=1:fieldcount(struct_T))...}}
#
#@generated function to_named_tuple_generated(x)
# nt = Expr(:quote, gentup(x))
# tup = Expr(:tuple)
# for i=1:fieldcount(x)
# push!(tup.args, :(getfield(x, $i)) )
# end
# return :($nt($tup))
#end
#        
#        

#to_named_tuple_naive(p) = (; (v=>getfield(p, v) for v in fieldnames(typeof(p)))...)

        

mutable struct Person
    age::Int64
    id::Int64
    name::String
    weight::Float64
    blood_type::String
    average_resting_heart_rate::Int64
    IQ::Int64
end

function update!(person::Person)
    person.age += 1
    person.weight += rand() - 0.5
end

function logging_df(a_struct)
        DataFrame(prepend!(map(x -> fieldtype(a_struct, x), 
                               fieldnames(a_struct)
                              ),
                           [Int64]
                          ),
                  prepend!(fieldnames(a_struct),
                           [:timestep]
                          ),
                  0)
end
    
p = Person(18, 123, "foo", 72.5, "AB", 66, 100);
    
function log!(simresults, t, p)
        push!(simresults, prepend!(to_named_tuple(p),[t]))
end
    
all_people = [p]
    
function run_simulation!(timesteps::Int64, all_people, simresults)
    for t in 1:timesteps
        map(update!,all_people)
        log!(simresults,t,all_people)
    end
end

simresults = logging_df(Person)
run_simulation!(10,all_people,simresults)
simresults

```

---

<div class="post-metadata">

**Author:** ![ValdarT](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/valdart/32/24146_2.png) [@ValdarT](https://discourse.julialang.org/u/ValdarT)\
**Post date:** [February 12, 2018, 9:37pm UTC](https://discourse.julialang.org/t/simulation-framework-with-logging-to-tabular-data/5205/22 "2018-02-12T21:37:18Z")

</div>

> [@philip](#):
>
> At a high level, how is a Table different from a DataFrame?  
> Can you inject a column of timesteps in here?  
> Can you build the Table row-by-row, again so you don’t have to keep a Vector{Person} through thousands of timesteps and people, which just won’t fit in memory?

I’m definitely not the best person to talk about `Tabulars` but I think that the main difference from `DataFrames` is that it is unopinionated about the underlying format of your data. It tries to ingest whatever you throw at it and turn it into a lightweight `Table` type that provides a consistent and efficient interface to interact with your data. Ferris presents it nicely in his short [JuliaCon presentation](https://www.youtube.com/watch?v=1sl-q3iW57k).

Note that it is a work in progress. I don’t think there is an easy way to add columns or build it up row by row (yet).

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [February 12, 2018, 10:17pm UTC](https://discourse.julialang.org/t/simulation-framework-with-logging-to-tabular-data/5205/23 "2018-02-12T22:17:13Z")

</div>

I think the proper way would be to skip the named tuples and directly write a dataframe.

I think I would go for a `@generated push!(some_df, foostruct, fieldmap)`, where the fieldmap is a value-type of a tuple of pairs `(i,j)` where i is the number of the field in foostruct and j is the number of the corresponding column in df. Note that this is still slow because dataframes store the set of columns as `Vector{Any}`; one can maybe get rid of the type instability by a type-assert, so your extra cost for convenience is only a lookup of the type-tag (afaik there is no type-assume that does Vector{Any}[i]::type\_t without checking the type).

Also note that you would trigger an extra compilation for each dataframe you use (but this is probably cheaper than looking up the col-index in a dict for each field and each object you push).

---

<div class="post-metadata">

**Author:** ![philip](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/philip/32/17191_2.png) [@philip](https://discourse.julialang.org/u/philip)\
**Post date:** [February 12, 2018, 11:07pm UTC](https://discourse.julialang.org/t/simulation-framework-with-logging-to-tabular-data/5205/24 "2018-02-12T23:07:11Z")

</div>

Yeah–I don’t actually expect this to be performant at the moment. But I figure that if the

> [@RandomString123](#):
>
> object/relational mapping problem

can be solved generically at least with some slow proof of concept, then whenever someone finally gets fast `DataFrame`s and however one deals with cutting out the `NamedTuple` middle-man, some performant solution will appear on the horizon.

I’ll try to toy around with this more after I get 0.7 running.

---

<div class="post-metadata">

**Author:** ![RandomString123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/randomstring123/32/3194_2.png) [@RandomString123](https://discourse.julialang.org/u/RandomString123)\
**Post date:** [February 13, 2018, 1:36am UTC](https://discourse.julialang.org/t/simulation-framework-with-logging-to-tabular-data/5205/25 "2018-02-13T01:36:52Z")

</div>

If I get some time this week I might take a wack at writing a quick library with the method foobar posted in the other thread and some other convenience functions to go from a structured to a number of unstructured formats and back again.

---

<div class="post-metadata">

**Author:** ![philip](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/philip/32/17191_2.png) [@philip](https://discourse.julialang.org/u/philip)\
**Post date:** [June 19, 2018, 2:45am UTC](https://discourse.julialang.org/t/simulation-framework-with-logging-to-tabular-data/5205/26 "2018-06-19T02:45:19Z")

</div>

I never did get 0.7 running locally, so couldn’t try this out. But I have some more thoughts on this and on related questions. I expect to post these in the next week or so.

In the meantime, I wonder if anyone is familiar with any other work that has been done in this vein. In general I’m finding that I’m spending a lot of time thinking through high-level simulation issues when I just want to focus on designing and interpreting my actual simulations. I would have to think somebody wiser has already tackled the generic versions of these issues.

That is, the issue of logging is one high-level simulation issue. Another is result interpretation with consistent matching from simulation parameters into results. Generically, what is a way to keep track of the “match” so that whenever I “view” results (in the form of a statistic, DataFrame, plot, whatever) the parameters that generated that result are “carried along”–i.e. displayed along with it? In some sense what I want is a simple way to say "If I did the work of defining the function

SimulateOnce : Parameters \to Results,

where Parameters is literally _any_ space and Results is also literally _any_ space, and a class of functions

InterpretResults = \{ f : Results \to Interpretations \} ,

where Interpretations again is any space (e.g. if I’m simulating population growth, then this might be \mathbb N for the number of people alive at the end of the simulation, or it might be a visual image of a map of all the living entities in the 7th period. The point is that it’s _literally_ anything),

then I would like

1. a metafunction

2. Some “magical” function that interacts with the metafunction and with all the functions InterpretResults to nicely give me the set of meta-results

in some sort of neat and still easily interpretable but also _cross-comparable_ way, i.e., I can compare the results of the changing $p$s on the $r$s.

As a concrete example, in the population simulation case, if I’m able to generate a map of living creatures as an interpretative function of a simulation result, then I’d like to be able to quickly see a table with parametrizations on the top and the maps on the bottom of a 2xN grid of charts so I can interpret the effects of parameter changes on this feature of the simulation.

What I’m finding is that I spend more time writing these parameter-interpretation mappings than I do working on the simulation itself, and I find that annoying and wonder if there’s a better way. I’d also accept an informal proof / argument that “there’s no way around this because the generality of things one might want to ask is too high” or something along those lines.

(Does this make _any_ sense? If not, I’ll try to revise it when I’m more lucid.)

Thanks!

[Previous page](https://discourse.julialang.org/t/simulation-framework-with-logging-to-tabular-data/5205.md?page=1)