# ANN: MetaFields.jl and Flatten.jl - packages for writing composable models
**URL:** https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849
**Category:** Modelling & Simulations
**Created:** [August 3, 2018, 7:27am UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849 "2018-08-03T07:27:06Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Raf](https://discourse.julialang.org/u/Raf)
#### Post date: [August 3, 2018, 7:27am UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849/1 "2018-08-03T07:27:06Z")
I’ve been putting together a few biophysical models over the last year, and have constantly encountered what I feel to be the core software problem in my field: sharing high performance code in general ways _never_ happens. This means there is no real software ecosystem, a huge amount of cut and paste, and massive monolithic architectures. Its a social problem with a technical cause.
One solution to this is composition of model parameters into nested heriarchies, using generic types, and method dispatch to run custom functions depending on the types. Julia is amazing for this, and I have a proof of concept: [DynamicEnergyBudgets.jl](https://github.com/rafaqz/DynamicEnergyBudgets.jl) and [Photosynthesis.jl](https://github.com/rafaqz/Photosynthesis.jl).
## Why Flatten and MetaFields?
The problem with using composition is that numerical tools need parameters to be provided in flat vectors, not nested in hierarchies. Bayesian methods need not only the parameters to be flat, but also priors for each parameter!
I want all of that to compose so running multiple model combinations can be automated, without writing any boilerplate conversion code. I also want to supply defaults for everything, but allow easy user overrides.
The solution I have ended up with is a way of flattening heterogeneous structs and metadata about them intto vectors:
> **[GitHub - rafaqz/FieldMetadata.jl: Metadata for julia fields](https://github.com/rafaqz/FieldMetadata.jl)**
>
> Metadata for julia fields
> **[GitHub - rafaqz/Flatten.jl: Flatten nested Julia objects to tuples, and...](https://github.com/rafaqz/Flatten.jl)**
>
> Flatten nested Julia objects to tuples, and reconstruct them later
MetaFields lets you attach meta data to fields, just like tags in go but more flexible. It lets you choose which fields to flatten, and attach extra data like priors to each field, that are not actually included on the struct.
Flatten.jl will flatten nested type hierarchies to vectors or tuples, and rebuild them, as an intermediary between a nested model and say, DiffEqSensitivity, to insert Dual numbers into the nested types. Credit goes to Robin Deits and Jan Weidner for a lot of the ideas, the original code minus MetaFields integrations was written by Deits.
The two key features of flatten is that you can skip fields in the hierarchy using a @flattenable metafield, and you can flatten metafields and other properties such as field name, or parent struct name into vectors.
Any comments or improvements would be really helpful before I put these into METADATA, I’m also going to discuss some of the more general ideas in a paper and conference in the next few months so feedback would be really appreciated.
As an this example, I’ll take a hierarchical, nested data type with metafield markup ([example source](https://github.com/rafaqz/Photosynthesis.jl/blob/master/src/parameters.jl#L81)), and flatten chosen fields and metadata about them to a dataframe, with intermixed parameters and metafields from multiple packages and custom overrides:
```julia-auto
julia> organism = DynamicEnergyBudgets.FvCBPlant();
julia> params = organism;
julia> def = Flatten.flatten(Vector, params);
julia> pri = metaflatten(Vector, params, DynamicEnergyBudgets.prior);
julia> fnames = metaflatten(Vector, params, fieldname_meta);
julia> unit = metaflatten(Vector, params, DynamicEnergyBudgets.units);
julia> form = metaflatten(Vector, params, fieldparent_meta);
julia> lab = metaflatten(Vector, params, DynamicEnergyBudgets.label);
jjulia> t = DataFrame(Formulation=form, Name=fnames, Default=def, Units=unit, Prior=pri, Label=lab)
82×6 DataFrames.DataFrame. Omitted printing of 1 columns
│ Row │ Formulation │ Name │ Default │ Units │ Prior │
├─────┼──────────────────────────────┼─────────────┼──────────┼────────────────┼─────────────────────────────────────────────────┤
│ 1 │ YingPingRadiationConductance │ rdfipt │ 1.0 │ │ │
│ 2 │ YingPingRadiationConductance │ tuipt │ 1.0 │ │ │
│ 3 │ YingPingRadiationConductance │ tdipt │ 1.0 │ │ │
│ 4 │ BoundaryConductance │ leafwidth │ 0.05 │ m │ Distributions.Gamma{Float64}(α=2.0, θ=0.025) │
│ 5 │ BallBerryStomatalConductance │ gamma │ 0.0 │ μmol mol^-1 │ Distributions.Gamma{Float64}(α=1.0, θ=2.0) │
│ 6 │ BallBerryStomatalConductance │ g1 │ 7.0 │ │ Distributions.Gamma{Float64}(α=10.0, θ=0.7) │
│ 7 │ PotentialSoilData │ swpexp │ 1.0 │ │ │
│ 8 │ BallBerryModel │ g0 │ 0.03 │ mol m^-2 s^-1 │ Distributions.Gamma{Float64}(α=10.0, θ=0.003) │
│ 9 │ Jmax │ jmax25 │ 184.0 │ μmol m^-2 s^-1 │ Distributions.Gamma{Float64}(α=100.0, θ=1.84) │
│ 10 │ Jmax │ delsj │ 640.02 │ J K^-1 mol^-1 │ Distributions.Gamma{Float64}(α=100.0, θ=6.4002) │
│ 11 │ Jmax │ eavj │ 37259.0 │ J mol^-1 │ Distributions.Gamma{Float64}(α=100.0, θ=372.59) │
│ 12 │ Jmax │ edvj │ 200000.0 │ J mol^-1 │ Distributions.Gamma{Float64}(α=100.0, θ=2000.0) │
│ 13 │ NoOptimumVcmax │ vcmax25 │ 110.0 │ μmol m^-2 s^-1 │ Distributions.Gamma{Float64}(α=100.0, θ=1.14) │
│ 14 │ NoOptimumVcmax │ eavc │ 47590.0 │ J mol^-1 │ Distributions.Gamma{Float64}(α=100.0, θ=475.9) │
│ 15 │ BernacchiCompensation │ Kc25 │ 404.9 │ μmol mol^-1 │ Distributions.Gamma{Float64}(α=10.0, θ=40.49) │
│ 16 │ BernacchiCompensation │ Ko25 │ 278400.0 │ μmol mol^-1 │ Distributions.Gamma{Float64}(α=10.0, θ=27840.0) │
│ 17 │ BernacchiCompensation │ Γ☆25 │ 42.75 │ μmol mol^-1 │ Distributions.Gamma{Float64}(α=10.0, θ=4.275) │
│ 18 │ BernacchiCompensation │ ΔHa_Kc │ 79.43 │ kJ mol^-1 │ Distributions.Gamma{Float64}(α=10.0, θ=7.943) │
│ 19 │ BernacchiCompensation │ ΔHa_Ko │ 36.38 │ kJ mol^-1 │ Distributions.Gamma{Float64}(α=10.0, θ=3.638) │
│ 20 │ BernacchiCompensation │ ΔHa_Γ☆ │ 37.83 │ kJ mol^-1 │ Distributions.Gamma{Float64}(α=10.0, θ=3.783) │
│ 21 │ BernacchiCompensation │ tref │ 25.0 │ °C │ │
│ 22 │ RubiscoRegen │ theta │ 0.4 │ │ Distributions.Beta{Float64}(α=8.0, β=12.0) │
│ 23 │ RubiscoRegen │ ajq │ 0.324 │ │ Distributions.Beta{Float64}(α=4.0, β=8.3) │
│ 24 │ Respiration │ q10f │ 0.67 │ K^-1 │ │
│ 25 │ Respiration │ dayresp │ 1.0 │ │ Distributions.Beta{Float64}(α=5.0, β=1.0) │
│ 26 │ Respiration │ rd0 │ 0.9 │ μmol m^-2 s^-1 │ Distributions.Gamma{Float64}(α=10.0, θ=0.09) │
│ 27 │ Respiration │ tbelow │ -100.0 │ °C │ │
⋮
│ 55 │ KooijmanArea │ M_Vscaling │ 400.0 │ mol │ Distributions.Gamma{Float64}(α=2.0, θ=0.2) │
│ 56 │ SqrtAllometry │ size │ 0.1 │ m │ Distributions.Gamma{Float64}(α=2.0, θ=0.2) │
│ 57 │ Maturity │ j_E_rep_mai │ 0.001 │ dy^-1 │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 58 │ Maturity │ κrep │ 0.05 │ │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 59 │ Maturity │ M_Vrep │ 10.0 │ mol │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 60 │ Maturity │ w_M │ 25.0 │ g mol^-1 │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 61 │ Maturity │ n_N_M │ 10.0 │ │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 62 │ Translocation │ proportions │ 1.0 │ │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 63 │ Params │ M_Vgerm │ 0.01 │ mol │ Distributions.Gamma{Float64}(α=2.0, θ=2.0) │
│ 64 │ Params │ κsoma │ 0.6 │ │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 65 │ Params │ y_P_V │ 0.02 │ │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 66 │ Params │ y_V_E │ 0.7 │ │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 67 │ Params │ y_E_ET │ 0.8 │ │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 68 │ Params │ y_EC_ECT │ 1.0 │ │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 69 │ Params │ y_EN_ENT │ 1.0 │ │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 70 │ Params │ j_E_mai │ 0.001 │ dy^-1 │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 71 │ Params │ j_P_mai │ 0.01 │ dy^-1 │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 72 │ Params │ k_E │ 0.2 │ dy^-1 │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 73 │ Params │ k_EC │ 0.2 │ dy^-1 │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 74 │ Params │ k_EN │ 0.2 │ dy^-1 │ Distributions.Beta{Float64}(α=2.0, β=2.0) │
│ 75 │ SharedParams │ n_N_N │ 10.0 │ │ Distributions.Gamma{Float64}(α=2.0, θ=2.0) │
│ 76 │ SharedParams │ w_P │ 25.0 │ g mol^-1 │ Distributions.Gamma{Float64}(α=2.0, θ=2.0) │
│ 77 │ SharedParams │ w_V │ 25.0 │ g mol^-1 │ Distributions.Gamma{Float64}(α=2.0, θ=2.0) │
│ 78 │ SharedParams │ w_C │ 25.0 │ g mol^-1 │ Distributions.Gamma{Float64}(α=2.0, θ=2.0) │
│ 79 │ SharedParams │ w_N │ 25.0 │ g mol^-1 │ Distributions.Gamma{Float64}(α=2.0, θ=2.0) │
│ 80 │ SharedParams │ w_E │ 25.0 │ g mol^-1 │ Distributions.Gamma{Float64}(α=2.0, θ=2.0) │
│ 81 │ SharedParams │ y_E_CH_NO │ 1.5 │ │ Distributions.Gamma{Float64}(α=2.0, θ=2.0) │
│ 82 │ SharedParams │ y_E_EN │ 1.5 │ │ Distributions.Gamma{Float64}(α=2.0, θ=2.0) │
```
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [August 3, 2018, 1:43pm UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849/2 "2018-08-03T13:43:35Z")
> [@Raf](#):
>
> Flatten.jl will flatten nested type hierarchies to vectors or tuples, and rebuild them, as an intermediary between a nested model and say, DiffEqSensitivity, to insert Dual numbers into the nested types. Credit goes to Robin Deits and Jan Weidner for a lot of the ideas, the original code minus MetaFields integrations was written by Deits.
>
> The two key features of flatten is that you can skip fields in the hierarchy using a @flattenable metafield, and you can flatten metafields and other properties such as field name, or parent struct name into vectors.
That’s pretty cool. I may be building off of that to get some easily broadcastable structures.
---
### Author:  [@piever](https://discourse.julialang.org/u/piever)
#### Post date: [August 3, 2018, 2:06pm UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849/3 "2018-08-03T14:06:12Z")
It looks very interesting!
It does seem like there’s a bit of overlap across different projects:
- The `Columns` type from IndexedTables can turn vectors of tuples / named tuples / pairs (potentially nested) into a tuple or named tuple of arrays
- StructofArrays can turn an array of structs into a struct of arrays (also works with nesting) but seems a bit unmaintained (though I may be wrong) and am not sure if it works on Julia master
At some point I had created a small StructArrays package trying to extend StructofArrays in a way that it could be used to replace the job of `Columns` in IndexedTables (and to take advantage of dot overloading, so that columns can be accessed with `x.colname`).
I was wondering whether Flatten and StructArrays can be joined (so that Flatten takes care of the flattening and StructArrays of creating a columnar storage with fast row iteration).
---
### Author:  [@Raf](https://discourse.julialang.org/u/Raf)
#### Post date: [August 3, 2018, 4:04pm UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849/4 "2018-08-03T16:04:57Z")
Great let me know if you have questions or need any changes! a large part of the motivation was integrating with the DiffEq ecosystem and parametrised functions.
@piever m not sure how IndexedTables or StructOfArrays work so can’t really offer any ideas there… This iteration of Flatten is mostly interesting when you don’t want to flatten all of the fields on the nested struct, or when you also have field metadata to flatten, but I’m not sure what the scope is really!
On the other hand I’ve been happily flattening nested models to build InteractBulma interfaces all day 🙂
---
### Author:  [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)
#### Post date: [August 3, 2018, 10:10pm UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849/5 "2018-08-03T22:10:56Z")
I have also put some basic low level machinery together for here:
> **[GitHub - simonbyrne/PackVec.jl: Pack/unpack Julia objects into vectors](https://github.com/simonbyrne/PackVec.jl)**
>
> Pack/unpack Julia objects into vectors. Contribute to simonbyrne/PackVec.jl development by creating an account on GitHub.
---
### Author:  [@Raf](https://discourse.julialang.org/u/Raf)
#### Post date: [August 4, 2018, 2:48am UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849/6 "2018-08-04T02:48:49Z")
Cool! Seems like this is a common requirement. Handling arrays could be useful too. Is packVec type-stable?
---
### Author:  [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)
#### Post date: [August 4, 2018, 2:18pm UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849/7 "2018-08-04T14:18:13Z")
> [@Raf](#):
>
> Is packVec type-stable?
It should be.
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [August 4, 2018, 2:27pm UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849/8 "2018-08-04T14:27:48Z")
> [@simonbyrne](#):
>
> It should be.
It’s not with units. When things are all Float64 it is though.
---
### Author:  [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)
#### Post date: [August 4, 2018, 2:34pm UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849/9 "2018-08-04T14:34:06Z")
Ah: I can add that. Please open an issue.
---
### Author:  [@Raf](https://discourse.julialang.org/u/Raf)
#### Post date: [August 4, 2018, 2:43pm UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849/10 "2018-08-04T14:43:42Z")
Because Forward diff etc don’t accept units I’ve opted for flattening to their internal float values, and reconstructing from float to units again. It makes type stability a lot easier and possible with vectors. Having units in a metafield means you can just flatten the units separately and reapply to the vector if you actually need them…
---
### Author:  [@Raf](https://discourse.julialang.org/u/Raf)
#### Post date: [August 7, 2018, 10:14am UTC](https://discourse.julialang.org/t/ann-metafields-jl-and-flatten-jl-packages-for-writing-composable-models/12849/11 "2018-08-07T10:14:35Z")
In case anyone is interested I’ve generalised the internals of the flattening/reconstruction process in [https://github.com/rafaqz/Nested.jl](https://github.com/rafaqz/Nested.jl)
Which should make it relatively easy to build these kind of high performance nested @generated functions.