# Serializing nested Dicts (or DataFrames) so that they can (easily) be loaded in Python as well?

**URL:** <https://discourse.julialang.org/t/serializing-nested-dicts-or-dataframes-so-that-they-can-easily-be-loaded-in-python-as-well/82809>\
**Category:** General Usage\
**Tags:** hdf5, dictionary, python, jld2, dataframes\
**Created:** [June 15, 2022, 12:45pm UTC](https://discourse.julialang.org/t/serializing-nested-dicts-or-dataframes-so-that-they-can-easily-be-loaded-in-python-as-well/82809 "2022-06-15T12:45:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![daharn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daharn/32/27088_2.png) [@daharn](https://discourse.julialang.org/u/daharn)\
**Post date:** [June 15, 2022, 12:45pm UTC](https://discourse.julialang.org/t/serializing-nested-dicts-or-dataframes-so-that-they-can-easily-be-loaded-in-python-as-well/82809/1 "2022-06-15T12:45:07Z")

</div>

From my simulations, I typically end up with (somewhat convoluted) Dicts and/or DataFrames made from these dicts that contain my input parameters as well as my results. Depending on the complexity of the results, they might live in their own Dict, so a nested Dict.

I can serialize these objects with `JLD2` without an issue and reload them into Julia. However, while I like Julia, quickly fiddling around with and visualizing data often feels a bit less cumbersome in Python. I would love the option to just load these serialized results there as well.

While loading jld2 files via `h5py` principally works, the result is pretty messy in terms of how the data is structured. I assume this is because HDF5 can only save uniform arrays and the jld2 needs to find its way around this.

For the same reason, directly saving dicts in HDF5 is not possible. Are there any other cross-language formats that might be viable or are there tool that might help me in converting my dicts to something that makes more sense in HDF5?

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [June 15, 2022, 12:53pm UTC](https://discourse.julialang.org/t/serializing-nested-dicts-or-dataframes-so-that-they-can-easily-be-loaded-in-python-as-well/82809/2 "2022-06-15T12:53:51Z")

</div>

Hi @daharn,

as you note yourself, this is not possible straight-away.  
JLD2 can not know whether to store your nested dicts in a HDF5 compatible way (losing type information) or not.

Here’s a workaround:

```julia
julia> using JLD2

julia> data = Dict("a" => 1,
       "b/c" => 2,
       "b/d" => 3,
       "b/e/f" => 4)
Dict{String, Int64} with 4 entries:
  "b/d" => 3
  "b/e/f" => 4
  "b/c" => 2
  "a" => 1

julia> save("test.jld2", data)

julia> f = jldopen("test.jld2")
JLDFile /home/isensee/test.jld2 (read-only)
 ├─🔢 a
 └─📂 b
    ├─🔢 d
    ├─🔢 c
    └─📂 e
       └─🔢 f

```

You should probably be able to write a simple function to “unroll” your nested data in this way.  
At top-level, the `FileIO.save` API accepts a `String`-key Dictionary.

---

<div class="post-metadata">

**Author:** ![IlyaOrson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ilyaorson/32/1681_2.png) [@IlyaOrson](https://discourse.julialang.org/u/IlyaOrson)\
**Post date:** [June 15, 2022, 1:13pm UTC](https://discourse.julialang.org/t/serializing-nested-dicts-or-dataframes-so-that-they-can-easily-be-loaded-in-python-as-well/82809/3 "2022-06-15T13:13:21Z")

</div>

For your purpose it could be easier to use the json format as an intermediary to take advantage of the available table writers/readers in both languages.  
DataFrame + Dict → JSONTables.jl → JSON → Pandas

---

<div class="post-metadata">

**Author:** ![daharn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daharn/32/27088_2.png) [@daharn](https://discourse.julialang.org/u/daharn)\
**Post date:** [June 15, 2022, 10:21pm UTC](https://discourse.julialang.org/t/serializing-nested-dicts-or-dataframes-so-that-they-can-easily-be-loaded-in-python-as-well/82809/4 "2022-06-15T22:21:06Z")

</div>

Indeed, going via JSON works absolutely flawless for both dataframes and dicts! Any binary format (BSON, JLD2 or HDF5) rendered the reloaded data irredeemably convoluted in python but JSON preserved the whole structure.

Is there a reason why it is not included in the `FileIO.jl` framework per default?