HDF5 high-level wrapper for nested Dicts

baptnz
Wed, 13 Mar 2024 19:52:25 +0000
General Usage

I’m storing some data in hdf5 format to use in a multilanguage environment (others will use Python, R, C, etc. to access those files). I’ve implemented the export using HDF5.jl without much trouble, but it’s not very well organised (compared to R or Matlab versions of the same code). I would like to store objects to be saved in Dictionaries, to make it tidier. Unfortunately, I am unable to get HDF5.jl to save dictionaries, and the function I tried to write does not save anything.

Here’s some dummy data, organised as I would like it, and my attempts:

using HDF5

# simple objects
a  = collect(400:50:800)
Nl = length(a)

m = reshape(repeat(collect(1:900.0), Nl), (30,30,Nl))

# now a dictionary, which should map to a group called "g"
g = Dict("shape" => "spheroid",
                "a" => 20.0,
                "c" => 40.0)

# another dictionary, which should map to a group called "e"
# this one is nested, should have 2 subgroups
e = Dict(
    "medium" => Dict("epsilon" => 1.0, "mu" => 1.0),
    "particle" => Dict("epsilon" => 1.0, "mu" => 1.0))

# ideally I'd group everything in a top-level Dict
# but that's optional
allthestuff = Dict("a" => a, "m" => m, "g" => g, "e" => e)

## ideal situation
# f = "debug.h5"
# fid = h5open(f, "w")
# save_dict_toh5(fid, allthestuff)
# close(fid)

## manual process (works, but messy)

f = "debug.h5"
fid = h5open(f, "w")

# assign simple datasets
fid["a"] = a
fid["m"] = m

# for flat Dicts, create a group, then iterate to assign datasets
group1 = create_group(fid, "g")
for (key, value) in g
    g[key] = value
end

# for nested Dicts, this becomes a mess
group2 = create_group(fid, "e")
for (key, value) in e
    # ridiculous way to handle nested group
    if(typeof(value) <: Dict)
        subgroup = create_group(fid, "e" * "/" * key)
        for (subkey, subvalue) in value
            subgroup[subkey] = subvalue
        end
    else
    group2[key] = value
    end
end

# also store some attributes
attributes(fid)["what's this"] = "some attributes"
attributes(fid["e"])["unit"] = "kg"

close(fid)

It works, but it’s verbose and inelegant. I’d like to write a high-level wrapper to make some steps more concise.

Here’s my attempt at writing a function to iterate over a (possibly-nested) Dict. It does nothing, unfortunately, which I assume is because it’s operating within the function’s environment instead of top level.

Many thanks.

function assign_dict(group, dict::Dict)

    for (key, value) in group
        if(typeof(value) <: Dict)
            assign_dict(group * "/" * key, value)
        else
        group[key] = value
        end
    end
    return group
end

# intended use within the above:
group2 = create_group(fid, "e")
assign_dict(group2, e)

Note: I’m aware of the JLD2 package, but it seems to me that the output would be too Julia-centric for my use case, as it seems to encode objects to ensure that I/O maps well to Julia (e.g. structs), but those extra attributes would be undesirable when read from another language.

4 posts - 2 participants