# Reading and writing HDF5 compound-typed array datasets

**URL:** <https://discourse.julialang.org/t/reading-and-writing-hdf5-compound-typed-array-datasets/97516>\
**Category:** Data\
**Tags:** question, hdf5\
**Created:** [April 15, 2023, 5:35pm UTC](https://discourse.julialang.org/t/reading-and-writing-hdf5-compound-typed-array-datasets/97516 "2023-04-15T17:35:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![airpmb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/airpmb/32/7826_2.png) [@airpmb](https://discourse.julialang.org/u/airpmb)\
**Post date:** [April 15, 2023, 5:35pm UTC](https://discourse.julialang.org/t/reading-and-writing-hdf5-compound-typed-array-datasets/97516/1 "2023-04-15T17:35:49Z")

</div>

I wanted to write vectors of [LabelledArrays](https://docs.sciml.ai/LabelledArrays/stable/) to HDF5 files, preserving the labels, and be able to read these back and reconstruct the original data. My data is pretty much entirely numeric.

I’ve found it hard to find any relevant documentation; this is mentioned in an [issue (#819)](https://github.com/JuliaIO/HDF5.jl/issues/819#issuecomment-780075470) in HDF5.jl:

> We are certainly lacking examples on writing compound data types in the documentation.

Indeed - the only mention at all is the (read and write) support for `Complex`. But I found that you can write arrays of `NamedTuple` and they end up being HDF5 array datasets with compound datatype as you’d expect. There are some restrictions, e.g. the fields can’t be strings.

Conversely, you can read such a dataset, even with a string field, and you get out a vector of `NamedTuple` including string field.

So for example:

```julia
# File downloaded from https://www.neonscience.org/resources/learning-hub/tutorials/hdf5-intro-python
julia> fn = "/Users/patrick/Desktop/NEONDSTowerTemperatureData.hdf5";

julia> data = h5open(fn, "r") do h5f
          read(h5f, "Domain_03/OSBS/min_1/boom_1/temperature")
        end;

julia> typeof(data), size(data)
(Vector{NamedTuple{(:date, :numPts, :mean, :min, :max, :variance, :stdErr, :uncertainty), Tuple{String, Int32, Vararg{Float64, 6}}}}, (4323,))

```

But the reverse doesn’t work (unless the string-typed field, `date`, is removed):

```julia
julia> h5open("test.h5", "w") do h5f
          write_dataset(h5f, "test_dataset", data)
        end
ERROR: ArgumentError: Could not convert non-bitstype NamedTuple{(:date, :numPts, :mean, :min, :max, :variance, :stdErr, :uncertainty), Tuple{String, Int32, Vararg{Float64, 6}}} to NamedTuple{(:date, :numPts, :mean, :min, :max, :variance, :stdErr, :uncertainty), Tuple{HDF5.FixedString{1, 0}, Int32, Vararg{Float64, 6}}} for writing to HDF5. Consider implementing `convert(::Type{NamedTuple{(:date, :numPts, :mean, :min, :max, :variance, :stdErr, :uncertainty), Tuple{HDF5.FixedString{1, 0}, Int32, Vararg{Float64, 6}}}}, ::NamedTuple{(:date, :numPts, :mean, :min, :max, :variance, :stdErr, :uncertainty), Tuple{String, Int32, Vararg{Float64, 6}}})`

```

So for my use case (all the fields will be numeric types, all `Int` and `Float`) the functionality for read and write is there, but I’m hesitant to use it given that it’s not documented.

From what I can tell looking at the PRs, adding compound datatype reading and writing functionality was a deliberate thing, but what I’m not clear on is whether the lack of documentation is just an oversight, or perhaps it’s because the functionality isn’t meant for external usage (maybe because of the problem with writing strings?).

I’m hoping it’s the former and I can just start using this as-is (and contribute a documentation PR). Does anyone happen to know?

(Edit: cross-referenced this in a [comment](https://github.com/JuliaIO/HDF5.jl/issues/819#issuecomment-1509903768) to the above-mentioned issue).

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [April 15, 2023, 6:08pm UTC](https://discourse.julialang.org/t/reading-and-writing-hdf5-compound-typed-array-datasets/97516/2 "2023-04-15T18:08:30Z")

</div>

Maybe this is irrelevant, but do you know you can use JLD2.jl and it saves in HDF5 format? IMO the main reason to use HDF5.jl directly is for interoperability with other programming languages: JLD2 makes specific choices about storage format, and perhaps that’s not the format you’d prefer things be stored in. (HDF5 allows many different solutions to the same problem.)

---

<div class="post-metadata">

**Author:** ![airpmb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/airpmb/32/7826_2.png) [@airpmb](https://discourse.julialang.org/u/airpmb)\
**Post date:** [April 15, 2023, 6:45pm UTC](https://discourse.julialang.org/t/reading-and-writing-hdf5-compound-typed-array-datasets/97516/3 "2023-04-15T18:45:20Z")

</div>

Definitely relevant, because I wasn’t aware of that fact, thanks! I don’t think it’s the right tool for this particular job since we do want interoperability, but it looks like it could be useful in some contexts. Basically it seems to be a near-equivalent to pickling in Python (with similar caveats), is that about right?

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [April 16, 2023, 2:33am UTC](https://discourse.julialang.org/t/reading-and-writing-hdf5-compound-typed-array-datasets/97516/4 "2023-04-16T02:33:46Z")

</div>

Our last approach on this was

> <https://github.com/JuliaIO/HDF5.jl/pull/1013>
>
> This PR contains a utility function (EDIT: actually after some discussions it's …now realised via adding a method to \`datatype()\`) which I carry around my projects and I discovered that there is already an open issue which is related too, see \[#819\](https://github.com/JuliaIO/HDF5.jl/issues/819)
> 
> It's by all means not fully fledged, but it works for \`struct\`s with primitive field types very well. It's not reversible, meaning that \`read\` will yield the usual \`NamedTuple\`. One way to make it reversible would be to attach metadata to the dataset with e.g. the type name and when read back, check if the type is already defined in the namespace and if not, create it dynamically. Kind of what I do in https://github.com/JuliaHEP/UnROOT.jl where generating types during runtime is mandatory.
> 
> I provided an example in the function docstring which demonstrates that it's fairly intuitive to use and covers the basic needs. Here is it:
> 
> \`\`\`julia
> julia\> struct Foo
> x::Int32
> y::Float64
> end
> 
> julia\> foos = \[\[Foo(1, 2) Foo(3, 4) Foo(5, 6)\]; \[Foo(7, 8) Foo(9, 10) Foo(11, 12)\]\]
> 2×3 Matrix{Foo}:
> Foo(1, 2.0) Foo(3, 4.0) Foo(5, 6.0)
> Foo(7, 8.0) Foo(9, 10.0) Foo(11, 12.0)
> 
> julia\> h5open("foo.h5", "w") do h5f
> write\_dataset(h5f, "the/foo", foos)
> end
> 
> julia\> thefoo = h5open("foo.h5", "r") do file
> read(file, "the/foo")
> end
> 2×3 Matrix{NamedTuple{(:x, :y), Tuple{Int32, Float64}}}:
> (x = 1, y = 2.0) (x = 3, y = 4.0) (x = 5, y = 6.0)
> (x = 7, y = 8.0) (x = 9, y = 10.0) (x = 11, y = 12.0)
> \`\`\`
> 
> The data can be reinterpreted to the original data type using \`reinterpret()\`:
> 
> \`\`\`julia
> julia\> reinterpret(Foo, thefoo)
> 2×3 reinterpret(Foo, ::Matrix{NamedTuple{(:x, :y), Tuple{Int32, Float64}}}):
> Foo(1, 2.0) Foo(3, 4.0) Foo(5, 6.0)
> Foo(7, 8.0) Foo(9, 10.0) Foo(11, 12.0)
> \`\`\`

```julia
julia> using HDF5
                                  julia> nt = (x=5, y=6)
(x = 5, y = 6)

julia> h5open("test.h5", "w") do h
           write_dataset(h, "test", [nt])
       end

julia> h5open("test.h5", "r") do h
           h["test"][]
       end
1-element Vector{NamedTuple{(:x, :y), Tuple{Int64, Int64}}}:
 (x = 5, y = 6)

```

The of the main issues you seem to be running into is that your `NamedTuple` contains a `String`. This create some headaches since variable length strings are not great to work with in HDF5. Preferably one would use fixed length strings.

There is `HDF5.FixedString` but this has limited utility. For this reason I started to work on StaticStrings.jl as an expanded version of this. An alternative would be using InlineStrings.jl but this does not allow arbitrary string sizes.

> **[GitHub - mkitti/StaticStrings.jl: Fixed-length strings in Julia represented...](https://github.com/mkitti/StaticStrings.jl)**
>
> Fixed-length strings in Julia represented by NTuples - GitHub - mkitti/StaticStrings.jl: Fixed-length strings in Julia represented by NTuples

Regarding the “should you use this question”, the strongest affirmative support is that we are testing for some cases of `NamedTuple`

> <https://github.com/JuliaIO/HDF5.jl/blob/cff8206fac5d2560f882f9d628adeac199261bd5/test/compound.jl>

The honest truth here is there is a lot work to do on this package, and we could use all of hte help we could get. My attention is currently on the compilation side reviewing and assisting with this pull request:

> <https://github.com/JuliaPackaging/Yggdrasil/pull/6551>
>
> Build HDF5 by cross-compiling instead of by downloading Conda packages.

If you would like to document your use case, I would be happy to accept the pull request.