Save Array of Arrays in HDF5

Continuing the discussion from here and here, how can I best save a large number of arrays using HDF5? And more generally, how can I save an array of arrays?

I know that HDF5 doesn’t support array of arrays, so trying the below doesn’t work:

julia> using HDF5

julia> x = [[1]]
1-element Array{Array{Int64,1},1}:

julia> fid = h5open("test.h5", "w")
HDF5 data file: test.h5

julia> g   = g_create(fid, "array-of-arrays")
HDF5 group: /array-of-arrays (file: test.h5)

julia> write(g, x)
ERROR: MethodError: no method matching write(::HDF5Group, ::Array{Array{Int64,1},1})

In my particular case I have a single array containing 10,000 arrays of varying lengths. I’d like the 10,000 arrays to be part of a “group”, but creating new datasets / groups for each array makes reading the file very slow, so I am seeking an alternative.


What did you try, and how did it fail? Please provide an MWE.