Continuing the discussion from here and here, how can I best save a large number of arrays using HDF5? And more generally, how can I save an array of arrays?
I know that HDF5 doesn’t support array of arrays, so trying the below doesn’t work:
julia> using HDF5
julia> x = [[1]]
1-element Array{Array{Int64,1},1}:
[1]
julia> fid = h5open("test.h5", "w")
HDF5 data file: test.h5
julia> g = g_create(fid, "array-of-arrays")
HDF5 group: /array-of-arrays (file: test.h5)
julia> write(g, x)
ERROR: MethodError: no method matching write(::HDF5Group, ::Array{Array{Int64,1},1})
In my particular case I have a single array containing 10,000 arrays of varying lengths. I’d like the 10,000 arrays to be part of a “group”, but creating new datasets / groups for each array makes reading the file very slow, so I am seeking an alternative.
Thanks!