Proposal: working with larger than memory data in hdf5 format using HDF5Arrays (implementation of DiskArrays.jl for HDF5)

[musm]

@Luapulu

As for merging this into HDF5.jl, I’m a bit hesitant. To me HDF5.jl is an IO package. HDF5Arrays.jl will become an array implementation that uses some kind of HDF5 IO library but it’s not fundamentally part of an IO module. Imagine someone starts working on a pure julia implementation of a HDF5 IO library (like JLD2) and I wanted to use that new library in some parts. Would that make sense if HDF5Arrays.jl was part of HDF5.jl?

Many thanks for the details! I’ll have to follow and explore more fully all the threads. For now it does make sense to see how HDF5Arrays.jl pans out independently. If all goes to plan, in other words integration of ChainedArrays.jl and DiskArrays.jl with TiledArrays.jl(which I think could be be a good way to consolidate everything), it might end up that HDF5Array.jl would be quite a small package over TiledArrays, and that HDF5.jl would then just reexport the package. It does sounds like we might just end up needed to return an HDF5Array within HDF5.jl for Array return types to make handling them more ergonomic, since right now HDF5.jl doesn’t have a great AbstractArray interface and has many methods missing. Glad to see this being worked on!