API/traits for chunked out-of-core arrays?

Do we have some API/traits for out-of core arrays (e.g. on-disk arrays), especially if access to the array data is most efficient in a certain block/chunk size?

I remember there was a discussion here: Common interface for chunked arrays?. @fabiangans created a draft package https://github.com/meggart/ChunkedArrayBase.jl

1 Like

Thanks for the link!

In addition to chunking information, I guess it would be useful if we had a standard API to determine where data (in an array, or maybe just data in general) is located (CPU-RAM, disk, GPU-RAM, etc.) so user code can make informed decisions on when and how to convert (e.g. resulting in a GPU to CPU transfer, …).

Maybe that’s a nice addition to https://github.com/SciML/ArrayInterface.jl or https://github.com/JuliaGPU/Adapt.jl.

Oh, yes!

There’s also https://github.com/meggart/DiskArrays.jl, of course, in a more specific context.

CC @fabiangans

Yes I was about to mention DiskArrays as well. However, in the future it would be nice if the package’s functionality could be integrated in whatever Array trait system people will come up with in the future so that DiskArrays integrate more nicely with other special array types.