API/traits for chunked out-of-core arrays?

General Usage
1

Do we have some API/traits for out-of core arrays (e.g. on-disk arrays), especially if access to the array data is most efficient in a certain block/chunk size?

2

I remember there was a discussion here: Common interface for chunked arrays?. @fabiangans created a draft package GitHub - meggart/ChunkedArrayBase.jl

1 Like
3

Thanks for the link!

In addition to chunking information, I guess it would be useful if we had a standard API to determine where data (in an array, or maybe just data in general) is located (CPU-RAM, disk, GPU-RAM, etc.) so user code can make informed decisions on when and how to convert (e.g. resulting in a GPU to CPU transfer, …).

4

Maybe that’s a nice addition to GitHub - JuliaArrays/ArrayInterface.jl: Designs for new Base array interface primitives, used widely through scientific machine learning (SciML) and other organizations or GitHub - JuliaGPU/Adapt.jl.

5

Oh, yes!

6

There’s also https://github.com/meggart/DiskArrays.jl, of course, in a more specific context.

CC @fabiangans

7

Yes I was about to mention DiskArrays as well. However, in the future it would be nice if the package’s functionality could be integrated in whatever Array trait system people will come up with in the future so that DiskArrays integrate more nicely with other special array types.