Do we have some API/traits for out-of core arrays (e.g. on-disk arrays), especially if access to the array data is most efficient in a certain block/chunk size?

Thanks for the link! In addition to chunking information, I guess it would be useful if we had a standard API to determine where data (in an array, or maybe just data in general) is located (CPU-RAM, disk, GPU-RAM, etc.) so user code can make informed decisions on when and how to convert (e.g. resu…

Maybe that’s a nice addition to GitHub - JuliaArrays/ArrayInterface.jl: Designs for new Base array interface primitives, used widely through scientific machine learning (SciML) and other organizations or GitHub - JuliaGPU/Adapt.jl .

There’s also https://github.com/meggart/DiskArrays.jl , of course, in a more specific context. CC @fabiangans

Yes I was about to mention DiskArrays as well. However, in the future it would be nice if the package’s functionality could be integrated in whatever Array trait system people will come up with in the future so that DiskArrays integrate more nicely with other special array types.

API/traits for chunked out-of-core arrays?

General Usage

tkf April 1, 2020, 9:36pm 2

I remember there was a discussion here: Common interface for chunked arrays?. @fabiangans created a draft package GitHub - meggart/ChunkedArrayBase.jl

Topic		Replies	Views
Common interface for chunked arrays? Data hdf5 , parallel , data	10	2026	August 26, 2019
[ANN] DiskArrays.jl Package Announcements	3	1696	November 2, 2020
Taking the array indexing interface seriously Internals & Design question	31	3201	February 3, 2020
Proposal: working with larger than memory data in hdf5 format using HDF5Arrays (implementation of DiskArrays.jl for HDF5) Data hdf5	11	1887	November 4, 2020
Convention/package for chunking array stream? General Usage array	4	627	January 12, 2019

API/traits for chunked out-of-core arrays?

Related topics