# API/traits for chunked out-of-core arrays?

**URL:** <https://discourse.julialang.org/t/api-traits-for-chunked-out-of-core-arrays/36873>\
**Category:** General Usage\
**Created:** [April 1, 2020, 8:59pm UTC](https://discourse.julialang.org/t/api-traits-for-chunked-out-of-core-arrays/36873 "2020-04-01T20:59:52Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [April 1, 2020, 8:59pm UTC](https://discourse.julialang.org/t/api-traits-for-chunked-out-of-core-arrays/36873/1 "2020-04-01T20:59:52Z")

</div>

Do we have some API/traits for out-of core arrays (e.g. on-disk arrays), especially if access to the array data is most efficient in a certain block/chunk size?

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [April 1, 2020, 9:36pm UTC](https://discourse.julialang.org/t/api-traits-for-chunked-out-of-core-arrays/36873/2 "2020-04-01T21:36:56Z")

</div>

I remember there was a discussion here: [Common interface for chunked arrays?](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009). @fabiangans created a draft package [GitHub - meggart/ChunkedArrayBase.jl](https://github.com/meggart/ChunkedArrayBase.jl)

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [April 1, 2020, 10:40pm UTC](https://discourse.julialang.org/t/api-traits-for-chunked-out-of-core-arrays/36873/3 "2020-04-01T22:40:14Z")

</div>

Thanks for the link!

In addition to chunking information, I guess it would be useful if we had a standard API to determine where data (in an array, or maybe just data in general) is located (CPU-RAM, disk, GPU-RAM, etc.) so user code can make informed decisions on when and how to convert (e.g. resulting in a GPU to CPU transfer, …).

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [April 2, 2020, 2:17am UTC](https://discourse.julialang.org/t/api-traits-for-chunked-out-of-core-arrays/36873/4 "2020-04-02T02:17:22Z")

</div>

Maybe that’s a nice addition to [GitHub - JuliaArrays/ArrayInterface.jl: Designs for new Base array interface primitives, used widely through scientific machine learning (SciML) and other organizations](https://github.com/SciML/ArrayInterface.jl) or [GitHub - JuliaGPU/Adapt.jl](https://github.com/JuliaGPU/Adapt.jl).

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [April 2, 2020, 9:14am UTC](https://discourse.julialang.org/t/api-traits-for-chunked-out-of-core-arrays/36873/5 "2020-04-02T09:14:12Z")

</div>

Oh, yes!

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [April 2, 2020, 1:31pm UTC](https://discourse.julialang.org/t/api-traits-for-chunked-out-of-core-arrays/36873/6 "2020-04-02T13:31:43Z")

</div>

There’s also [https://github.com/meggart/DiskArrays.jl](https://github.com/meggart/DiskArrays.jl), of course, in a more specific context.

CC @fabiangans

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [April 2, 2020, 3:23pm UTC](https://discourse.julialang.org/t/api-traits-for-chunked-out-of-core-arrays/36873/7 "2020-04-02T15:23:23Z")

</div>

Yes I was about to mention DiskArrays as well. However, in the future it would be nice if the package’s functionality could be integrated in whatever Array trait system people will come up with in the future so that DiskArrays integrate more nicely with other special array types.