# Taking the array indexing interface seriously

**URL:** <https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035>\
**Category:** Internals & Design\
**Tags:** question\
**Created:** [December 9, 2019, 4:42am UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035 "2019-12-09T04:42:53Z")\
**Posts on this page:** 12\
**Page:** 2

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [December 10, 2019, 8:25am UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/21 "2019-12-10T08:25:13Z")

</div>

> [@Raf](#):
>
> The key base methods seem to be calling `to_index(A, I)` which then requires the method `eachindex(::LinearIndices, A::MyArrayLike)` and `setindex_shape_check(value, I...)` for the value array in `setindex!` .

Just looked at your PR, this looks like a very good solution. I will try to adapt this for NetCDF and Zarr so that we get identical indexing behaviors across these packages.

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [December 10, 2019, 8:34am UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/22 "2019-12-10T08:34:39Z")

</div>

> [@Raf](#):
>
> An `AbstractDiskArray` that follows `AbstractArray` indexing correctly, but defines more sensible baviors for disk-backed array-like data - like avoiding the issues with `show` or broadcast. Traits could be used for chunking?

I think this sounds like a reasonable idea. Your PR for HDF5 solves the problem of pure indexing into these arrays, but not the broadcast and other problems that you get when doing operations on these. Of course the next step would be to customize broadcasting on these arrays, but still a user calling stuff like sum, reduce, and basically everything you want to do with and array would face terrible slowness.

So I agree that an AbstractDiscArray would be a nice array type, where we can define relatively efficient default operations for broadcasting, reductions, and other operations.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [December 10, 2019, 10:47am UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/23 "2019-12-10T10:47:22Z")

</div>

Absolutely agree. We should put together a DiskArrays package at some stage. Maybe it could leverage ChunkedArrayBase.jl for reduce/broadcast efficiency.

---

<div class="post-metadata">

**Author:** ![Ratingulate](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ratingulate/32/9242_2.png) [@Ratingulate](https://discourse.julialang.org/u/Ratingulate)\
**Post date:** [December 10, 2019, 2:00pm UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/24 "2019-12-10T14:00:43Z")

</div>

I think this is something that [dagger](https://github.com/JuliaParallel/Dagger.jl) is/was meant to help with cc @shashi

---

<div class="post-metadata">

**Author:** ![musm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/musm/32/3675_2.png) [@musm](https://discourse.julialang.org/u/musm)\
**Post date:** [December 11, 2019, 5:56pm UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/25 "2019-12-11T17:56:17Z")

</div>

can you explain what aspect of dagger can help here?

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [January 22, 2020, 1:48pm UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/26 "2020-01-22T13:48:38Z")

</div>

For everyone who was involved in this thread, but is not following the JuliaGeo community, I want to announce that we started [https://github.com/meggart/DiskArrays.jl](https://github.com/meggart/DiskArrays.jl) It implements an `AbstractDiskArray` type which subtypes `AbstractArray` and replaces Base methods that would be inefficient in random access.

The plan is then to use this package as a dependency for different disk array packages like NetCDF, Zarr, ArchGDAL or HDF5 to improve the user-experience when they use these arrays and to take away maintenance burden from package authors by having the Abstract Array interface in a single place.

One question that came up is which organization this DiskArrays should belong to. We might host it under JuliaGeo, but since these data formats are not only Geo-format specific maybe JuliaArrays would be a better option. So, any feedback on the package itself and the organization choice would be great.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [January 23, 2020, 1:46am UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/27 "2020-01-23T01:46:14Z")

</div>

I think JuliaArrays would be more appropriate. I‘d also encourage you to make a separate announcement thread for DiskArrays showing it off to the wider Julia community.

I have a question in the meantime though: could you clarify for me the advantages / disadvantage of this relative to the mmaped arrays from the standard library?

---

<div class="post-metadata">

**Author:** ![laborg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laborg/32/5474_2.png) [@laborg](https://discourse.julialang.org/u/laborg)\
**Post date:** [January 23, 2020, 5:31am UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/28 "2020-01-23T05:31:01Z")

</div>

> [@Mason](#):
>
> I have a question in the meantime though: could you clarify for me the advantages / disadvantage of this relative to the mmaped arrays from the standard library?

As an example, I think this would allow nice access to HDF5 files that uses chunked (and compressed) data layouts. (HDF5 `readmmap` is only allowed for contiguous layout)

---

<div class="post-metadata">

**Author:** ![laborg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laborg/32/5474_2.png) [@laborg](https://discourse.julialang.org/u/laborg)\
**Post date:** [January 23, 2020, 5:43am UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/29 "2020-01-23T05:43:16Z")

</div>

That sounds great and would reduce the implementation I’m doing right now.

One question: Are you planing to support a “intelligent” `StepRange`s? When sampling an array with a large step it’s not clear if there aren’t access patterns that would be more efficient. eg. Instead of loading all chunks from the file and exhausting the cache, single reads might be better. A simple start could be to fall back to single reads if `step` \> `chunksize`.

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [January 23, 2020, 8:55am UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/30 "2020-01-23T08:55:11Z")

</div>

> [@laborg](#):
>
> As an example, I think this would allow nice access to HDF5 files that uses chunked (and compressed) data layouts. (HDF5 `readmmap` is only allowed for contiguous layout)

Exactly. In adition, although the package is named “DiskArrays”, it is supposed to play nicely together with cloud-backed data (e.g. through Zarr.jl), where chunks are represented as objects in an object storage and accessing them has significant overhead through http transfer.

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [January 23, 2020, 9:06am UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/31 "2020-01-23T09:06:27Z")

</div>

> [@laborg](#):
>
> One question: Are you planing to support a “intelligent” `StepRange` s? When sampling an array with a large step it’s not clear if there aren’t access patterns that would be more efficient. eg. Instead of loading all chunks from the file and exhausting the cache, single reads might be better. A simple start could be to fall back to single reads if `step` \> `chunksize` .

I have definitely thought about this and it would be nice to implement something efficient and not too complex here. On the other hand, at least NetCDF and HDF5 already provide special methods to read data that way (e.g.[`nc_get_vars_`](https://www.unidata.ucar.edu/software/netcdf/docs/group__variables.html#gac695ffee87e2c3cd2dd22a59da5f2cc6) family in NetCDF) where you can specify a stride along every dimension, so it would still be good to have this access pattern be customisable by the backend and provide a simple fallback like the one you described.

Feel free to start an issue in the package and suggest what exactly you had in mind.

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [February 3, 2020, 3:16pm UTC](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035/32 "2020-02-03T15:16:13Z")

</div>

> [@Mason](#):
>
> I‘d also encourage you to make a separate announcement thread for DiskArrays showing it off to the wider Julia community.

The package is registered now and I just announced it here: [[ANN] DiskArrays.jl](https://discourse.julialang.org/t/ann-diskarrays-jl/34119)

[Previous page](https://discourse.julialang.org/t/taking-the-array-indexing-interface-seriously/32035.md?page=1)