I’m doing some work standardizing array interfaces for the spatial data ecosystem in HDF5 , ArchGDAL and in GeoData . It’s not uncommon for widely use packages to have non-standard indexing inputs and outputs. HDF5.jl currently doesn’t drop dims indexed with Int, so it returns unexpected array dimen…

AbstractArrays have a handful of special behaviors, but it really just boils down to these four things: Extensible support for converting the passed indices to a single location or array of locations with to_indices/to_index for each dimension. This is what handles Integer conversions, CartesianI…

Generally I agree that types should not pun on functions from another context. Eg an FTP client and a drawing library should not use the same connect method. But getindex and setindex! is somewhat of a special case since they have a nice custom syntax with []. So a lot of packages which want to use…

For sure, it has a lot of uses. But if the indexing looks identical to array indexing and returns an array - it really should stick to the array interface, even if it isn’t an Array itself. This includes HDF5Dataset and many similar Dataset objects that are loaded from disk in other packages.

[image] Raf: Although quirks like the one in my first post are not. Single- and multi-dimensional Arrays · The Julia Language talks about “extra indices” I really do suspect that HDF5 is a special case due to the peculiarities of disk-access patterns and its heritage dating from the Julia 0.…

Right. HDF5.jl doesn’t support omitted or extra indices either… Seems like a real <: AbstractArray object would be needed to do these things properly - maybe that’s the answer to this. Just make a wrapper that is actually <: AbstractArray whenever possible.

@fabiangans thinking about what you said a separate interface for disk-based arrays is really something that is missing here. An AbstractDiskArray that follows AbstractArray indexing correctly, but defines more sensible baviors for disk-backed array-like data - like avoiding the issues with show or…

Ok that is a great summary. Maybe I’ll integrate that in the introduction paragraph in the array docs. Using to_indices/to_index is something that isn’t happening in array-like data packages. HDF5.jl, Zarr.jl, my pull request to ArchGDAL.jl (lol), probably a few others. Some other disk based array…

Thanks everyone, this should lead to proper indexing for at least HDF5 https://github.com/JuliaIO/HDF5.jl/pull/603 and ArchGDAL. The key base methods seem to be calling to_index(A, I) which then requires the method eachindex(::LinearIndices, A::MyArrayLike) and setindex_shape_check(value, I...) for…

[image] Raf: The key base methods seem to be calling to_index(A, I) which then requires the method eachindex(::LinearIndices, A::MyArrayLike) and setindex_shape_check(value, I...) for the value array in setindex! . Just looked at your PR, this looks like a very good solution. I will try to ad…

[image] Raf: An AbstractDiskArray that follows AbstractArray indexing correctly, but defines more sensible baviors for disk-backed array-like data - like avoiding the issues with show or broadcast. Traits could be used for chunking? I think this sounds like a reasonable idea. Your PR for HDF5…

Absolutely agree. We should put together a DiskArrays package at some stage. Maybe it could leverage ChunkedArrayBase.jl for reduce/broadcast efficiency.

Taking the array indexing interface seriously

Internals & Design

fabiangans February 3, 2020, 3:16pm 32

The package is registered now and I just announced it here: [ANN] DiskArrays.jl

5 Likes

Topic		Replies	Views
Proposal: working with larger than memory data in hdf5 format using HDF5Arrays (implementation of DiskArrays.jl for HDF5) Data hdf5	11	1821	November 4, 2020
Is OffsetArrays.jl a poison pill? Internals & Design	63	5594	December 5, 2022
Does `AbstractArray` itself need to support indexing beyond 1-based? Internals & Design indexing	15	1599	July 5, 2022
`eachindex(x)` replacement for `1:length(x)-1` and similar General Usage	27	3367	May 24, 2022
DiskArrays get array of values given a vector of Indices New to Julia question	6	432	April 27, 2022

Taking the array indexing interface seriously

Related topics