# Common interface for chunked arrays?

**URL:** <https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009>\
**Category:** Data\
**Tags:** hdf5, parallel, data\
**Created:** [July 4, 2019, 1:13pm UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009 "2019-07-04T13:13:24Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [July 4, 2019, 1:13pm UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009/1 "2019-07-04T13:13:24Z")

</div>

Hi all,

there is quite a zoo of data formats that allow you to store multidimensional arrays tiled into chunks, with or without compression. Examples are [HDF5.jl](https://github.com/JuliaIO/HDF5.jl), [NetCDF.jl](https://github.com/JuliaGeo/NetCDF.jl), [Zarr.jl](https://github.com/meggart/Zarr.jl), or [BigArrays.jl](https://github.com/seung-lab/BigArrays.jl), maybe there will be a [TileDB](https://tiledb.io/) wrapper soon.

It would be really nice if there was a generic way to access and process these arrays with respect to their chunked nature, so that one could iterate over chunks, copy from one format to another, efficiently distribute computations over workers respecting the chunks, etc through a common interface. This would greatly enhance possible coupling to tiled in-memory data formats and processing tools like [TiledIteration.jl](https://github.com/JuliaArrays/TiledIteration.jl) or [DistributedArrays.jl](https://github.com/JuliaParallel/DistributedArrays.jl)

When searching for existing interfaces, the closest thing I found was [BlockArrays.jl](https://github.com/JuliaArrays/BlockArrays.jl). However, the model does not completely fit the formats mentioned above, since it seems to focues on variable-sized “Blocks” while in the formats mentioned above, the chunks have all equal size. Then there is [Blocks.jl](https://github.com/JuliaParallel/Blocks.jl) from JuliaParrallel, which seems to provide exactly the interface I thought about but is abandoned.

My main question is if there are people who would support an effort to harmonize a chunk interface for some of the above-mentioned data formats, ideally contributors/authors of these packages and if there are other already existing interfaces that might be ready to use and that I have missed in my search.

It is also unclear to me if Chunk type defined in [Dagger.jl](https://github.com/JuliaParallel/Dagger.jl/blob/master/src/chunks.jl#L34) is generic enough to deal with our use case and would give us a lot of chunked processing for free. So far I have seen only examples of Dagger being applied to JuliaDB instances, although it should be generic enough to deal with arbitrary chunked datasets, so maybe this is the way to go. Any explanation/input here would be welcome as well.

**TLDR**

What is the best interface/framework to deal with chunked, dense multidimensional arrays data formats

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [July 4, 2019, 2:02pm UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009/2 "2019-07-04T14:02:25Z")

</div>

I think that there are at least two parallel goals the libraries you mention address:

1. some structure, typically sparsity, that can be taken advantage of (BlockArrays),

2. a purely mechanical partition of elements to facilitate processing the data in parts (HDF & friends).

They have some intersection, but neither is a subset of the other.

As for a common API for chunks, one could imagine an `eachchunk(A)` iterator that returns some `I::CartesianIndices` that conform to the storage layout so that `A[I]` can be retrieved efficiently.

Do you need anything more?

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [July 4, 2019, 2:47pm UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009/3 "2019-07-04T14:47:07Z")

</div>

There are also `Apache Arrow`, `Feather` and `Parquet` with somewhat different set of features / requirements: [python - What are the differences between feather and parquet? - Stack Overflow](https://stackoverflow.com/questions/48083405/what-are-the-differences-between-feather-and-parquet)

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [July 5, 2019, 7:11am UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009/4 "2019-07-05T07:11:43Z")

</div>

> I think that there are at least two parallel goals the libraries you mention address:

Thanks for the clarification I am currently more interested in your number 2 and this is the set of packages I am currently using. I was just wondering if concepts from (1) might help here.

> Do you need anything more?

I think this might already bring me a long way. In particular if the `eachchunk` iterator would implement `HasShape` so that one would know beforehand how to organize the individual chunks. Maybe in the future an optional `affinity` function would help as well to facilitate interaction with `DistributedArrays`.

I will experiment a bit and keep you updated. Do you think it would be feasible to create a package that only defines a single function name?

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [July 5, 2019, 7:15am UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009/5 "2019-07-05T07:15:21Z")

</div>

Thanks, I did not mention columnar formats here, because I am more interested in dense, n-dimensional arrays. However, there might be some overlap in concepts and maybe one could define a common interface for these, but it is currently not my focus.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [July 5, 2019, 7:31am UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009/6 "2019-07-05T07:31:48Z")

</div>

> [@fabiangans](#):
>
> Do you think it would be feasible to create a package that only defines a single function name?

Yes, I think a minimal “interface” package is the way to go. You may want to coordinate with the above packages though.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [July 5, 2019, 7:35am UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009/7 "2019-07-05T07:35:46Z")

</div>

For the chunked array containers similar to `BlockArray`, see also `Vcat` and `Hcat` in [https://github.com/JuliaArrays/LazyArrays.jl](https://github.com/JuliaArrays/LazyArrays.jl) and `ArrayPartition` in [GitHub - SciML/RecursiveArrayTools.jl: Tools for easily handling objects like arrays of arrays and deeper nestings in scientific machine learning (SciML) and other applications](https://github.com/JuliaDiffEq/RecursiveArrayTools.jl). Those are different from `BlockArray` in that number and type of chunks are encoded in the type. I think they still fit in @Tamas_Papp’s category 1. If you are to design some kind of a common API, it might be useful to consider an interface that can deal with homogeneous and heterogeneous types. But it probably just means to return a tuple from `eachchunk` for the heterogeneous case.

(fixed wording: chunk size and types → number and type of chunks)

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [July 5, 2019, 8:40am UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009/8 "2019-07-05T08:40:24Z")

</div>

Thanks for the additional pointers. I had not thought about heterogeneous arrays before and will try to consider this as well.

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [July 8, 2019, 2:31pm UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009/9 "2019-07-08T14:31:34Z")

</div>

I drafted a small interface package here [GitHub - meggart/ChunkedArrayBase.jl](https://github.com/meggart/ChunkedArrayBase.jl) and added a few usage example in the Readme that test the functionality. Basically the interface works fine and comments on details would be very welcome as an issue in the repository. I opened an issue in HDF5 as well, Zarr and NetCDF will follow. For now I omitted dealing with the heterogenous case, but I think this would be quite easy to extend by either adding a second iterator (`eachchunktype`) or indeed by returning a tuple instead.

Thanks again for the input.

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [August 24, 2019, 10:16pm UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009/10 "2019-08-24T22:16:20Z")

</div>

Which of those formats allow to add new columns or reshape the data directly on disk, without needing to copy and create a new one?  
It will be very useful when dealing with larger than memory datasets.

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [August 26, 2019, 6:56am UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009/11 "2019-08-26T06:56:16Z")

</div>

> [@Juan](#):
>
> Which of those formats allow to add new columns or reshape the data directly on disk, without needing to copy and create a new one?

I think all of them. In NetCDF/HDF5 you can define a dimension with _unlimited_ length, which means you can keep appending data to it. For Zarr you can always [resize/append along any dimension](https://meggart.github.io/Zarr.jl/latest/tutorial/#Resizing-and-appending-1).

For all of these packages this will only be efficient if you define you chunking accordingly so that ideally you always append complete chunks to your dataset.