# Call for a data structure that is: multi-dimensional, columnar-typed, and in-memory in the core Julia language

**URL:** <https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524>\
**Category:** General Usage\
**Created:** [July 12, 2021, 6:21pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524 "2021-07-12T18:21:13Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![aiqc](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@aiqc](https://discourse.julialang.org/u/aiqc)\
**Post date:** [July 12, 2021, 6:21pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/1 "2021-07-12T18:21:13Z")

</div>

It appears that the entire Julia data science ecosystem is built around Julia’s Array. However, from what I’ve observed, Array’s flexibility makes it too disuniform to serve as an in-memory data structure for sharing information between data science applications:

- Any element within an array can be of any type. It’s a list of lists with somewhat passive metadata about shape and type.
- Its dimensionality is hard to pin down (often returns a vector of matrices or vectors instead of `Array{T, 3}`) → “I just want an array”
- It’s a challenge to combine arrays (LazyArray RecursiveArrayTools.jl) into multi-dimensional structures.

Although there are alternatives to Array with stronger columnar typing like StructArrays.jl and DataFrames.jl - they are only meant for handling 2D data (rows, columns). This falls apart pretty quickly in the deep learning space where a single image has 3D data (row, column, color) and a single time series has 3D data (batch, timestep, feature).

In order to mass-convert the Python data science community, who will be coming from daily NumPy/ TF/ Torch/ Parquet usage - Julia needs to provide a n-dimensional, columnar-typed, and in-memory data structure. This could of course be one of the existing 2D structures that evolves to handle 3D+ data.

- Ship this class in the core language.
- Promote/ encourage its adoption throughout the ecosystem.

* * *

Update example:

```julia
tensor = Tensor(
	[#3D (batch, sample, site, etc.)
		[#2D (channel, timestep, sample, etc.)
			#Types: Int, Float, String
			[1, 1.1, "a"],
			[2, 2.2, "b"],
			[3, 3.3, "c"]
		],
		[
			#Types: Int, Float, String
			[4, 4.4, "d"],
			[5, 5.5, "e"],
			[6, 6.6, "f"]
		],
		[
			#Types: Int, Float, String
			[7, 7.7, "g"],
			[8, 8.8, "h"],
			[9, 9.9, "i"]
		]
	]
)

```

---

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [July 12, 2021, 6:26pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/2 "2021-07-12T18:26:38Z")

</div>

I think this discussion will be more productive if you edit to remove the “in the core language” part of it. That will likely draw a lot of the attention but it’s bound to be far off if we don’t even have a package implementation yet.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [July 12, 2021, 6:31pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/3 "2021-07-12T18:31:19Z")

</div>

What does “columnar-typed” mean when the number of dimensions is greater than 2?

---

<div class="post-metadata">

**Author:** ![aiqc](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@aiqc](https://discourse.julialang.org/u/aiqc)\
**Post date:** [July 12, 2021, 6:37pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/4 "2021-07-12T18:37:29Z")

</div>

@CameronBieganek Just updated the original post with an example. Beyond 2D, everything is essentially just grouping brackets. E.g. many color channels across many images.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [July 12, 2021, 6:39pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/5 "2021-07-12T18:39:11Z")

</div>

OP, can you take the tone down a little bit on this post? I worry about a lengthy flamewar which I will inevitably mute. A few notes

- There are many posts that start with “In order to convert X community, we need to do Y”. Julia’s adoption is steadily increasing, so claims to the effect of “if only we had this feature, we would increase the userbase this much” stem from a flawed assumption that the userbase is stagnant.
- Adding to the core language. Why does this need to be added to the core language? Plenty of people use `AbstractArray`s all the time, and this is the first we’ve heard of this particular problem. I would suggest writing a package first that will show people _concretely_ the flaws in the current design and how an alternative will improve it.

You have added code, but not a full MWE for why your proposed structure will be worth it. Many people on slack (including myself) mentioned that it’s difficult to move the conversation forward without a clear picture of the problem you are trying to solve.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [July 12, 2021, 7:05pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/7 "2021-07-12T19:05:34Z")

</div>

What are the advantages of this over just using DataFrames.jl? You say that it is only for representing 2D data but that is clearly not the case. Your example can be made into a DataFrame and then you can `groupby` whatever columns you want (some of which will define which dimension you are slicing).

---

<div class="post-metadata">

**Author:** ![aiqc](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@aiqc](https://discourse.julialang.org/u/aiqc)\
**Post date:** [July 12, 2021, 7:15pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/8 "2021-07-12T19:15:35Z")

</div>

@tbeason good point. i guess it’s really a matter of accessing the data, where it originates from, and how you want to distribute the execution. if you’re iterative reading data from sources like individual Array or files, then assembling separate arrays and accessing/batching them via index-per array is more intuitive (don’t have to group by pixels) than managing the dimensions yourself in queries. E.g. give me the 5th sample from this 4D array `[:,;,:,5]` On the other hand, if your data can more naturally be persisted in tabular form then the grouping you describe makes sense. that’s how dask goes about fetching data from parquet columns in that partition/ grouping/ chunk\_size style. but at the end of the day, you feed an n-d array, not dataframes, into a neural network.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [July 12, 2021, 7:33pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/9 "2021-07-12T19:33:06Z")

</div>

An `AbstractArray` type where different axes have different types, but everything still inferred correctly, is probably doable, and would be a very interesting package. All you would have to do would be to encode the types of the different axes in your struct declaration. I would encourage you to write that package and see if it gains traction in the community.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [July 12, 2021, 7:35pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/10 "2021-07-12T19:35:52Z")

</div>

Perhaps [https://github.com/SciML/RecursiveArrayTools.jl](https://github.com/SciML/RecursiveArrayTools.jl) would be of interest as well?

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [July 12, 2021, 7:58pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/11 "2021-07-12T19:58:06Z")

</div>

As I mentioned on Slack (but evidently seems to have been buried), the statically-typed case here is handled by StructArrays. For more dynamic manipulation, something like [https://github.com/JuliaGeo/NetCDF.jl](https://github.com/JuliaGeo/NetCDF.jl), [https://github.com/meggart/YAXArrays.jl](https://github.com/meggart/YAXArrays.jl) or [https://github.com/JuliaHEP/UpROOT.jl](https://github.com/JuliaHEP/UpROOT.jl) is worth a look. If you feel like nothing already out there cuts it, than [Xarray documentation](http://xarray.pydata.org/en/stable/) is a good source of inspiration for writing a package. In general though, the Physics/Astronomy/Earth + Climate science people (many of which have probably seen or commented on your posts) are already way ahead of us folks who work with “AI” on neatly representing complex higher-dimensional data.

---

<div class="post-metadata">

**Author:** ![aiqc](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@aiqc](https://discourse.julialang.org/u/aiqc)\
**Post date:** [July 22, 2021, 10:49am UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/12 "2021-07-22T10:49:21Z")

</div>

@ToucheSir you’re right. the goal of _xarray_ is the multidimensionality of numpy + with the labeling metadata of pandas. neither is sufficient alone.

> **[Overview: Why xarray?](https://docs.xarray.dev/en/stable/getting-started-guide/why-xarray.html)**
>
> Xarray introduces labels in the form of dimensions, coordinates and attributes on top of raw NumPy-like multidimensional arrays, which allows for a more intuitive, more concise, and less error-pron...

Also (inactive?) [GitHub - nbren12/XArray.jl: Labeled ndarrays in julia](https://github.com/nbren12/XArray.jl)

---

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [July 22, 2021, 5:22pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/13 "2021-07-22T17:22:09Z")

</div>

There are some packages like xarray [Status of AxisArrays.jl - #20 by Raf](https://discourse.julialang.org/t/status-of-axisarrays-jl/28682/20)

---

<div class="post-metadata">

**Author:** ![aiqc](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@aiqc](https://discourse.julialang.org/u/aiqc)\
**Post date:** [July 24, 2021, 11:43am UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/14 "2021-07-24T11:43:52Z")

</div>

[https://github.com/rafaqz/DimensionalData.jl](https://github.com/rafaqz/DimensionalData.jl)

`DimArray` named dimensions plus Tables.jl interface with `DimTable`

---

<div class="post-metadata">

**Author:** ![aiqc](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@aiqc](https://discourse.julialang.org/u/aiqc)\
**Post date:** [August 25, 2021, 11:17pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/15 "2021-08-25T23:17:20Z")

</div>

> **[GitHub - JuliaIO/Zarr.jl](https://github.com/JuliaIO/Zarr.jl)**
>
> Contribute to JuliaIO/Zarr.jl development by creating an account on GitHub.

![image](https://global.discourse-cdn.com/julialang/original/3X/9/b/9b029a17a80d28d487dd18ea225dab68924a0862.png)

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [August 26, 2021, 8:26am UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/16 "2021-08-26T08:26:34Z")

</div>

@aiqc, could you point out what `Zarr.jl` does/will do that `DimensionalData.jl` doesn’t?

With so many arrays around, some heads are in disarray:

 ![disarray_where_is_my_pencil](https://global.discourse-cdn.com/julialang/original/3X/5/5/5543c345a19697d5ab8ebea42a99071c8c6ed60a.jpeg)

_ **NB:** _  
_Btw, may be a package logo that is more Julian?_

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [August 30, 2021, 4:42pm UTC](https://discourse.julialang.org/t/call-for-a-data-structure-that-is-multi-dimensional-columnar-typed-and-in-memory-in-the-core-julia-language/64524/17 "2021-08-30T16:42:49Z")

</div>

Zarr is a [cross-language standard](https://zarr.readthedocs.io/en/stable/) and is focused on persistence (kind of like HDF5 with less legacy baggage). Since Zarr.jl exposes an `AbstractArray` implementation, I imagine you could use it to back fancy array types like those in DimensionalData.