It appears that the entire Julia data science ecosystem is built around Julia’s Array. However, from what I’ve observed, Array’s flexibility makes it too disuniform to serve as an in-memory data structure for sharing information between data science applications: Any element within an array can be…

I think this discussion will be more productive if you edit to remove the “in the core language” part of it. That will likely draw a lot of the attention but it’s bound to be far off if we don’t even have a package implementation yet.

What does “columnar-typed” mean when the number of dimensions is greater than 2?

@CameronBieganek Just updated the original post with an example. Beyond 2D, everything is essentially just grouping brackets. E.g. many color channels across many images.

OP, can you take the tone down a little bit on this post? I worry about a lengthy flamewar which I will inevitably mute. A few notes There are many posts that start with “In order to convert X community, we need to do Y”. Julia’s adoption is steadily increasing, so claims to the effect of “if only…

What are the advantages of this over just using DataFrames.jl? You say that it is only for representing 2D data but that is clearly not the case. Your example can be made into a DataFrame and then you can groupby whatever columns you want (some of which will define which dimension you are slicing).

@tbeason good point. i guess it’s really a matter of accessing the data, where it originates from, and how you want to distribute the execution. if you’re iterative reading data from sources like individual Array or files, then assembling separate arrays and accessing/batching them via index-per arr…

An AbstractArray type where different axes have different types, but everything still inferred correctly, is probably doable, and would be a very interesting package. All you would have to do would be to encode the types of the different axes in your struct declaration. I would encourage you to writ…

Perhaps https://github.com/SciML/RecursiveArrayTools.jl would be of interest as well?

As I mentioned on Slack (but evidently seems to have been buried), the statically-typed case here is handled by StructArrays. For more dynamic manipulation, something like https://github.com/JuliaGeo/NetCDF.jl , https://github.com/meggart/YAXArrays.jl or https://github.com/JuliaHEP/UpROOT.jl is worth…

@ToucheSir you’re right. the goal of xarray is the multidimensionality of numpy + with the labeling metadata of pandas. neither is sufficient alone. [image] Overview: Why xarray? Xarray introduces labels in the form of dimensions, coordinates and attributes on top of raw NumPy-like mul…

Call for a data structure that is: multi-dimensional, columnar-typed, and in-memory in the core Julia language

General Usage

jzr July 22, 2021, 5:22pm 13

There are some packages like xarray Status of AxisArrays.jl - #20 by Raf

Topic		Replies	Views
Most popular tabular/multidimensional data types in Julia New to Julia data , type , dataframes	18	1494	December 8, 2021
Implementing an equivalent of python's xarray (labeled nd-arrays) in julia Data package	15	5636	October 3, 2024
Can DataFrames.jl handle multi-dimensional arrays General Usage question	6	4328	November 6, 2019
The fate of DimensionalArrays / AxisArrays in Julia, and which to actually use Specific Domains	7	3478	May 19, 2022
Converting between AxisArray like packages? General Usage	18	1612	February 27, 2021

Call for a data structure that is: multi-dimensional, columnar-typed, and in-memory in the core Julia language

Related topics