Status of AxisArrays.jl

Hello,
What is the status of this package?
Even though there was some activity lately it looks as if it lacks a little bit of love an careโ€ฆ
Can it already be used in a production system, or are a lot of hidden bugs to be expected?
Uwe

There is also the new DimensionalData.jl, announced recently ANN: DimensionalData.jl and GeoData.jl.

2 Likes

Thanks for the hint!

Is somebody aware of a package that does not encode the axes in its type?

See also https://github.com/JuliaCollections/AxisArraysFuture/issues/1
and the two packages that do each of the main halves and are designed around being composable:

There is a lot going on in the area, and I think that is great.
Many options/parts to shake out the best way.

@davidanthoffโ€™s https://github.com/davidavdav/NamedArrays.jl

Note that if you want zero overhead at runtime (like NamedDims does),
you basically need to encode the axes in the type, so that they can be compiled out of existance,
during specialization.

1 Like

You can use AxisArrays in production, Invenia does.
But they are very touchy and so can be frustrating to work with.

Any production system should have good enough integration tests that your confident that you are not hitting bugs (of course you are wrong, there are always bugs, but you want to minimize how often that happens)
As such any package can be used in production, the question is how frustrating will it be.

AxisArrays does not have hidden bugs that will sneak past you tests.
It has obvious bugs (/missing features),
like a ton of operations dropping the axes,
and missing overloads,
and confusing notation for how to work with things that are indexed with different Int ranges.
Which will easily be caught by your tests.

And it itself does still prevent certain categories of coding mistakes

1 Like

Its not fully clear to me, why this needs to be like that. The axis are just metadata and indexing could be done on the raw array. Its clear that permutedims and slicing require some additional things but things can keep type stable as long as one uses traditional indexing.

What I am looking for is a way to represent a tomographic data (3D) that has some center, some pixelspacing, and some rotation matrix in space.

My issue with AxisArrays is that I, for instance, cannot change the center, or the rotation matrix without creating a new type.

DimensionalData.jl will definitely have more bugs than AxisArrays.jl, its only at 0.1.0! But they will also be fixed promptly if you post an issue.

@tobias.knopp Iโ€™m wondering what the negative consequences are for you of โ€œcreating a new typeโ€? do you mean you have problem with creating a new struct instance with a different type than the original?

DimensionalData.jl is mostly functional so the objects are frequently rebuilt. The compiler can elide the allocations most of the time anyway.

3 Likes

For instance I have written a Gtk based data viewer that can display 4D tomographic data. It looks something like:

  struct DataViewer
    data:: ???
  end

With a simple Array, I can do

  struct DataViewer
    data::Array{Float32,4}
  end

With an AxisArray this is much more complicated. And no,

  struct DataViewer{T}
    data::T
  end

is not an option, because data can be changed at runtime.

You could just do:

mutable struct DataViewer
    data::AbstractArray
end

The performance implictations are not as bad as you might think,
and are kinda similar to the ones that one avoids by putting the axes into the types.

1 Like

To get somewhat more concrete, here is the type that I am using

ImageMeta{Float32,5,AxisArray{Float32,5,Array{Float32,5},Tuple{Axis{:color,UnitRange{Int64}},Axis{:x,StepRangeLen{Quantity{Float64,๐‹,Unitful.FreeUnits{(mm,),๐‹,nothing}},Base.TwicePrecision{Quantity{Float64,๐‹,Unitful.FreeUnits{(mm,),๐‹,nothing}}},Base.TwicePrecision{Quantity{Float64,๐‹,Unitful.FreeUnits{(mm,),๐‹,nothing}}}}},Axis{:y,StepRangeLen{Quantity{Float64,๐‹,Unitful.FreeUnits{(mm,),๐‹,nothing}},Base.TwicePrecision{Quantity{Float64,๐‹,Unitful.FreeUnits{(mm,),๐‹,nothing}}},Base.TwicePrecision{Quantity{Float64,๐‹,Unitful.FreeUnits{(mm,),๐‹,nothing}}}}},Axis{:z,StepRangeLen{Quantity{Float64,๐‹,Unitful.FreeUnits{(mm,),๐‹,nothing}},Base.TwicePrecision{Quantity{Float64,๐‹,Unitful.FreeUnits{(mm,),๐‹,nothing}}},Base.TwicePrecision{Quantity{Float64,๐‹,Unitful.FreeUnits{(mm,),๐‹,nothing}}}}},Axis{:time,StepRangeLen{Quantity{Float64,๐“,Unitful.FreeUnits{(s,),๐“,nothing}},Base.TwicePrecision{Quantity{Float64,๐“,Unitful.FreeUnits{(s,),๐“,nothing}}},Base.TwicePrecision{Quantity{Float64,๐“,Unitful.FreeUnits{(s,),๐“,nothing}}}}}}},Dict{String,Any}}
1 Like

what a glorious type.
Godlike and powerful.
Huge like a mountain.

:joy:

5 Likes

AbstractArray is, what I am currently doing. My issue is that my data viewer has a compile time of more than 16 seconds. second call is 0.2 s. Therefore my hope is that making data DataViewer concrete will make compile time (seems to be basically inference) smaller.

Powerful it is, and I like AxisArrays design actually. But then, please write a simple function that changes the center of an AxisArray that has spatial dimensions in the first three dims. Not obvious how to do that.

By the way: Is there an easy way for stripping the units from my type, while first converting things to SI of course (e.g. convert ms to s and so on). This would make the type much shorter in many cases.

1 Like

For those who are working on these new array interfaces Iโ€™ve been working on a AbstractIndices.jl package. I donโ€™t know if it will ever make it out the door though because I have a lot of other obligations. The heart of it is this file https://github.com/Tokazama/AbstractIndices.jl/blob/master/src/abstractindex.jl. The goal was to make something that relied on very little unique internal behavior so that any changes to performance in base (in terms of sorting, indexing, and maybe even multithreading) would just come along with it.

The idea is that the only obstacle to making any AbstractVector into an index is knowing how to transform the user input into the index for the to_index function in base. Once thatโ€™s done most indexing behavior is taken care of by to_axes in base.

The basic idea of how it works can be seen in the examples found here https://github.com/Tokazama/AbstractIndices.jl/blob/master/src/asindex.jl. It isnโ€™t currently optimized for performance and I havenโ€™t figured out the show method. Feel free to take whatever you want or let me know if you want some help implementing it in one of your packages. My only goal here is to have something thatโ€™s very flexible and maintainable.

3 Likes

I should mention that coordination across packages is extremely important here. In the end, all what I want is

A = load("myimage.nii")
B = load("mydicomdata.dcm")
DataViewer(A)
DataViewer(B)
1 Like

I can almost guarantee that this wonโ€™t be too much of an issue for the images interface as I have been specifically concerned with this very issue while rewriting the NIfTI package. If you take a look at some of the recent additions to ImageCore.jl, youโ€™ll see that Iโ€™ve started implementing a minimal trait based interface that will hopefully allow compatibility across different array paradigms. It basically comes down to returning a NamedTuple for the image based traits. I imagine this would be possible no matter what the community converges on.

So the problem is using a fixed-type mutable container of an immutable, functionally updated object that may change type in some of those updates. It would be interesting to see how using ::AbstractArray works to know how much of a problem this really is.

AbstractIndices looks interesting, funny how many of us have been writing similar things at the same time. I wonder if itโ€™s possible to end up with one package that fits all of our requirements.

1 Like