Hello,
What is the status of this package?
Even though there was some activity lately it looks as if it lacks a little bit of love an careโฆ
Can it already be used in a production system, or are a lot of hidden bugs to be expected?
Uwe
There is also the new DimensionalData.jl, announced recently ANN: DimensionalData.jl and GeoData.jl.
2 Likes
Thanks for the hint!
Is somebody aware of a package that does not encode the axes in its type?
See also WIP: The Plan ยท Issue #1 ยท JuliaCollections/AxisArraysFuture ยท GitHub
and the two packages that do each of the main halves and are designed around being composable:
- https://github.com/invenia/IndexedDims.jl/blob/master/src/IndexedDims.jl (WIP)
- GitHub - invenia/NamedDims.jl: For working with dimensions of arrays by name (less WIP, but still not fully happy. Am using in production though)
There is a lot going on in the area, and I think that is great.
Many options/parts to shake out the best way.
@davidanthoffโs GitHub - davidavdav/NamedArrays.jl: Julia type that implements a drop-in replacement of Array with named dimensions
Note that if you want zero overhead at runtime (like NamedDims does),
you basically need to encode the axes in the type, so that they can be compiled out of existance,
during specialization.
1 Like
You can use AxisArrays in production, Invenia does.
But they are very touchy and so can be frustrating to work with.
Any production system should have good enough integration tests that your confident that you are not hitting bugs (of course you are wrong, there are always bugs, but you want to minimize how often that happens)
As such any package can be used in production, the question is how frustrating will it be.
AxisArrays does not have hidden bugs that will sneak past you tests.
It has obvious bugs (/missing features),
like a ton of operations dropping the axes,
and missing overloads,
and confusing notation for how to work with things that are indexed with different Int ranges.
Which will easily be caught by your tests.
And it itself does still prevent certain categories of coding mistakes
1 Like
Its not fully clear to me, why this needs to be like that. The axis are just metadata and indexing could be done on the raw array. Its clear that permutedims and slicing require some additional things but things can keep type stable as long as one uses traditional indexing.
What I am looking for is a way to represent a tomographic data (3D) that has some center, some pixelspacing, and some rotation matrix in space.
My issue with AxisArrays is that I, for instance, cannot change the center, or the rotation matrix without creating a new type.
DimensionalData.jl will definitely have more bugs than AxisArrays.jl, its only at 0.1.0! But they will also be fixed promptly if you post an issue.
@tobias.knopp Iโm wondering what the negative consequences are for you of โcreating a new typeโ? do you mean you have problem with creating a new struct instance with a different type than the original?
DimensionalData.jl is mostly functional so the objects are frequently rebuilt. The compiler can elide the allocations most of the time anyway.
3 Likes
For instance I have written a Gtk based data viewer that can display 4D tomographic data. It looks something like:
struct DataViewer
data:: ???
end
With a simple Array, I can do
struct DataViewer
data::Array{Float32,4}
end
With an AxisArray this is much more complicated. And no,
struct DataViewer{T}
data::T
end
is not an option, because data can be changed at runtime.
You could just do:
mutable struct DataViewer
data::AbstractArray
end
The performance implictations are not as bad as you might think,
and are kinda similar to the ones that one avoids by putting the axes into the types.
1 Like
To get somewhat more concrete, here is the type that I am using
ImageMeta{Float32,5,AxisArray{Float32,5,Array{Float32,5},Tuple{Axis{:color,UnitRange{Int64}},Axis{:x,StepRangeLen{Quantity{Float64,๐,Unitful.FreeUnits{(mm,),๐,nothing}},Base.TwicePrecision{Quantity{Float64,๐,Unitful.FreeUnits{(mm,),๐,nothing}}},Base.TwicePrecision{Quantity{Float64,๐,Unitful.FreeUnits{(mm,),๐,nothing}}}}},Axis{:y,StepRangeLen{Quantity{Float64,๐,Unitful.FreeUnits{(mm,),๐,nothing}},Base.TwicePrecision{Quantity{Float64,๐,Unitful.FreeUnits{(mm,),๐,nothing}}},Base.TwicePrecision{Quantity{Float64,๐,Unitful.FreeUnits{(mm,),๐,nothing}}}}},Axis{:z,StepRangeLen{Quantity{Float64,๐,Unitful.FreeUnits{(mm,),๐,nothing}},Base.TwicePrecision{Quantity{Float64,๐,Unitful.FreeUnits{(mm,),๐,nothing}}},Base.TwicePrecision{Quantity{Float64,๐,Unitful.FreeUnits{(mm,),๐,nothing}}}}},Axis{:time,StepRangeLen{Quantity{Float64,๐,Unitful.FreeUnits{(s,),๐,nothing}},Base.TwicePrecision{Quantity{Float64,๐,Unitful.FreeUnits{(s,),๐,nothing}}},Base.TwicePrecision{Quantity{Float64,๐,Unitful.FreeUnits{(s,),๐,nothing}}}}}}},Dict{String,Any}}
1 Like
what a glorious type.
Godlike and powerful.
Huge like a mountain.
5 Likes
AbstractArray is, what I am currently doing. My issue is that my data viewer has a compile time of more than 16 seconds. second call is 0.2 s. Therefore my hope is that making data DataViewer concrete will make compile time (seems to be basically inference) smaller.
Powerful it is, and I like AxisArrays design actually. But then, please write a simple function that changes the center of an AxisArray that has spatial dimensions in the first three dims. Not obvious how to do that.
By the way: Is there an easy way for stripping the units from my type, while first converting things to SI of course (e.g. convert ms to s and so on). This would make the type much shorter in many cases.
1 Like
For those who are working on these new array interfaces Iโve been working on a AbstractIndices.jl package. I donโt know if it will ever make it out the door though because I have a lot of other obligations. The heart of it is this file https://github.com/Tokazama/AbstractIndices.jl/blob/master/src/abstractindex.jl. The goal was to make something that relied on very little unique internal behavior so that any changes to performance in base (in terms of sorting, indexing, and maybe even multithreading) would just come along with it.
The idea is that the only obstacle to making any AbstractVector into an index is knowing how to transform the user input into the index for the to_index function in base. Once thatโs done most indexing behavior is taken care of by to_axes in base.
The basic idea of how it works can be seen in the examples found here https://github.com/Tokazama/AbstractIndices.jl/blob/master/src/asindex.jl. It isnโt currently optimized for performance and I havenโt figured out the show method. Feel free to take whatever you want or let me know if you want some help implementing it in one of your packages. My only goal here is to have something thatโs very flexible and maintainable.
3 Likes
I should mention that coordination across packages is extremely important here. In the end, all what I want is
A = load("myimage.nii")
B = load("mydicomdata.dcm")
DataViewer(A)
DataViewer(B)
1 Like
I can almost guarantee that this wonโt be too much of an issue for the images interface as I have been specifically concerned with this very issue while rewriting the NIfTI package. If you take a look at some of the recent additions to ImageCore.jl, youโll see that Iโve started implementing a minimal trait based interface that will hopefully allow compatibility across different array paradigms. It basically comes down to returning a NamedTuple for the image based traits. I imagine this would be possible no matter what the community converges on.
So the problem is using a fixed-type mutable container of an immutable, functionally updated object that may change type in some of those updates. It would be interesting to see how using ::AbstractArray works to know how much of a problem this really is.
AbstractIndices looks interesting, funny how many of us have been writing similar things at the same time. I wonder if itโs possible to end up with one package that fits all of our requirements.
1 Like