I am trying to find a way to remove a specific type from a Union of types. In the code below, I would like to inherit from AbtractArray{T,N} if R is Nothing but otherwise inherit from the Union{T,R}:
julia> struct MyT{T,N,R} <: AbstractArray{ifelse(R isa Type{Nothing}, T, Union{T,R}), N}
parent
end
julia> foo(a :: AbstractArray{T,N}) where {T,N} = println("Type: $T Dims:$N")
foo (generic function with 1 method)
julia> q = MyT{Float64, 2, Nothing}(rand(3,4));
julia> foo(q)
Type: Union{Nothing, Float64} Dims:2
This does not work (since in the definition the types are a TypeVar), so I tried using typeintersect but there the problem is that I don’t know how to define a type which includes all types BUT Nothing such that I can use typeintersect. Another alternative would be to have a function typesubtract which removes a specific type from a union. This seems to be possible with this code:
remove_nothing_type(::Type{T}) where {T} = Core.Compiler.typesubtract(T, Nothing, 2)
But this approach causes an error ERROR: LoadError: UndefVarError: T not defined, if called inside the type definition.
Any ideas how to get this to work?
It’s actually kind of horrifying that your struct definition runs because isa and typeintersect can technically accept TypeVars. For example this would throw an error:
abstract type A{N} end
struct B{N} <: A{ifelse(N>0, N, -N)} end
# error: MethodError: no method matching isless(::Int64, ::TypeVar)
AFAIK there’s no way to do any meaningful computation on type parameters in the struct header. The calls in the type parameters are not delayed until the type exists, it just immediately runs on the T::TypeVar you gave it. It wouldn’t make sense in general for subtyping to have complex runtime behavior, imagine if I put a call that could return any one of multiple types like a rand call, I could make 1 concrete type have different abstract types at different times. I think your best bet is matching the T in the concrete type and the supertype, and you can specify whether that is a Float64 or Union{Float64, Nothing} at construction.
In my case such a type calculation is wanted, since I am “misusing” the homemade class CircShift as a dummy to fill as pad values in a ShiftedArray which then gets to be a CircShiftedArray. This may be a little bit hacky but I did not want to copy/paste large amounts of broadcasting boilerplate code between the two array types. Subtyping via an Abstract class may be an option. maybe I should look into this to find out how to share the code in this way. Are there any existing abstract SubTypes of AbstractArray which have some boilerplate code for broadcasting? I guess ShiftedArrays.jl, CircShiftedArrays, PaddedArrays, CatViews all are in need of something similar to properly work with CUDA.jl.
AbstractArray itself has broadcasting, you just need to implement some interface methods it needs. But that’s a different topic, the subtyping problem here comes first. The thing is, subtyping isn’t flexible enough to put concrete types within arbitrary abstract types, and they do have to share type parameters one-to-one, there’s no A{T} <: B{f(T)} except when A = B = Tuple; f = supertype.
Is there no way for you to refactor? There’s nothing wrong with MyT{Union{Float64, Int64}, 2}, it’s much clearer that way to show an array’s element type, especially if you’re already wrapping a parent array. If you want to get rid of the Nothing, you should copy the elements of your parent array to an array that can’t contain nothing, then wrap the new array instead. If I had a type MyT{Float64, 2, Nothing}, I’d expect it to be able to contain a nothing somewhere, so I’d prefer a MyT{Float64, 2} if I can’t have nothings.
Thanks a lot for these insights. Refactoring is for my application definately not the way to go.
I think I found a solution to my problem. The key insight was that the constructor has to be deciding which type to use:
struct MyType{T,N,R} <: AbstractArray{T, N}
parent
function MyType(myparent::AbstractArray{T, N}, default::R =nothing) where {T,N,R}
new{remove_nothing_type(Union{T,R}), N, R}(myparent)
end
end
Or even better by using multiple dispatch with multiple constructures for type stability.
As for the other issue with broadcasting: Yes, there is broadcasting in AbstractArray but this seems to assume that “get_index” can be called at will. Yet, when trying to avoid calls to “get_index”, which seems a necessity for the CuArray type, a surprisingly large amount of broadcasting code had to be written.
Inner constructor constraints make more sense because types do exist then, and maybe it can happen at compile-time.
There’s still problems, which I comment in the code example below. I don’t know of an implementation of remove_nothing_type so I just ad-libbed. I think the thing that stands out to me is that the same type can wrap parent arrays with different eltypes, which may not match the wrapper type’s supertype.
julia> begin
remove_nothing_type(::Type{T}) where T = Base.typesplit(T, Nothing)
struct MyType{T,N,R} <: AbstractArray{T, N}
parent # not annotated, so defaults to ::Any
function MyType(myparent::AbstractArray{T, N}, default::R =nothing) where {T,N,R}
new{remove_nothing_type(Union{T,R}), N, R}(myparent)
end
end
x = MyType([1]) # need to suppress REPL display, <:AbstractArray needs implementation
y = MyType(Union{Int, Nothing}[1, nothing])
z = MyType(Union{Int, Float64}[1, 1.5], 0.0) # weird need to instantiate R
nothing
end
julia> typeof(x), typeof(x.parent) # parent eltype matches
(MyType{Int64, 1, Nothing}, Vector{Int64})
julia> typeof(y), typeof(y.parent) # parent eltype does not match
(MyType{Int64, 1, Nothing}, Vector{Union{Nothing, Int64}})
julia> typeof(y) <: AbstractVector{Int64}, typeof(y.parent) <: AbstractVector{Int64}
(true, false)
julia> typeof(z), typeof(z.parent) # this is not what you intended
(MyType{Union{Float64, Int64}, 1, Float64}, Vector{Union{Float64, Int64}})
Thanks! The implementation of remove_nothing_type was stated at the very top in this thread
remove_nothing_type(::Type{T}) where {T} = Core.Compiler.typesubtract(T, Nothing, 2)
But yes, The type of the array is not always the type of its parent. But this is actually totally fine and the whole point. In the actual usecase the ShiftedArray can also return missing for elements of the shifted array, which are outside the original boundaries. This all depends on the type of the default argument. Since the array needs to return the correct type on calls of similar etc. we need to change both, the answer to eltype and the basetype of the AbstractArray to cover all use-cases.
I mostly expect eltype(w) === eltype(parent(w)), like CircShiftedArrays. The exceptions should be predictable like ShiftedArray including missings, and even then that’s a separate parameter M from the T parameter, which it does share with its parent. I’m not exactly sure what sort of ShiftedArray variant you are making, but having a wrapper’s type parameters not fully dependent on the parent’s type paremeters seems really hard to work with, even if you do annotate the .parent field’s type.
I think the functionality of the original ShiftedArray and CircShiftedArray implementation is now fully supported and the CircShiftedArray is actually a ShiftedArray with a different type default parameter. Click on this link to look at the code (and comment), if this is of interest. ShiftedArray previously also did not generally adhere to eltype(w) === eltype(parent(w)), which is why the revised implementation had to also adhere to this sheme.
This is an interesting refactorization, ShiftedArray and the AbstractArray’s element type don’t have the M parameter anymore. So what happens if you run the doc’s example (or its equivalent) and check the eltype? Is the Missing still there, and if so how did you put it back after removing M from the AbstractArray?
julia> v = reshape(1:16, 4, 4);
julia> s = ShiftedArray(v, (2, 0));
julia> eltype(s)
This is more just me asking a question, as for your topic, I think you’re on the right track handling it in an inner constructor.
I don’t quite understand what you mean by the M parameter. Presumably you are referring to N which is present in the inheritance but NOT in the parent array used in the contruction. Indeed. This allowed these two to differ.
julia> v = reshape(1:16, 4, 4);
julia> s = ShiftedArray(v, (2, 0));
julia> eltype(s)
Union{Missing, Int64}
Skimming through the link to your ocmmits, I saw this change to the ShiftedArray struct in shiftedarray.jl:
-struct ShiftedArray{T, M, N, S<:AbstractArray} <: AbstractArray{Union{T, M}, N}
- parent::S
+struct ShiftedArray{T, N, A<:AbstractArray, R} <: AbstractArray{T, N}
+ parent::A
and wondered how eltype would replicate the prior Union{T, M}, but now that I look again, I see that shifted_array_base_type(Tb, default) takes care of that in the inner constructor.
If ShiftedArray with an R is your basis for the MyType example here, I would say that if I can getindex the default value at any indices, I fully expect the array’s eltype to be Union{T, R}. If the default value is not present at any index, I think you can use your approach when R=CircShift where you just return T, doesn’t seem necessary to remove_nothing_type(Union{T,R}) then.
