Type management for NamedTuple's inconsistent or intended?

Internals & Design
1

Is there an inherent reason that type management of NamedTuple’s works differently from normal Tuple’s (in 1.7 at least)?

In particular, is the behavior of the following second command (case 1.2) intended for something? If so, can you give an explaining example? I have not really been able to find a simpler logic behind the given functionality.

I have included more examples below (which are less important but still haunt me), but I think it all boils down to the weird(?) syntax of the following fourth command (case 1.4):

typeof(convert(Tuple{AbstractArray}, (1:2,)))
# Tuple{UnitRange{Int64}} (case 1.1)
typeof(convert(NamedTuple{(:a,),Tuple{AbstractArray}}, (a=1:2,)))
# NamedTuple{(:a,), Tuple{AbstractArray}} (case 1.2)
typeof(convert(NamedTuple{(:a,),Tuple{T} where T<:AbstractArray}, (a=1:2,)))
# NamedTuple{(:a,), Tuple{AbstractArray}} (case 1.3)
typeof(convert(NamedTuple{(:a,),<:Tuple{AbstractArray}}, (a=1:2,)))
# NamedTuple{(:a,), Tuple{UnitRange{Int64}}} (case 1.4)
typeof(convert(NamedTuple{(:a,),Tuple{UnitRange}}, (a=1:2,)))
# NamedTuple{(:a,), Tuple{UnitRange}} (case 1.5)

Since for Tuple’s, we have Tuple{UnitRange{Int64}} <: Tuple{AbstractArray}, the first command makes sense. However, in the second and even the third command, the type Tuple{AbstractArray} appears, and only the fourth command will behave analogously to the first command. The fifth changes in turn the eltype of UnitRange to Any, which is somewhat what the second command does, but now for the eltype.

Secondly, there is also some trouble with Vector’s of NamedTuple’s, while for Tuple’s, types are automatically derived. Is this intended or maybe just not (yet) implemented?

typeof([(1:2,), ([1, 2],)]))
# Vector{Tuple{AbstractVector{Int64}}} (case 2.1)
typeof([(a=1:2,), (a=1:2,)])
# Vector{NamedTuple{(:a,), Tuple{UnitRange{Int64}}}} (case 2.2)
typeof([(a=1:2,), (a=[1, 2],)])
# Vector{NamedTuple{(:a,)}} (case 2.3)

Though one can manually convert the latter Vector, this becomes increasingly cumbersome due to the first issue:

typeof(convert(Vector{NamedTuple{(:a,),Tuple{AbstractArray}}}, [(a=1:2,), (a=[1, 2],)]))
# Vector{NamedTuple{(:a,), Tuple{AbstractArray}}} (case 3.1)
typeof(convert(Vector{NamedTuple{(:a,),<:Tuple{AbstractArray}}}, [(a=1:2,), (a=[1, 2],)]))
# Vector{NamedTuple{(:a,), <:Tuple{T} where T<:AbstractArray}} (case 3.2)
typeof(convert(Vector{NamedTuple{(:a,),<:Tuple{AbstractArray{Int64}}}}, [(a=1:2,), (a=[1, 2],)]))
# Vector{NamedTuple{(:a,), <:Tuple{AbstractArray{Int64}}}} (case 3.3)

Thirdly, when one in turn includes an abstract eltype Number, an error may be thrown. I have included some more commands for better reference, while the error is thrown by the last command.

typeof(convert(Tuple{AbstractArray{<:Number}}, (1:2,)))
# Tuple{UnitRange{Int64}} (case 4.1)
typeof(convert(Tuple{AbstractArray{Number}}, (1:2,)))
# Tuple{Vector{Number}} (case 4.2)
typeof(convert(NamedTuple{(:a,),Tuple{AbstractArray{<:Number}}}, (a=1:2,)))
# NamedTuple{(:a,), Tuple{AbstractArray{<:Number}}} (case 4.3)
typeof(convert(NamedTuple{(:a,),Tuple{AbstractArray{Number}}}, (a=1:2,)))
# NamedTuple{(:a,), Tuple{AbstractArray{Number}}}((Number[1, 2],)) (case 4.4)
typeof(convert(NamedTuple{(:a,),<:Tuple{AbstractArray{<:Number}}}, (a=1:2,)))
# NamedTuple{(:a,), Tuple{UnitRange{Int64}}} (case 4.5)
convert( NamedTuple{(:a,),<:Tuple{AbstractArray{Number}}}, (a=1:2,))
# throws error (case 4.6)
2

(you probably know some/all of this … its here in case any of this is new info. I understand your questions are not directly addressed below; perhaps it shows why converting to type containing an abstraction is disrecommended, and why Base.convert is extensible should you need special handling.)

An AbstractArray is an abstraction over Array-like entities. Neither a Tuple nor a NamedTuple is a subtype of AbstractArray. Conversion from either into an AbstractArray or a subtype thereof (e.g. a Vector) is an unsupported operation:

julia> atuple = (1, 2);
julia> anamedtuple = (a=1, b=2);

julia> convert(Vector, atuple)
ERROR: MethodError: Cannot `convert` an object of type
  Tuple{Int64, Int64} to an object of type
  Vector

julia> convert(AbstractArray, anamedtuple)
ERROR: MethodError: Cannot `convert` an object of type
  NamedTuple{(:a, :b), Tuple{Int64, Int64}} to an object of type
  AbstractArray

julia> convert(Vector{Int}, atuple)
ERROR: MethodError: Cannot `convert` an object of type Tuple{Int64, Int64} to an object of type Vector{Int64}

julia> Base.convert(::Type{Vector{T}}, atuple::NTuple{N,T}) where {N,T} = [atuple...]

julia> convert(Vector{Int}, atuple)
2-element Vector{Int64}:
 1
 2
1 Like
3

This is not about converting Tuple’s to Array’s.

I should maybe provide some context that hopefully helps. What I had initially in mind was to use constructions similar to

x = [(a=1:2, b=1:2), (a=[1,2], b=2:3)]
foo(x)

where foo has a method tailored to foo(x::Vector{NamedTuple{(:a,:b), Tuple{AbstractVector{Int64},AbstractVector{Int64}}}}). I could certainly use a custom struct

struct S
   a::AbstractArray
   b::AbstractArray
end

and overload foo(x::Vector{S}), but just defining const S as aboves NamedTuple type seemed more convenient, as the given application is not so much about speed. This is also since a struct S will not retain subtypes of AbstractArray.

Though in danger of this getting even longer, see the following which works quite nicely with Tuple’s but not so much with NamedTuple’s.

struct S
    a::AbstractArray{Int64}
    b::AbstractArray{Int64}
end
const NT = NamedTuple{(:a,:b), <:Tuple{<:AbstractArray{Int64},<:AbstractArray{Int64}}}
const URNT = NamedTuple{(:a,:b), <:Tuple{<:UnitRange{Int64},<:UnitRange{Int64}}}
const T = Tuple{AbstractArray{Int64},AbstractArray{Int64}}
const URT = Tuple{UnitRange{Int64},UnitRange{Int64}}
foo(x::Any) = "Any"
foo(x::Vector{<:NT}) = "NamedTuple"
foo(x::Vector{<:URNT}) = "NamedTuple of UnitRanges"
foo(x::Vector{<:T}) = "Tuple"
foo(x::Vector{<:URT}) = "Tuple of UnitRanges"
foo(x::Vector{<:S}) = "Struct"

display(foo([(1:2,1:2),([1,2],2:3)]))
# "Tuple"
display(foo([(1:2,1:2),(1:2,2:3)]))
# "Tuple of UnitRanges"     <- quite convenient
display(foo([(a=1:2, b=1:2), (a=[1,2], b=2:3)]))
# "Any"
display(foo([(a=1:2, b=1:2), (a=1:2, b=2:3)]))
# "NamedTuple of UnitRanges"
display(foo([S(1:2,1:2),S([1,2],2:3)]))
# "Struct"
display(foo([S(1:2,1:2),S(1:2,2:3)]))
# "Struct"     <- can not retain NamedTuple with custom struct

display(foo(T[(1:2,1:2),([1,2],2:3)]))
# "Tuple"
display(foo(T[(1:2,1:2),(1:2,2:3)]))
# "Tuple"     <- this a-priorly sets abstract type
display(foo(NT[(a=1:2, b=1:2), (a=[1,2], b=2:3)]))
# "NamedTuple"
display(foo(NT[(a=1:2, b=1:2), (a=1:2, b=2:3)]))
# "NamedTuple"     <- better, but remedies UnitRange type
display(foo(S[S(1:2,1:2),S([1,2],2:3)]))
# "Struct"
display(foo(S[S(1:2,1:2),S(1:2,2:3)]))
# "Struct"

display(foo(convert(Vector{<:T},[(1:2,1:2),([1,2],2:3)])))
# "Tuple"
display(foo(convert(Vector{<:T},[(1:2,1:2),(1:2,2:3)])))
# "Tuple of UnitRanges"     <- again convenient as one can enforce this via bar(...)::Vector{<:T}
display(foo(convert(Vector{NT},[(a=1:2, b=1:2), (a=[1,2], b=2:3)]))) # 
# "NamedTuple"     <- Vector{<:NT} instead throws error
display(foo(convert(Vector{NT},[(a=1:2, b=1:2), (a=1:2, b=2:3)]))) # Vector{<:NT} throws error
# "NamedTuple"     <- Vector{<:NT} instead throws error
4

IIUC your complaint is that promotion of tuples gives a more specific element type than promotion of NamedTuples:

julia> eltype([(1:2, 1:2), ([1,2], 2:3)])
Tuple{AbstractVector{Int64}, UnitRange{Int64}}

julia> eltype([(a=1:2, b=1:2), (a=[1,2], b=2:3)])
NamedTuple{(:a, :b)}

This is due to tuples using particular promotion rules (they are very special in the type system), while NamedTuples just use the fallback promotion rules of any struct, which call typejoin. I agree it can be annoying, I had made a pull request to change this. But I don’t really know what would be the drawbacks.

1 Like
5

Well I would not go as far as to wish to complain, but rather be curious about it. I tried to describe potential drawbacks in the example above, that is, the difficulties to work with types of Array’s of NamedTuple’s.

Apart from that, it took me a while to realize where my errors came from due to the different behaviors and I thought to have misunderstood rules for types. So this might be something others struggle or have struggled with too. The pull request you linked is unfortunately yet above my understanding of Julia. I think it is not just about promotion.

In that sense, I so far only have a naturally arising, superficial perspective on Julia that is not much influenced what may be behind the implementation.

One could maybe argue that the name NamedTuple is a bit unfortunate when the behavior rather seems to rely on struct rules, if I understood it correctly.

It is just that Julia seems overall very consistent and intuitive, but for example explaining this behavior already takes some deeper insight. Naively, one would think that NamedTuple’s just adds names and otherwise behaves equivalently to Tuple’s. Yet the contained type <:Tuple{AbstractArray} as in aboves case 1.4 does not even exist outside of the NamedTuple.