How is the type of a heterogeneous vector determined when splatting and doing list comprehension?

General Usage
1

Consider the type of the resulting Vector in these three cases:

[1, 1.0]  # case 1: Vector{Float64}, elements are promoted
v = Any[1, 1.0]
[v...]  # case 2: gives identical result to case 1
[x for x in v]  # case 3: Vector{Real}, elements are not promoted

My first question is, what is the logic behind case 3 being different from case 1 and case 2?

In any case, I would like to replicate behavior analogous to this with custom types. I was able to reproduce the splatting behavior (case 1/2), by implementing methods for promote_rule and convert, but I do not know how to achieve the behaivor analogous to case 3 for list comprehension.

Let me walk through an example implementing the desired promotion for splatting (but failing for list comprehension):

struct Foo{T}
    x::T
end

f1 = Foo(1)
f2 = Foo(1.0)

Note that at this stage these three cases all give the same Vector{Foo}:

[f1, f2]  # case1
[Any[f1, f2]...] # case 2
[x for x in Any[f1, f2]]  # case 3

However, the desired behavior, in analogy with the first Number example, is that the first two cases should give a Vector{Foo{Float64}} by promoting the elements, and the last case should give a Vector{Foo{Real}} leaving the elements untouched. To achieve the right behavior for case 1 & 2, I implement promote_rule and convert methods (following examples given in the docs):

Base.promote_rule(::Type{Foo{T}}, ::Type{Foo{S}}) where {T, S} =
    Foo{promote_type(T, S)}
Base.convert(::Type{F}, f::Foo) where {F <: Foo} = F(f)

Foo{T}(f::Foo) where {T} = Foo{T}(f.x)

Now case 1 and case 2 correctly promotes the elements to Vector{Foo{Float64}}, however, case 3 gives an error

ERROR: MethodError: Cannot `convert` an object of type Int64 to an object of type Foo{Int64}

This error came as a surprise. Why does it end up attempting to convert an Int to a Foo here? How do I get the desired result of case 3 resulting in a Vector{Foo{Real}} with elements Foo{Int64}(1), Foo{Float64}(1.0) in analogy with Numbers?

1 Like
2

The stacktrace is informative here, and the cause of the trouble is the call to convert, combined with the method Foo{T}(f::Foo) where {T} = Foo{T}(f.x):

julia> convert(Foo, f1)
ERROR: MethodError: Cannot `convert` an object of type Int64 to an object of type Foo{Int64}
Closest candidates are:
  convert(::Type{F}, ::Foo) where F<:Foo at REPL[17]:1
  convert(::Type{T}, ::T) where T at ~/programs/julia/julia-1.7.0/share/julia/base/essentials.jl:218
  Foo{T}(::Any) where T at REPL[11]:2
Stacktrace:
 [1] Foo{Foo{Int64}}(x::Int64)
   @ Main ./REPL[11]:2
 [2] Foo{Foo{Int64}}(f::Foo{Int64})
   @ Main ./REPL[20]:1
 [3] Foo(x::Foo{Int64})
   @ Main ./REPL[11]:2
 [4] convert(#unused#::Type{Foo}, f::Foo{Int64})
   @ Main ./REPL[17]:1
 [5] top-level scope
   @ REPL[32]:1

The way I would remedy the situation is by removing the Foo{T}(f::Foo) method and then changing the convert method to

Base.convert(::Type{Foo{T}}, f::Foo) where {T} = Foo(convert(T, f.x))
1 Like
3

Thanks Steven. That indeed leads to running code for all three cases. With that solution,“case 3”,

[x for x in Any[f1, f2]]

gives a Vector{Foo} which is the default behavior. I wonder how to get this to become a Vector{Foo{Real}} though? Is there some other function than promote_rule that has to have a method, something that specifies the first common supertype or something like that? It’s still opaque to me what happens under the hood for “case 3”.

4

The reason why the result is a Vector{Foo} is because Foo is the first common supertype of Foo{Int64} and Foo{Float64}. Importantly, Foo{Real} is a concrete type, and thus

julia> Foo{Int64} <: Foo{Real}
false

so a Vector{Foo{Real}} would have to be constructed with objects that are all of type Foo{Real}.

1 Like
5

Not quite; Foo{<:Real} is the first common supertype.

julia> (Foo{Int}, Foo{Float64}) .<: Foo{T} where T<:Real
(true, true)

julia> Foo{<:Real}[Foo(1), Foo(1.0)]
2-element Vector{Foo{<:Real}}:
 Foo{Int64}(1)
 Foo{Float64}(1.0)
6

That Foo{Int64} <: Foo{Real} is false always trips me up. Would it make sense for [x for x in Any[f1, f2]] to return a Vector{Foo{<:Real}}, and if so, what method(s) would I need to overload to make that happen?

7

This seems to do the trick:

julia> struct Foo{T}
           x::T
       end

julia> Base.promote_rule(::Type{Foo{T}}, ::Type{Foo{S}}) where {T, S} =
           Foo{promote_type(T, S)}

julia> Base.convert(::Type{Foo{T}}, f::Foo) where {T} = Foo(convert(T, f.x))

julia> Base.typejoin(::Type{Foo{T1}}, ::Type{Foo{T2}}) where {T1,T2} = Foo{T} where {T<:typejoin(T1, T2)}

julia> [x for x ∈ Any[Foo(1), Foo(1.0)]]
2-element Vector{Foo{T} where T<:Real}:
 Foo{Int64}(1)
 Foo{Float64}(1.0)
3 Likes
8

Not quite; Foo{<:Union{Int,Float64}} is the first common supertype :wink:

The point is, Foo is the only explicitly defined type in the tree, and what somebody means by “first common supertype” isn’t necessarily clear. For example, given Foo{A} and Foo{B}, how would Julia know to take the Union of A and B vs traversing the type trees of A and B to find a common supertype vs just focusing on the type tree of Foo? So I would argue that in some sense Foo is the first common supertype (of types explicitly defined in the type tree of Foo), and that developers have to tell Julia otherwise if they want a different definition of first common supertype, which seems to be accomplished with typejoin (which I didn’t know about until today).

1 Like
9

I stand corrected. :sweat_smile:

10

I also didn’t know about typejoin which solves the problem!

One small issue with the proposed solution though is that convert(Foo{T}, f::Foo) isn’t guaranteed to return a value of type Foo{T}. E.g. convert(Foo{Any}, Foo(1)) returns Foo{Int64}(1) which is not a Foo{Any}. So I suppose the correct thing would be to define Base.convert(::Type{Foo{T}}, f::Foo) where {T} = Foo{T}(f.x)

1 Like