Constructors for structs with NamedTuple fields

New to Julia
, ,
1

I am creating a struct with a field that is a named tuple. I am getting different behavior when I define a constructor inside vs. outside the struct. Below, F1() runs fine but F2() gives a “UndefVarError: T not defined” error. I’m using Julia 1.9.

Can someone help me understand what’s going on? I suspect it has something to do with A<:Real What should I keep in mind when using NamedTuples as struct fields?

struct F1
    x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}
end
F1(x=(a=NaN,)) = F1(x)

struct F2
    x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}
    F2(x=(a=NaN,)) = new(x)
end

# F1() # okay
# F2() # gives "UndefVarError: `T` not defined" error
1 Like
2

You end up with the NamedTuple being parametrized - when the inner Tuple should be the target (original: NamedTuple{(:a,), Tuple{A}} where {A<:Real} vs. NamedTuple{(:a,), Tuple{A} where {A<:Real}}):

struct F2
    x::NamedTuple{(:a,), Tuple{A} where {A<:Real}}      
    F2(x=(a=NaN,)) = new(x)    
end

F2() # this will work

If you inspect the methods using methods(F2), you’ll only find F2() and F(x) methods - that means that for the inner constructor, an implicit conversion attempt needs to take place. However, if you inspect methods(F1) output, you’ll find an additional method defined: F1(x::NamedTuple{(:a,), Tuple{A}} where A<:Real) - and this will be able to dispatch properly without the need of additional conversion.

I cannot tell you precisely at this point why the inner constructor usage does not produce an identical set of methods as the outer constructor - I am curious myself and will dig more over the next days (or maybe somebody who knows can jump in the discussion and let us know).

3

It seems to work, but I don’t understand why. What does it mean to parametrize the NamedTuple versus the Tuple inside it? Are they imposing different constraints?

I think my original code ends up defining 3 F1 methods: one is the default/implicit constructor inside the struct, and two from the external constructor. Whereas for F2, the default inner constructor gets overridden.

4

When you don’t add an inner constructor, Julia defines some constructors for you. Your first struct definition actually expands to this:

struct F1
    x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}
    F1(x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}) = new(x)
end
F1(x) = F1(convert(NamedTuple{(:a,), Tuple{A}} where {A<:Real}, x))
F1(x=(a=NaN,)) = F1(x)

When you add an inner constructor you’re opting out of default constructors, so for F2 you’re missing out on the crucial convert call.

Other things to consider:

  • Tuple{A} where {A<:Real} is equivalent to Tuple{Real} because tuples are covariant, so you can save a lot of keystrokes and clutter there (and if tuples weren’t covariant, Tuple{<:Real} would also do the job)
  • You might want to consider adding the type parameter A to the struct instead, that way x will be concretely typed:
struct F1{A<:Real}
    x::NamedTuple{(:a,), Tuple{A}}
end
F1(x=(a=NaN,)) = F1(x)
  • The @NamedTuple macro lets you specify NamedTuple types in a way that’s simpler and cleaner:
struct F1{A<:Real}
    x::@NamedTuple{a::A}
end
F1(x=(a=NaN,)) = F1(x)
3 Likes
5

In this scenario the issue is the subsequent convert call. Consider this:

convert(NamedTuple{(:a,), Tuple{A} where {A<:Real}}, (a=NaN,)) # successful 
convert(NamedTuple{(:a,), Tuple{Real}}, (a=NaN,)) # successful
convert(NamedTuple{(:a,), Tuple{A}} where {A<:Real}, (a=NaN,)) # unsuccessful (also - your reported error)

T_orig = NamedTuple{(:a,), Tuple{A}} where {A<:Real}
T_new = NamedTuple{(:a,), Tuple{A} where {A<:Real}}

T_new <: T_orig # true

Now, from the documentation:

If any inner constructor method is defined, no default constructor method is provided: it is presumed that you have supplied yourself with all the inner constructors you need. The default constructor is equivalent to writing your own inner constructor method that takes all of the object’s fields as parameters (constrained to be of the correct type, if the corresponding field has a type), and passes them to new , returning the resulting object

So the only constraint that I see here is that convert that is called inside the inner fails with ERROR: UndefVarError: `T` not defined: you can see more details about this here. It seems there is a possibility that a certain convert fallback still doesn’t work properly in some corner cases (although the dispatch doesn’t fail). I still need to investigate this more.

cc: @jakobnissen - connected to this

1 Like
6

Thanks. I didn’t realize that the constructors are using convert behind the scenes. I’ll need to do more homework to understand what’s going on.

This was a minimum working example I created. The actual code I have is more complicated, so it’s possible I’ll have some follow up questions. Thanks for your help so far.

1 Like
7

Rather than changing data, a “convert” to an abstract type is more of a check whether the instance fits:

julia> convert(Union{Int, Float64}, 1)
1
julia> convert(Union{Int, Float64}, 2.3)
2.3

As you pointed out, the two following types are different, and one is concrete while the other is an abstract Union including the former:

julia> NamedTuple{(:a,), Tuple{A} where {A<:Real}} === NamedTuple{(:a,), Tuple{Real}}
true
julia> isconcretetype(NamedTuple{(:a,), Tuple{Real}})
true
julia> isconcretetype(NamedTuple{(:a,), Tuple{A}} where {A<:Real})
false

NamedTuple methods handled the convert from one concrete type to another (convert(T, x) seems to do T(x) sometimes), but not to an abstract Union. That issue about it did get patched on master, maybe it works there?

EDIT: this concrete/abstract stuff wouldn’t explain why F1 works.

1 Like
8

Thanks @Benny,

While you can avoid manual convert to abstract Union, this seems to happen in the scenario of the OP MWE implicitly:

struct F2
    x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}
    F2(x=(a=NaN,)) = new(x)
end
F2()

# ERROR: UndefVarError: `T` not defined
# Stacktrace:
# [1] convert(#unused#::Type{NamedTuple{(:a,), Tuple{A}} where A<:Real}, nt::NamedTuple{(:a,), #Tuple{Float64}})
#  @ Base ./namedtuple.jl:152
# ...

Also - you are right that the fix might work on master (only?), because the example used to showcase the issue still throws in 1.9.2:

convert(@NamedTuple{a::S} where S, (; a=1))

# ERROR: UndefVarError: `T` not defined
# Stacktrace:
# [1] convert(#unused#::Type{NamedTuple{(:a,), Tuple{S}} where S}, nt::NamedTuple{(:a,), #Tuple{Int64}})
#  @ Base ./namedtuple.jl:152
...
9

Now that I realize that the concrete-abstract thing doesn’t explain why F1 works, I have a different approach and it’s a little alarming. F2 is actually missing a method.

julia> methods(F1)
# 3 methods for type constructor:
[1] F1() in Main at REPL[2]:1
[2] F1(x::NamedTuple{(:a,), Tuple{A}} where A<:Real) in Main at REPL[1]:2
[3] F1(x) in Main at REPL[2]:1

julia> methods(F2)
# 2 methods for type constructor:
[1] F2() in Main at REPL[3]:3
[2] F2(x) in Main at REPL[3]:3

The documentation claims that the following types have equivalent methods, but it doesn’t pan out in a similar way:

julia> struct T1
         x::Int64
       end

julia> struct T2
         x::Int64
         T2(x) = new(x)
       end

julia> methods(T1) # both implicit inner, new in @code_warntype
# 2 methods for type constructor:
[1] T1(x::Int64) in Main at REPL[6]:2
[2] T1(x) in Main at REPL[6]:2

julia> methods(T2) # explicit inner
# 1 method for type constructor:
[1] T2(x) in Main at REPL[7]:3

With parameters it’s a bit more complicated so I’ll collapse it:

Example with type parameters 
julia> struct Point{T<:Real}
         x::T
         y::T
       end

julia> struct Point2{T<:Real}
         x::T
         y::T
         Point2{T}(x,y) where {T<:Real} = new(x,y)
       end

julia> methods(Point) # implicit outer(?) new not in @code_warntype
# 1 method for type constructor:
[1] Point(x::T, y::T) where T<:Real in Main at REPL[10]:2

julia> methods(Point{Real}) # implicit inner
# 1 method for type constructor:
[1] Point{T}(x, y) where T<:Real in Main at REPL[10]:2

julia> methods(Point2)
# 0 methods for type constructor

julia> methods(Point2{Real}) # explicit inner
# 1 method for type constructor:
[1] Point2{T}(x, y) where T<:Real in Main at REPL[11]:4

I can’t tell which of these methods are inner vs outer, but it’s clear the documented methods aren’t actually equivalent to what is implicitly generated, it consistently misses the implicit method with arguments sharing annotations with the fields.

The thing is, I can’t figure out what that implicit method does, it’s not just new(x).

struct F22 # based on F2
    x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}
    F22(x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}) = new(x)
    F22(x=(a=NaN,)) = F22(x) # dispatches to first inner method
end

F22() # T not defined

struct F12 # based on F1
    x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}
    F12(x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}) = new(x)
end
F12(x=(a=NaN,)) = F12(x)

F12() # T not defined
2 Likes
10

I also noticed the missing method:

However, I am still clueless about the reason behind inner constructor behavior (relative to the generated methods). And I don’t have time to dig deeper right now.

11

Looks like there are two issues.

(1) the default struct constructors doing things I personally don’t expect them to

struct A
    x::Integer
end
methods(A) # 2 methods defined -> A(x::Integer) and A(x)
A(2.0) # does not throw an error ?
A('a') # does not throw an error ???
A(2.2) # calls A(x::Float64) which not explicitly listed by `methods(A)`; seems to call `convert` and fail.

(2) issue with NamedTuples. I spent some time exploring this one. I created a NamedTuple instance and tried to understand its type.

Here are some things I learned:

  • type1 (see below) is not a concrete type, as @Benny pointed out. But it’s not an abstract type either. I don’t know if that’s legal.
  • Even though the type assertion instance::type1 holds, convert(type1, instance) fails. I don’t know if this is legal either.
  • f1 is a method that is defined only for a type1 argument. And it correctly dispatches on instance. But it looks like internal struct constructors are still trying to convert an argument that satisfies type assertions (see struct F below).
instance = (a=NaN,)
type1 = NamedTuple{(:a,), Tuple{A}} where A<:Real # neither concrete nor abstract type
type2 = NamedTuple{(:a,), Tuple{A}  where A<:Real} # concrete type; === NamedTuple{(:a,), Tuple{Real}}
f1(x::type1) = true
f2(x::type2) = true

isconcretetype(type1) # false
isabstracttype(type1) # false
instance isa type1 # true
instance::type1 # no error
convert(type1, instance) # ERROR: UndefVarError: `T` not defined
f1(instance) # works; no error

isconcretetype(type2) # true
isabstracttype(type2) # false
instance isa type2 # false
instance::type2 # TypeError:
convert(type2, instance) # works
f2(instance) # MethodError: no matching method found

# internal constructor tries to `convert` arguments that satisfy type assertion
#   even when you restrict the argument type of the constructor.
struct F
    x::type1
    F(x::type1=instance) = new(x)
end
F() # UndefVarError: `T` not defined
12

It’s just convert trying to make the inputs fit into the field

julia> Integer(2.0), Integer('a')
(2, 0x00000061)

julia> Integer(2.2)
ERROR: InexactError: Int64(2.2)

That’s just the call’s signature, not a method’s signature.

It is an abstract type, just not isabstracttype, which is a method that checks if a type was declared with the abstract type ... end block. Use !isconcretetype to check for abstract types.

Yep, that’s not ideal.

By the way, you can make an instance of type2, using the constructor with the explicit parameters instead of inferring them:

julia> instance2 = type2(instance) # converting concrete NamedTuples works

julia> f2(instance2)
true
2 Likes
13

The stacktrace is a little confusing in this regard. The function argument annotations in the stacktrace indicate the type of object that was passed, whether or not a method for that argument type exists. Use @which to determine which method is actually called:

julia> struct A
           x::Integer
       end

julia> @which A(2.0)
A(x)
     @ Main REPL[21]:2

julia> @which A('a')
A(x)
     @ Main REPL[21]:2

julia> @which A(2.2)
A(x)
     @ Main REPL[21]:2

In the first two cases, the A(x) constructor was able to convert the input to an Integer. In the third case, it was not able to.

2 Likes
14

Thanks @Benny and @CameronBieganek.

I understand now the default constructor is doing a convert. I just didn’t expect that to be the default.

I didn’t know that that’s how isabstracttype worked. I also didn’t know there’s a difference between call signature and method signature (not a computer science person).

I’m still left without a solution to (what I think is) my problem: How do I create a struct F so that it only takes type1 (abstract type) arguments? In the case below, the type assertion is valid but the constructor still tries to convert the argument, running into an error.

Or should I just redefine my struct so it uses only concrete types? (I know this is the official suggestion, but I thought that was because of optimization concerns.)

struct F
    x::type1
    F(x::type1=instance) = new(x)
end

EDIT: For now, I think I’ll define my own struct that works like a NamedTuple.

15

@sadish-d, do you need to stick with an inner constructor? Because the outer one seemed to generate the appropriate dispatch, and the convert step was avoided.

Also - you can consider the @kwdef route, which also works nicely (because the macro generates the appropriate methods).

I am using the exact type from OP:

@kwdef struct F2
    x::NamedTuple{(:a,), Tuple{A}} where {A<:Real} = (a=NaN,)    
end

F2() # works
F2((a=6,)) # works
F2(x=(a=6,)) # works
16

It’s more that we don’t know what the implicit constructor is doing. It’s not needed in this case but we should probably figure this out for general knowledge. Another example:

julia> struct F1
           x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}
       end

julia> methods(F1) # I have no idea how F1(x) would be explicitly defined
# 2 methods for type constructor:
[1] F1(x::NamedTuple{(:a,), Tuple{A}} where A<:Real) in Main at REPL[1]:2
[2] F1(x) in Main at REPL[1]:2

julia> @code_warntype F1((a=NaN,))
MethodInstance for F1(::NamedTuple{(:a,), Tuple{Float64}})
  from F1(x::NamedTuple{(:a,), Tuple{A}} where A<:Real) in Main at REPL[1]:2
Arguments
  #self#::Core.Const(F1)
  x::NamedTuple{(:a,), Tuple{Float64}}
Body::F1
1 ─ %1 = %new(Main.F1, x)::Core.PartialStruct(F1, Any[NamedTuple{(:a,), Tuple{Float64}}])
└──      return %1

julia> struct F12 # based on F1
           x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}
           F12(x::NamedTuple{(:a,), Tuple{A}} where {A<:Real}) = new(x)
       end

julia> methods(F12)
# 1 method for type constructor:
[1] F12(x::NamedTuple{(:a,), Tuple{A}} where A<:Real) in Main at REPL[7]:3

julia> @code_warntype F12((a=NaN,)) # convert throws T not defined
MethodInstance for F12(::NamedTuple{(:a,), Tuple{Float64}})
  from F12(x::NamedTuple{(:a,), Tuple{A}} where A<:Real) in Main at REPL[7]:3
Arguments
  #self#::Core.Const(F12)
  x::NamedTuple{(:a,), Tuple{Float64}}
Body::F12
1 ─ %1 = Main.F12::Core.Const(F12)
│   %2 = Core.fieldtype(%1, 1)::Type{NamedTuple{(:a,), Tuple{A}} where A<:Real}
│   %3 = Base.convert(%2, x)::NamedTuple{(:a,), _A} where _A<:(Union{Tuple{A1}, Tuple{A}} where {A<:Real, A1<:Real})
│   %4 = %new(%1, %3)::F12
└──      return %4

I just wrote new(x), why is it so different

1 Like
17

I fully agree with that. It also bothers me - especially because the user-defined inner constructor fails even if you add the argument types and “force” the methods definitions.

On the other hand, it is clear that the inner constructor is/should be more complex: since that is the way to achieving incomplete initialization (I am not sure if that has something to do with the behavior we are discussing).

18

I’ve never really used @kwdef. But it also seems to define extra methods. In the example below, the method A(x) without type annotations is defined. That method resorts to convert. So if I want the constructor methods of A to only ever accept Integer arguments, I can not use the definition below.

julia> @kwdef struct A
       x::Integer=0
       end
A

julia> methods(A)
# 3 methods for type constructor:
 [1] A(; x)
     @ util.jl:567
 [2] A(x::Integer)
     @ REPL[3]:2
 [3] A(x)
     @ REPL[3]:2

I’ve found a work around to my problem (avoiding using NamedTuples entirely for now). But if people much smarter than I are finding it hard to get the constructors to do what they want, maybe there is a problem.

19

If you really have a problem with a method, you can delete it:

@kwdef struct A
    x::Integer = 0
end

A('a') # works this time

methods(A)
# 3 methods for type constructor:
# [1] A(; x)
# @ util.jl:567
# [2] A(x::Integer)
# @ ~/code/playtest/innerconstr.jl:20
# [3] A(x)
# @ ~/code/playtest/innerconstr.jl:20

Base.delete_method(methods(A)[3])

methods(A)
# 2 methods for type constructor:
# [1] A(; x)
# @ util.jl:567
# [2] A(x::Integer)
# @ ~/code/playtest/innerconstr.jl:20

A('a')
# ERROR: MethodError: no method matching A(::Char)
# Closest candidates are:
#   A(; x)
#    @ Main util.jl:567
#   A(::Integer)
#    @ Main ~/code/playtest/innerconstr.jl:20

You can also delete the keyword one and only keep the A(x::Integer) if you want.

I never found myself in need of method deletion. I find it easier to enforce the type at the calling site (if you build the struct using some function output of variable type).

1 Like
20

Oh cool. That’s useful to know.

1 Like