Slowness of fieldnames and propertynames

So I’ve seen disucssions of this pop up a few times, but I figure it might be good to have a more recent thread here on discourse that we can point people to (and maybe it can motivate a PR / issue to julia)

The problem:

julia> struct Foo
           f1::Int
           f2::Int
           f3::Int
       end

julia> @btime fieldnames($Foo)
  396.512 ns (1 allocation: 32 bytes)
(:f1, :f2, :f3)

julia> @btime propertynames($(Foo(1, 2, 3)))
  394.821 ns (1 allocation: 32 bytes)
(:f1, :f2, :f3)

There are various usecases where one might want to reflect on these things without paying such a hefty performance penalty.

To my naive eye, it seems like at the very least the fieldnames is information that can be statically known and reflected on. It’s not like we are allowed to redefine the fieldnames:

julia> struct Foo
           f1::Int
           f2::Int
           some_other_fieldname::Int
       end
ERROR: invalid redefinition of constant Foo

In this thread: Static fieldnames, the OP was told to just use a generated function to manually hoist the reflection to compiletime, but that leaves me a little puzzled.

If it’s okay to reflect on this sort of thing inside a generated function body, why is it not okay for the compiler to just make this a tfunc or whatever and give us these results as compiletime constants?


Edit: I somehow missed Inferrability of `fieldnames` Β· Issue #29100 Β· JuliaLang/julia Β· GitHub when I was searching around, so there is a somewhat up to date discussion of this.

4 Likes

I think this might be a related issue: I wanted to convert a struct into a NamedTuple (with the keys being the field names of the struct), and while it seems like the compiler has all the info it needs with

named_tuple(obj::T) where {T} = NamedTuple{fieldnames(T), Tuple{fieldtypes(T)...}}(ntuple(i -> getfield(obj, i), fieldcount(T)))

it is much slower than the equivalent generated function

@generated function named_tuple_gen(obj)
    NT = NamedTuple{fieldnames(obj), Tuple{fieldtypes(obj)...}}
    return :($NT(tuple($((:(getfield(obj, $i)) for i in 1:fieldcount(obj))...))))
end

which I got working by modifying https://discourse.julialang.org/t/static-fieldnames/5619/2; thanks for the link! (I actually had to expand it out as

@generated function  named_tuple_gen(obj::T) where {T}
    NT = NamedTuple{fieldnames(obj), Tuple{fieldtypes(obj)...}}
    return :(
                $NT(
                    tuple(
                        $(
                            (
                                :(getfield(obj, $i)) for i in 1:fieldcount(obj)
                            )...
                        )
                    )
                )
            )
end

to see what’s going on better; VSCode makes little vertical lines which makes it a bit easier to parse the paren alignment than here, though).

It seemed like I could recover some of the performance without generated functions if I wanted it just for a specific type T by doing

const NT = NamedTuple{fieldnames(T), Tuple{fieldtypes(T)...}}
T_as_NT(obj::T) = NT(ntuple(i -> getfield(obj, i), fieldcount(T)))

which at least lets it infer the output type correctly, though if there are more than 10 fields it can’t infer the output of ntuple. (It seems basically as fast as the generated function for fewer than 10 fields; the cutoff of course is from julia/ntuple.jl at a319ae45a15fb005c7d7e29e3f3a62f27e18e3a6 Β· JuliaLang/julia Β· GitHub).

Example with 11 fields
julia> using BenchmarkTools

julia> named_tuple(obj::T) where {T} = NamedTuple{fieldnames(T), Tuple{fieldtypes(T)...}}(ntuple(i -> getfield(obj, i), fieldcount(T)))
named_tuple (generic function with 1 method)

julia> @generated function  named_tuple_gen(obj::T) where {T}
           NT = NamedTuple{fieldnames(obj), Tuple{fieldtypes(obj)...}}
           return :($NT(tuple($((:(getfield(obj, $i)) for i in 1:fieldcount(obj))...))))
       end
named_tuple_gen (generic function with 1 method)

julia> struct ManyBools
           x1::Bool
           x2::Bool
           x3::Bool
           x4::Bool
           x5::Bool
           x6::Bool
           x7::Bool
           x8::Bool
           x9::Bool
           x10::Bool
           x11::Bool
       end

julia> const NT = NamedTuple{fieldnames(ManyBools), Tuple{fieldtypes(ManyBools)...}}
NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9, :x10, :x11), NTuple{11, Bool}}

julia> ManyBools_as_NT(obj::ManyBools) = NT(ntuple(i -> getfield(obj, i), fieldcount(ManyBools)))
ManyBools_as_NT (generic function with 1 method)

julia> obj = ManyBools(rand(Bool, 11)...)
ManyBools(false, false, false, false, false, true, false, true, false, false, true)

julia> @benchmark named_tuple($(Ref(obj))[])

BenchmarkTools.Trial: 
  memory estimate:  528 bytes
  allocs estimate:  6
  --------------
  minimum time:     2.768 ΞΌs (0.00% GC)
  median time:      2.806 ΞΌs (0.00% GC)
  mean time:        2.856 ΞΌs (0.75% GC)
  maximum time:     218.773 ΞΌs (98.07% GC)
  --------------
  samples:          10000
  evals/sample:     9

julia> @benchmark named_tuple_gen($(Ref(obj))[])
BenchmarkTools.Trial: 
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     2.125 ns (0.00% GC)
  median time:      2.250 ns (0.00% GC)
  mean time:        2.257 ns (0.00% GC)
  maximum time:     23.250 ns (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     1000

julia> @benchmark ManyBools_as_NT($(Ref(obj))[])
BenchmarkTools.Trial: 
  memory estimate:  128 bytes
  allocs estimate:  2
  --------------
  minimum time:     351.827 ns (0.00% GC)
  median time:      357.280 ns (0.00% GC)
  mean time:        368.931 ns (1.09% GC)
  maximum time:     9.198 ΞΌs (95.65% GC)
  --------------
  samples:          10000
  evals/sample:     214

julia> @code_warntype named_tuple(obj)
Variables
  #self#::Core.Const(named_tuple)
  obj::ManyBools
  #5::var"#5#6"{ManyBools}

Body::NamedTuple
1 ─ %1  = Main.fieldnames($(Expr(:static_parameter, 1)))::Tuple{Vararg{Symbol, N} where N}
β”‚   %2  = Core.tuple(Main.Tuple)::Core.Const((Tuple,))
β”‚   %3  = Main.fieldtypes($(Expr(:static_parameter, 1)))::Tuple
β”‚   %4  = Core._apply_iterate(Base.iterate, Core.apply_type, %2, %3)::Type
β”‚   %5  = Core.apply_type(Main.NamedTuple, %1, %4)::Type{NamedTuple{_A, _B}} where {_A, _B}
β”‚   %6  = Main.:(var"#5#6")::Core.Const(var"#5#6")
β”‚   %7  = Core.typeof(obj)::Core.Const(ManyBools)
β”‚   %8  = Core.apply_type(%6, %7)::Core.Const(var"#5#6"{ManyBools})
β”‚         (#5 = %new(%8, obj))
β”‚   %10 = #5::var"#5#6"{ManyBools}
β”‚   %11 = Main.fieldcount($(Expr(:static_parameter, 1)))::Core.Const(11)
β”‚   %12 = Main.ntuple(%10, %11)::Tuple{Vararg{Bool, N} where N}
β”‚   %13 = (%5)(%12)::NamedTuple
└──       return %13

julia> @code_warntype named_tuple_gen(obj)
Variables
  #self#::Core.Const(named_tuple_gen)
  obj::ManyBools

Body::NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9, :x10, :x11), NTuple{11, Bool}}
1 ─ %1  = Main.getfield(obj, 1)::Bool
β”‚   %2  = Main.getfield(obj, 2)::Bool
β”‚   %3  = Main.getfield(obj, 3)::Bool
β”‚   %4  = Main.getfield(obj, 4)::Bool
β”‚   %5  = Main.getfield(obj, 5)::Bool
β”‚   %6  = Main.getfield(obj, 6)::Bool
β”‚   %7  = Main.getfield(obj, 7)::Bool
β”‚   %8  = Main.getfield(obj, 8)::Bool
β”‚   %9  = Main.getfield(obj, 9)::Bool
β”‚   %10 = Main.getfield(obj, 10)::Bool
β”‚   %11 = Main.getfield(obj, 11)::Bool
β”‚   %12 = Main.tuple(%1, %2, %3, %4, %5, %6, %7, %8, %9, %10, %11)::NTuple{11, Bool}
β”‚   %13 = (NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9, :x10, :x11), NTuple{11, Bool}})(%12)::NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9, :x10, :x11), NTuple{11, Bool}}
└──       return %13

julia> @code_warntype ManyBools_as_NT(obj)
Variables
  #self#::Core.Const(ManyBools_as_NT)
  obj::ManyBools
  #10::var"#10#11"{ManyBools}

Body::NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9, :x10, :x11), NTuple{11, Bool}}
1 ─ %1 = Main.:(var"#10#11")::Core.Const(var"#10#11")
β”‚   %2 = Core.typeof(obj)::Core.Const(ManyBools)
β”‚   %3 = Core.apply_type(%1, %2)::Core.Const(var"#10#11"{ManyBools})
β”‚        (#10 = %new(%3, obj))
β”‚   %5 = #10::var"#10#11"{ManyBools}
β”‚   %6 = Main.fieldcount(Main.ManyBools)::Core.Const(11)
β”‚   %7 = Main.ntuple(%5, %6)::Tuple{Vararg{Bool, N} where N}
β”‚   %8 = Main.NT(%7)::NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9, :x10, :x11), NTuple{11, Bool}}
└──      return %8
Example with 9 fields
julia> using BenchmarkTools

julia> named_tuple(obj::T) where {T} = NamedTuple{fieldnames(T), Tuple{fieldtypes(T)...}}(ntuple(i -> getfield(obj, i), fieldcount(T)))
named_tuple (generic function with 1 method)

julia> @generated function  named_tuple_gen(obj::T) where {T}
           NT = NamedTuple{fieldnames(obj), Tuple{fieldtypes(obj)...}}
           return :($NT(tuple($((:(getfield(obj, $i)) for i in 1:fieldcount(obj))...))))
       end
named_tuple_gen (generic function with 1 method)

julia> struct FewerBools
           x1::Bool
           x2::Bool
           x3::Bool
           x4::Bool
           x5::Bool
           x6::Bool
           x7::Bool
           x8::Bool
           x9::Bool
       end

julia> const NT = NamedTuple{fieldnames(FewerBools), Tuple{fieldtypes(FewerBools)...}}
NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9), NTuple{9, Bool}}

julia> FewerBools_as_NT(obj::FewerBools) = NT(ntuple(i -> getfield(obj, i), fieldcount(FewerBools)))
FewerBools_as_NT (generic function with 1 method)

julia> obj = FewerBools(rand(Bool, 9)...)
FewerBools(false, true, false, true, true, true, false, true, false)

julia> @benchmark named_tuple($(Ref(obj))[])
BenchmarkTools.Trial: 
  memory estimate:  384 bytes
  allocs estimate:  5
  --------------
  minimum time:     2.083 ΞΌs (0.00% GC)
  median time:      2.120 ΞΌs (0.00% GC)
  mean time:        2.152 ΞΌs (0.00% GC)
  maximum time:     4.940 ΞΌs (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     9

julia> @benchmark named_tuple_gen($(Ref(obj))[])
BenchmarkTools.Trial: 
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     1.541 ns (0.00% GC)
  median time:      1.583 ns (0.00% GC)
  mean time:        1.590 ns (0.00% GC)
  maximum time:     5.791 ns (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     1000

julia> @benchmark FewerBools_as_NT($(Ref(obj))[])
BenchmarkTools.Trial: 
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     1.833 ns (0.00% GC)
  median time:      1.917 ns (0.00% GC)
  mean time:        1.948 ns (0.00% GC)
  maximum time:     15.917 ns (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     1000

julia> @code_warntype named_tuple(obj)
Variables
  #self#::Core.Const(named_tuple)
  obj::FewerBools
  #5::var"#5#6"{FewerBools}

Body::NamedTuple
1 ─ %1  = Main.fieldnames($(Expr(:static_parameter, 1)))::Tuple{Vararg{Symbol, N} where N}
β”‚   %2  = Core.tuple(Main.Tuple)::Core.Const((Tuple,))
β”‚   %3  = Main.fieldtypes($(Expr(:static_parameter, 1)))::Tuple
β”‚   %4  = Core._apply_iterate(Base.iterate, Core.apply_type, %2, %3)::Type
β”‚   %5  = Core.apply_type(Main.NamedTuple, %1, %4)::Type{NamedTuple{_A, _B}} where {_A, _B}
β”‚   %6  = Main.:(var"#5#6")::Core.Const(var"#5#6")
β”‚   %7  = Core.typeof(obj)::Core.Const(FewerBools)
β”‚   %8  = Core.apply_type(%6, %7)::Core.Const(var"#5#6"{FewerBools})
β”‚         (#5 = %new(%8, obj))
β”‚   %10 = #5::var"#5#6"{FewerBools}
β”‚   %11 = Main.fieldcount($(Expr(:static_parameter, 1)))::Core.Const(9)
β”‚   %12 = Main.ntuple(%10, %11)::NTuple{9, Bool}
β”‚   %13 = (%5)(%12)::NamedTuple
└──       return %13

julia> @code_warntype named_tuple_gen(obj)
Variables
  #self#::Core.Const(named_tuple_gen)
  obj::FewerBools

Body::NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9), NTuple{9, Bool}}
1 ─ %1  = Main.getfield(obj, 1)::Bool
β”‚   %2  = Main.getfield(obj, 2)::Bool
β”‚   %3  = Main.getfield(obj, 3)::Bool
β”‚   %4  = Main.getfield(obj, 4)::Bool
β”‚   %5  = Main.getfield(obj, 5)::Bool
β”‚   %6  = Main.getfield(obj, 6)::Bool
β”‚   %7  = Main.getfield(obj, 7)::Bool
β”‚   %8  = Main.getfield(obj, 8)::Bool
β”‚   %9  = Main.getfield(obj, 9)::Bool
β”‚   %10 = Main.tuple(%1, %2, %3, %4, %5, %6, %7, %8, %9)::NTuple{9, Bool}
β”‚   %11 = (NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9), NTuple{9, Bool}})(%10)::NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9), NTuple{9, Bool}}
└──       return %11

julia> @code_warntype FewerBools_as_NT(obj)
Variables
  #self#::Core.Const(FewerBools_as_NT)
  obj::FewerBools
  #10::var"#10#11"{FewerBools}

Body::NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9), NTuple{9, Bool}}
1 ─ %1 = Main.:(var"#10#11")::Core.Const(var"#10#11")
β”‚   %2 = Core.typeof(obj)::Core.Const(FewerBools)
β”‚   %3 = Core.apply_type(%1, %2)::Core.Const(var"#10#11"{FewerBools})
β”‚        (#10 = %new(%3, obj))
β”‚   %5 = #10::var"#10#11"{FewerBools}
β”‚   %6 = Main.fieldcount(Main.FewerBools)::Core.Const(9)
β”‚   %7 = Main.ntuple(%5, %6)::NTuple{9, Bool}
β”‚   %8 = Main.NT(%7)::NamedTuple{(:x1, :x2, :x3, :x4, :x5, :x6, :x7, :x8, :x9), NTuple{9, Bool}}
└──      return %8
2 Likes