CategoricalArrays forces OpenSpecFun_jll recompilation

Internals & Design
1

@tim.holy here’s a fresh invalidation issue. CategoricalArrays seems to invalidate OpenSpecFun_jll compilation.

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.0-DEV.1433 (2022-09-24)
 _/ |\__'_|_|_|\__'_|  |  Commit e6d99792e6* (0 days old master)
|__/                   |

julia> @time_imports using CategoricalArrays, OpenSpecFun_jll
      1.6 ms  DataAPI
     12.1 ms  Missings
      0.4 ms  Requires
      1.1 ms  Statistics
    205.0 ms  CategoricalArrays 13.56% compilation time (10% recompilation)
     38.1 ms  Preferences
      0.5 ms  JLLWrappers
    403.5 ms  OpenSpecFun_jll 99.80% compilation time (99% recompilation)

Unfortunately, JET.jl is having issues with master at the moment, so I cannot load SnoopCompile.jl to analyze further on master.

julia> using SnoopCompile
[ Info: Precompiling SnoopCompile [aa65fe97-06da-5843-b5b1-d5d13cad87d2]
ERROR: LoadError: MethodError: no method matching Core.Compiler.OptimizationParams(; inlining::Bool, inline_cost_threshold::Int64, inline_nonleaf_penalty::Int64, inline_tupleret_bonus::Int64, inline_error_path_cost::Int64, max_methods::Int64, tuple_splat::Int64, union_splitting::Int64)

Closest candidates are:
  Core.Compiler.OptimizationParams(; inlining, inline_cost_threshold, inline_nonleaf_penalty, inline_tupleret_bonus, inline_error_path_cost, tuple_splat, compilesig_invokes, trust_inference, assume_fatal_throw) got unsupported keyword arguments "max_methods", "union_splitting"
   @ Core compiler/types.jl:78

I can replicate the issue with Julia 1.8.1.

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.1 (2022-09-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @time_imports using CategoricalArrays, OpenSpecFun_jll
      4.6 ms  DataAPI
     31.7 ms  Missings
      0.9 ms  Requires
    206.2 ms  CategoricalArrays 14.67% compilation time (6% recompilation)
     26.5 ms  Preferences
      0.5 ms  JLLWrappers
      0.4 ms  CompilerSupportLibraries_jll
    481.3 ms  OpenSpecFun_jll 99.86% compilation time (99% recompilation)

using SnoopCompileCore
invalidations = @snoopr using CategoricalArrays, OpenSpecFun_jll
trees = invalidation_trees(invalidations)

julia> trees = invalidation_trees(invalidations)
14-element Vector{SnoopCompile.MethodInvalidations}:
 inserting insert!(A::CategoricalVector{>:Missing}, i::Integer, v::Missing) in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/missingarray.jl:30 invalidated:
   mt_backedges: 1: signature Tuple{typeof(insert!), Any, Any, Any} triggered MethodInstance for InteractiveUtils.gen_call_with_extracted_types(::Module, ::Function, ::Expr, ::Vector{Expr}) (0 children)
                 2: signature Tuple{typeof(insert!), Any, Any, Any} triggered MethodInstance for InteractiveUtils.gen_call_with_extracted_types(::Module, ::typeof(Base.return_types), ::Expr, ::Vector{Expr}) (0 children)

 inserting (::Base.var"#sort!##kw")(::Any, ::typeof(sort!), v::CategoricalVector) in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/array.jl:1088 invalidated:
   mt_backedges: 1: signature Tuple{Base.var"#sort!##kw", NamedTuple{(:by,), Tuple{typeof(identity)}}, typeof(sort!), Any} triggered MethodInstance for TOML.Internals.Printer.var"#_print#10"(::Int64, ::Bool, ::Bool, ::typeof(identity), ::typeof(TOML.Internals.Printer._print), ::Nothing, ::IOStream, ::AbstractDict, ::Vector{String}) (0 children)

 inserting similar(A::Matrix, ::Type{CategoricalValue{T}}, dims::Tuple{Vararg{Int64, N}}) where {T, N} in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/array.jl:675 invalidated:
   backedges: 1: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Array, ::Type, ::Tuple{Int64}) (2 children)

 inserting similar(A::Matrix, ::Type{CategoricalValue{T, R}}, dims::Tuple{Vararg{Int64, N}}) where {T, R, N} in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/array.jl:672 invalidated:
   backedges: 1: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Array, ::DataType, ::Tuple{Int64}) (2 children)

 inserting similar(A::AbstractRange, ::Type{CategoricalValue{T, R}}, dims::Tuple{Vararg{Int64, N}}) where {T, R, N} in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/array.jl:672 invalidated:
   backedges: 1: superseding similar(a::AbstractArray, ::Type{T}, dims::Tuple{Vararg{Int64, N}}) where {T, N} in Base at abstractarray.jl:806 with MethodInstance for similar(::UnitRange{Int64}, ::DataType, ::Tuple{Int64}) (3 children)

 inserting similar(A::AbstractRange, ::Type{Union{Missing, CategoricalValue{T}}}, dims::Tuple{Vararg{Int64, N}}) where {T, N} in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/array.jl:681 invalidated:
   backedges: 1: superseding similar(a::AbstractArray, ::Type{T}, dims::Tuple{Vararg{Int64, N}}) where {T, N} in Base at abstractarray.jl:806 with MethodInstance for similar(::UnitRange{Int64}, ::Type, ::Tuple{Int64}) (3 children)
              2: superseding similar(a::AbstractArray, ::Type{T}, dims::Tuple{Vararg{Int64, N}}) where {T, N} in Base at abstractarray.jl:806 with MethodInstance for similar(::UnitRange{Int64}, ::Type, ::Tuple{Int64}) (5 children)

 inserting similar(A::Vector, ::Type{CategoricalValue{T, R}}, dims::Tuple{Vararg{Int64, N}}) where {T, R, N} in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/array.jl:672 invalidated:
   backedges:  1: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Vector{SubString{String}}}, ::DataType, ::Tuple{Int64}) (1 children)
               2: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{String}, ::DataType, ::Tuple{Int64}) (1 children)
               3: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Pair{DataType, Function}}, ::DataType, ::Tuple{Int64}) (1 children)
               4: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Any}, ::DataType, ::Tuple{Int64}) (1 children)
               5: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Dict{Any, Any}}, ::DataType, ::Tuple{Int64}) (1 children)
               6: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Dict{Char, Any}}, ::DataType, ::Tuple{Int64}) (1 children)
               7: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{SubString{String}}, ::DataType, ::Tuple{Int64}) (1 children)
               8: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Symbol}, ::DataType, ::Tuple{Int64}) (1 children)
               9: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{LibGit2.FetchHead}, ::DataType, ::Tuple{Int64}) (1 children)
              10: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{LibGit2.GitAnnotated}, ::DataType, ::Tuple{Int64}) (1 children)
              11: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Set{Int64}}, ::DataType, ::Tuple{Int64}) (1 children)
              12: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Vector{Pkg.Resolve.FieldValue}}, ::DataType, ::Tuple{Int64}) (1 children)
              13: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Tuple{Base.UUID, String, String, VersionNumber}}, ::DataType, ::Tuple{Int64}) (1 children)
              14: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Base.PkgId}, ::DataType, ::Tuple{Int64}) (1 children)
              15: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Pkg.REPLMode.Option}, ::DataType, ::Tuple{Int64}) (1 children)
              16: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Pkg.REPLMode.Statement}, ::DataType, ::Tuple{Int64}) (1 children)
              17: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Base.StackTraces.StackFrame}, ::DataType, ::Tuple{Int64}) (1 children)
              18: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Tuple{Base.StackTraces.StackFrame, Int64}}, ::DataType, ::Tuple{Int64}) (1 children)
              19: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{REPL.REPLCompletions.Completion}, ::DataType, ::Tuple{Int64}) (1 children)
              20: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Tuple{Int64, Int64}}, ::DataType, ::Tuple{Int64}) (1 children)
              21: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Pkg.REPLMode.OptionSpec}, ::DataType, ::Tuple{Int64}) (1 children)

 inserting convert(::Type{S}, x::CategoricalValue) where S<:Union{AbstractChar, AbstractString, Number} in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/value.jl:92 invalidated:
   mt_backedges: 1: signature Tuple{typeof(convert), Type{SubString{String}}, Any} triggered MethodInstance for push!(::Vector{SubString{String}}, ::Any) (5 children)
                 2: signature Tuple{typeof(convert), Type{Union{Bool, String}}, Any} triggered MethodInstance for setindex!(::Dict{String, Union{Bool, String}}, ::Any, ::String) (9 children)
   backedges: 1: superseding convert(::Type{Union{}}, x) in Base at essentials.jl:213 with MethodInstance for convert(::Core.TypeofBottom, ::Any) (13 children)

 inserting similar(A::Vector, ::Type{CategoricalValue{T}}) where T in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/array.jl:675 invalidated:
   backedges: 1: superseding similar(a::Vector{T}, S::Type) where T in Base at array.jl:375 with MethodInstance for similar(::Vector{Tuple{Symbol, Symbol}}, ::Type) (3 children)
              2: superseding similar(a::Vector{T}, S::Type) where T in Base at array.jl:375 with MethodInstance for similar(::Vector{UnionAll}, ::Type) (5 children)
              3: superseding similar(a::Vector{T}, S::Type) where T in Base at array.jl:375 with MethodInstance for similar(::Vector, ::Type) (128 children)
   10 mt_cache

 inserting similar(A::Vector, ::Type{CategoricalValue{T}}, dims::Tuple{Vararg{Int64, N}}) where {T, N} in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/array.jl:675 invalidated:
   mt_backedges: 1: signature Tuple{typeof(similar), Vector, Any, Tuple{Int64}} triggered MethodInstance for similar(::Vector, ::Tuple{Base.OneTo{Int64}}) (2 children)
   backedges:  1: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{UInt32}, ::Type, ::Tuple{Int64}) (1 children)
               2: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector, ::Type, ::Tuple{Int64}) (1 children)
               3: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Vector{SubString{String}}}, ::Type, ::Tuple{Int64}) (8 children)
               4: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Pair{DataType, Function}}, ::Type, ::Tuple{Int64}) (8 children)
               5: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{SubString{String}}, ::Type, ::Tuple{Int64}) (8 children)
               6: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Dict{Any, Any}}, ::Type, ::Tuple{Int64}) (8 children)
               7: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Dict{Char, Any}}, ::Type, ::Tuple{Int64}) (8 children)
               8: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Symbol}, ::Type, ::Tuple{Int64}) (8 children)
               9: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{LibGit2.FetchHead}, ::Type, ::Tuple{Int64}) (8 children)
              10: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{LibGit2.GitAnnotated}, ::Type, ::Tuple{Int64}) (8 children)
              11: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Set{Int64}}, ::Type, ::Tuple{Int64}) (8 children)
              12: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Vector{Pkg.Resolve.FieldValue}}, ::Type, ::Tuple{Int64}) (8 children)
              13: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Tuple{Base.UUID, String, String, VersionNumber}}, ::Type, ::Tuple{Int64}) (8 children)
              14: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Base.PkgId}, ::Type, ::Tuple{Int64}) (8 children)
              15: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Pkg.REPLMode.Option}, ::Type, ::Tuple{Int64}) (8 children)
              16: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Pkg.REPLMode.Statement}, ::Type, ::Tuple{Int64}) (8 children)
              17: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Base.StackTraces.StackFrame}, ::Type, ::Tuple{Int64}) (8 children)
              18: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Tuple{Base.StackTraces.StackFrame, Int64}}, ::Type, ::Tuple{Int64}) (8 children)
              19: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{REPL.REPLCompletions.Completion}, ::Type, ::Tuple{Int64}) (8 children)
              20: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Tuple{Int64, Int64}}, ::Type, ::Tuple{Int64}) (8 children)
              21: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Pkg.REPLMode.OptionSpec}, ::Type, ::Tuple{Int64}) (8 children)
              22: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{String}, ::Type, ::Tuple{Int64}) (9 children)
              23: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Any}, ::Type, ::Tuple{Int64}) (23 children)
   30 mt_cache

 inserting isequal(x::Union{AbstractChar, AbstractString, Number}, y::CategoricalValue) in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/value.jl:142 invalidated:
   backedges: 1: superseding isequal(x, y) in Base at operators.jl:140 with MethodInstance for isequal(::Int64, ::Any) (3 children)
              2: superseding isequal(x, y) in Base at operators.jl:140 with MethodInstance for isequal(::Char, ::Any) (44 children)
              3: superseding isequal(x, y) in Base at operators.jl:140 with MethodInstance for isequal(::String, ::Any) (241 children)

 inserting isequal(x::CategoricalValue, y::Union{AbstractChar, AbstractString, Number}) in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/value.jl:141 invalidated:
   backedges: 1: superseding isequal(x, y) in Base at operators.jl:140 with MethodInstance for isequal(::Any, ::Char) (228 children)
              2: superseding isequal(x, y) in Base at operators.jl:140 with MethodInstance for isequal(::Any, ::String) (298 children)
   143 mt_cache

 inserting similar(::Type{T}, dims::Tuple{Vararg{Int64, N}} where N) where {U, T<:(Array{Union{Missing, CategoricalValue{U}}})} in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/array.jl:692 invalidated:
   backedges: 1: superseding similar(::Type{T}, dims::Tuple{Vararg{Int64, N}} where N) where T<:AbstractArray in Base at abstractarray.jl:841 with MethodInstance for similar(::Type{Array{_A}} where _A, ::Tuple{Int64}) (64 children)
              2: superseding similar(::Type{T}, dims::Tuple{Vararg{Int64, N}} where N) where T<:AbstractArray in Base at abstractarray.jl:841 with MethodInstance for similar(::Type{Vector{_A}} where _A, ::Tuple{Int64}) (569 children)

 inserting convert(::Type{Union{Nothing, S}}, x::CategoricalValue) where S<:Union{AbstractChar, AbstractString, Number} in CategoricalArrays at /home/mkitti/.julia/packages/CategoricalArrays/zg8z5/src/value.jl:96 invalidated:
   mt_backedges: 1: signature convert(::Type{Nothing}, x) in Base at some.jl:37 (formerly convert(::Type{Nothing}, x) in Base at some.jl:37) triggered MethodInstance for Preferences.load_preference(::Base.UUID, ::String, ::Nothing) (0 children)
   backedges: 1: superseding convert(::Type{Nothing}, x) in Base at some.jl:37 with MethodInstance for convert(::Type{Nothing}, ::Any) (2537 children)
   34 mt_cache

Breaking this down the issue seems to be convert(::Type{Nothing}, ::Any) being invalidated inserting convert(::Type{Union{Nothing, S}}, x::CategoricalValue) where S<:Union{AbstractChar, AbstractString, Number} in CategoricalArrays at CategoricalArrays/zg8z5/src/value.jl:96:

julia> trees[end].backedges[1]
MethodInstance for convert(::Type{Nothing}, ::Any) at depth 0 with 2537 children

julia> trees[end].backedges[1].children
3-element Vector{SnoopCompile.InstanceNode}:
 MethodInstance for setindex!(::IdDict{Any, Nothing}, ::Any, ::Any) at depth 1 with 2527 children
 MethodInstance for setindex!(::IdDict{Dict{String, Any}, Nothing}, ::Any, ::Any) at depth 1 with 7 children
 MethodInstance for SnoopCompileCore.__init__() at depth 1 with 0 children

julia> trees[end].backedges[1].children[1]
MethodInstance for setindex!(::IdDict{Any, Nothing}, ::Any, ::Any) at depth 1 with 2527 children

julia> trees[end].backedges[1].children[1].children
6-element Vector{SnoopCompile.InstanceNode}:
 MethodInstance for push!(::Base.IdSet{Any}, ::Any) at depth 2 with 988 children
 MethodInstance for push!(::Base.IdSet{Any}, ::TypeVar) at depth 2 with 1239 children
 MethodInstance for push!(::Base.IdSet{Any}, ::Vector{Union{}}) at depth 2 with 0 children
 MethodInstance for push!(::Base.IdSet{Any}, ::Vector) at depth 2 with 293 children
 MethodInstance for push!(::Base.IdSet{Any}, ::DataType) at depth 2 with 1 children
 MethodInstance for SnoopCompileCore.__init__() at depth 2 with 0 children
2

Why would Nothing be in that Union? That seems, wrong? When can a categorical value convert to a Nothing?

1 Like
3

The other problem seems to be the heavy use of Requires by CategoricalArrays? Some of the optional deps that are handled that way are fairly common, so at least for me at that point load times really get pretty bad… Almost 4s for just CategoricalArrays is painful in situations where I don’t even need that package at all and it is just pulled in by some other package…

1 Like
4

I removed convert to the Union and all the tests passed. I think this is meant to support an Array with nullable elements.

diff --git a/src/value.jl b/src/value.jl
index 7206698..f84cde2 100644
--- a/src/value.jl
+++ b/src/value.jl
@@ -93,8 +93,6 @@ Base.convert(::Type{S}, x::CategoricalValue) where {S <: SupportedTypes} =
     convert(S, unwrap(x))
 Base.convert(::Type{Union{S, Missing}}, x::CategoricalValue) where {S <: SupportedTypes} =
     convert(Union{S, Missing}, unwrap(x))
-Base.convert(::Type{Union{S, Nothing}}, x::CategoricalValue) where {S <: SupportedTypes} =
-    convert(Union{S, Nothing}, unwrap(x))
5

I’m pretty sure that line does nothing but cause invalidations.

julia> T = Union{Int, Nothing}
Union{Nothing, Int64}

julia> convert(T, 5.0)
5

julia> convert(T, nothing)

julia> ans |> isnothing
true
6

Yes, because the only non-erroring conversion to nothing is the one in Base, so :sweat_smile: many times these invalidation fixes are just improvements.

7

Good catch. I think these were needed at some point, but they appear to be redundant since at least Julia 1.0. I’ve made a PR to remove them: Drop redundant `convert(Union{T, Missing/Nothing}, x)` methods by nalimilan · Pull Request #396 · JuliaData/CategoricalArrays.jl · GitHub.

Tell that to packages that are not able to interact with CategoricalArrays without direct support from it. :wink:
But in what cases does loading CategoricalArrays take 4 seconds? Most of the dependencies used with Requires are interface packages and therefore fairly small.

8

I just tried it again, and I exaggerated with 4 seconds, it is 2.6 or so. Which still strikes me as way too long, though. On my system I get that with @time_imports using MimiGIVE (that is a public package out of my lab group):

2641.1 ms  CategoricalArrays 93.02% compilation time (92% recompilation)

If I just remove the entire __init__() function from CategoricalArrays, the load time for it drops to about 200ms:

197.3 ms  CategoricalArrays

The only thing inside the __init__() function is the Requires stuff.

I’m not a real expert on this, but my conclusion from reading around a bit was that Requires probably causes more pain in terms of load times than it solves…

The solution to this is probably to try to slim down all the deps that are currently handled via Requires enough so that they can become normal, always deps? I also experimented with that a bit, so with a version of CategoricalArrays that has JSON, RecipesBase, SentinelArrays and StructTypes as regular depdencies, I get:

 50.5 ms  JSON
121.5 ms  RecipesBase
348.6 ms  SentinelArrays 31.04% compilation time
 20.2 ms  StructTypes
168.1 ms  CategoricalArrays

My best guess is that one could fix something about the load times for SentinalArrays (UPDATE: this helps a bit), and maybe JSON.lower should be moved into a tiny package JSONBase that just defines JSONBase.lower? And maybe at that point CategoricalArrays could just have regular dependencies on all of these and ditch Requires? Don’t know what to do about RecipesBase, that seems (surprisingly) slow to load for an interface package.

1 Like
9

That’s not a good reason because this requires usage makes the functions of the package in the glue part not amenable to precompilation. Those small interface functions are exactly the kinds of functions that show up in a lot of places downstream and will this wait until runtime to compile (and invalidate any downstream usage of the interface functions that could possibly hit the dispatch). Possibly hit the dispatch meaning anything sufficiently not inferred as well. Since everything in DataFrames comes out of a Vector{Any} with function barriers, that’s exactly the kind of code that would drop precompilation with this. Requires.jl is really only safe for large dependency obscure features, not for interface functions.

10

The basic emerging strategy is to break up the package into subpackages along dependency lines.

  1. A CategoricalArraysCore package that is minimalist. Just declare your types and some constructors here. Other packages that want to build incorporate some compatibility with your types can depend on this. Ideally, the CategoricalArraysCore package has minimal dependencies.
  2. CategoricalArrays package depends on CategoricalArraysCore that provides the full interface.
  3. CategoricalArraysJSON package that loads both CategoricalArrays and JSON
  4. CategoricalArraysRecipesBase package that loads both CategoricalArrays and RecipesBase
  5. CategoricalSentinelArrays package that loads both CategoricalArrays and SentinelArrays
  6. CategoricalArraysStructTypes package that loads both CategoricalArrays and StructTypes
  7. CategoricalArraysFull package that loads 1-6.

I’ve mainly seen 1-2 implemented so far. In HDF5.jl, we’ve been breaking out the HDF5 filters into their own subpackages: HDF5.jl/filters at master · JuliaIO/HDF5.jl · GitHub . This is similar to 3-6.

The emerging convention is to be put these into a subdirectory package within a libs folder.

As part of the transition, you could use Requires.jl to just load the glue packages, but eventually you would want to transition direct dependents to either using a selection of the above. Library packages would mainly load CategoricalArraysCore. Terminal applications packages may load CategoricalArrays. End user projects may want to use CategoricalArraysFull for convenience.

This is all quite new and perhaps initially tedious, but eventually I think it will lead to a more robust ecosystem than we have today.

11

Having a separate package for each current use of Requires won’t work AFAICT. If a package loads CategoricalArraysCore/CategoricalArrays and returns a CategoricalArray, and then the user writes that to a .json file using JSON.jl, the result won’t be correct unless CategoricalArraysJSON is loaded. Same for plotting or sentinel arrays. Depending on the case, you’d get errors or plainly misleading behavior. So in the end CategoricalArrays would be useless.

What could work is having a CategoricalArraysCore interface package that JSON, StructTypes, RecipesBase and SentinelArrays depend on. But I’ll leave you convince their maintainers to take an additional dependency… especially given that these are interface packages supposed to be lightweight.

I think we could stop using Requires for interface packages (RecipesBase and StructTypes), but as @davidanthoff notes we would need an interface package for JSON as depending on it is absurd (I’m still not sure why JSON can’t use StructTypes like JSON3). For SentinelArrays, there’s no good solution AFAICT. See discussion at Trouble with `categorical(v)` when `v` is a `SentinelArrays.ChainedVector` · Issue #361 · JuliaData/CategoricalArrays.jl · GitHub, where Tim advised using Requires.

12

I agree.

Just one more observation here: if I remove the precompile statements from SentinelArrays (and use the PR I made that removes the package specific RNGs), load times drop to something like 20ms. Maybe there is another model to 1) move everything from SentinelArrays into SentinelArraysCore, except the precompile statements, 2) have SentinelArrays depend on SentinelArraysCore and then essentially only add the precompile statements. CategoricalArrays could then depend directly on SentinelArraysCore and that would add so little overhead that all might be well? It does seem weird and hacky, but maybe it could get the job done? I generally feel that if a small interface package loads really fast (like StructTypes), then it is probably fine to just take a direct dep on that.

1 Like
13

A year ago, I would have also recommended using Requires.jl. Given what we know now, I think it’s time to transition away from Requires.jl.

The transition is to move the glue code into their own glue packages, but still try to import them via Requires.jl while issuing warnings to explicitly load the glue package. Eventually, we stop loading the glue package packages automatically. Maybe we’ll have improved optional dependency support by then.

Ultimately, the goal is make code loading intentional and explicit.

14

Thanks for the v0.10.7 update by the way. Loading OpenSpecFun_jll dropped from 487 ms to 78 ms on Julia 1.8.1 for me. There seems to be a regression on Julia master (1.9) at the moment though.

Julia 1.8.1 with CategoricalArrays v0.10.6 (486.9 ms to load OpenSpecFun_jll)

julia> @time_imports using CategoricalArrays, OpenSpecFun_jll
      4.6 ms  DataAPI
     31.1 ms  Missings
      0.9 ms  Requires
    207.1 ms  CategoricalArrays 14.75% compilation time (6% recompilation)
     26.4 ms  Preferences
      0.5 ms  JLLWrappers
      0.4 ms  CompilerSupportLibraries_jll
    486.9 ms  OpenSpecFun_jll 99.85% compilation time (99% recompilation)

Julia 1.8.1 with CategoricalArrays v0.10.7 (77.5 ms to load OpenSpecFun_jll)

julia> @time_imports using CategoricalArrays, OpenSpecFun_jll
      4.7 ms  DataAPI
     31.0 ms  Missings
      0.9 ms  Requires
    205.3 ms  CategoricalArrays 14.54% compilation time (6% recompilation)
     28.5 ms  Preferences
      0.4 ms  JLLWrappers
      0.4 ms  CompilerSupportLibraries_jll
     77.5 ms  OpenSpecFun_jll 99.06% compilation time (96% recompilation)

julia> using Pkg

julia> pkg"st CategoricalArrays"
Status `~/.julia/environments/ross/Project.toml`
  [324d7699] CategoricalArrays v0.10.7

Julia 1.9 with CategorialArrays v0.10.7 (388.0 ms to load OpenSpecFun_jll)

julia> @time_imports using CategoricalArrays, OpenSpecFun_jll
      3.8 ms  DataAPI
     33.0 ms  Missings
      0.9 ms  Requires
      2.7 ms  Statistics
    207.6 ms  CategoricalArrays 13.42% compilation time (10% recompilation)
     38.7 ms  Preferences
      0.5 ms  JLLWrappers
    388.0 ms  OpenSpecFun_jll 99.79% compilation time (99% recompilation)

julia> versioninfo()
Julia Version 1.9.0-DEV.1433
Commit e6d99792e6* (2022-09-24 14:32 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
15

@mkitti can you think of a reason why this idea to move precompile statements into their own package might not work in practice?

16

I’m not sure if I completely understand the proposal and its merits.

My sense is that we probably should make the execution of precompile statements contingent on some condition. SnoopPrecompile already has implemented this mechanism:

https://timholy.github.io/SnoopCompile.jl/dev/snoop_pc/

Specifically that condition is if ccall(:jl_generating_output, Cint, ()) == 1 .

Also see Add `Base.is_serializing_code()` for guarding precompilation code by IanButterworth · Pull Request #45650 · JuliaLang/julia · GitHub for some of the triage discussion on the topic.

jl_generating_output essentially indicates if we are in the process of generating native code and caching it to disk. If we are not doing that, then there is no point in the precompilation statements.

I also notice that Requires.jl includes a isprecompiling() method for its internal use.

Would using isprecompiling() or @precompile_setup be sufficient for what you mean?

17

Fair enough :slight_smile: Here is a bit more what I was thinking:

If I compare the load time of this branch with one where I simply remove these two lines, I get about 200ms vs 20ms. In both cases I’m timing a scenario where everything is already precompiled. The first scenario is one where a lot of precompile info is stored, the second one where none is stored.My sense is that SentinelArrays with precompile statements is already guarding properly via the jl_generating_output construction that you mention.

So that suggests that simply loading the precompiled data increases the load time of SentinelArrays by an order of magnitude relative to a scenario where no precompile data is stored. To me this seems to be about load time of precompile data that exists.

So now we are in a pickle: if I just want to use CategoricalArrays (and not SentinelArrays), then I clearly don’t want to have these 180ms of extra load time of precompile information that I’m not going to need. But of course if I actually want to utilize SentinelArrays, then that precompile data is great to have and I probably do want it to be loaded, even if that increases package load times.

So the idea would be that SentinelArraysCore has all the code but zero precompile statements. CategoricalArrays takes a dependency on that, and that will only increase the load time by 20ms. Great, that is not a big price to pay for CategoricalArrays, and if a user just wants to use CategoricalArrays and not SentinelArrays, all is good. If someone plans to actually utilize SentinelArrays, then they load that package, and that will now trigger loading of all the precompiled data because the precompile statements are all in SentinelArrays. So that will be slower than just loading SentinelArraysCore, but presumably that is OK if someone is loading SentinelArrays because they actually intend to use the package, not just integrate with the interface there.

So, this is not about invalidations at all. The problem we seem to have here (after the invalidation and recompile stuff is fixed) is that some packages that have a good set of precompile statements will load slower because of that extra precompile data, and there are scenarios where that extra precompile data is actually not needed at all because all we want to do is integrate with some interface, not necessarily actually run stuff from that package.

18

Two quick points:

  • Re Requires.jl, the issue is not Requires itself (that mechanism is fine), the issue is that the way most of us use it causes the code not to be cached. But you could simply define a sub-package and make the @require @eval using MyRequiresSubPackage and then it would be fully compatible with precompilation.
  • Measuring load time without measuring TTFX is only half the story. Sure, if load time increases from 20ms to 200ms, that’s bad; but if removing the precompiles causes TTFX to go from 2s to 20s, that’s much worse.
4 Likes
19

Oh, and one more random comment: I’m not a huge fan of this :slight_smile: I’ve come across this a couple of times now, and it was way, way more difficult to find than just following the standard conventions of having one repo per package. There is also a fair bit of tooling that doesn’t know about this convention (e.g. the VS Code extension). And I also don’t understand how tagging for example works? In general, for choices like this I think there is huge value in just doing it the “normal” way that most folks are familiar with.

Not sure anyone will follow up on this thread, but if, it might make sense to split this into its own conversation, really not related to the main point here.

20

Yes, but the problem we are grappling with here is that we have packages where in one scenario the right trade-off is to pay the extra load time caused by a lot of precompiled data to have the lower TTFX, and in another (equally legitimate) scenario that becomes a real problem because for example CategoricalArrays only loads SentinelArrays to make sure things are integrated, but in this scenario no one will ever use any functionality from SentinelArrays. So then one pays the extra load time from the precompile data load, but never gets anything back in terms of TTFX because the functionality that is being precompiled never actually runs.

So that sounds interesting! I guess in that case the idea would be that we could define four new packages (one for each of the @require integration points that CategoricalArrays currently has), put all the code that is currently in these @require blocks into these packages, and then only load these new integration packages in the @require clause? If that would solve the problem, it might be the least involved to get going?

2 Likes