Slow compile time with a lot of keyword arguments

General Usage
1

Hi!

My package PrettyTables.jl started to have a lot of configuration possibilities, which are handle by keyword arguments like:

function _pt_text(io::IO, pinfo::PrintInfo;
                  border_crayon::Crayon = Crayon(),
                  header_crayon::Union{Crayon,Vector{Crayon}} = Crayon(bold = true),
                  subheader_crayon::Union{Crayon,Vector{Crayon}} = Crayon(foreground = :dark_gray),
                  rownum_header_crayon::Crayon = Crayon(bold = true),
                  text_crayon::Crayon = Crayon(),
                  autowrap::Bool = false,
                  body_hlines::Vector{Int} = Int[],
                  body_hlines_format::Union{Nothing,NTuple{4,Char}} = nothing,
                  continuation_row_alignment::Symbol = :c,
                  crop::Symbol = :both,
                  crop_subheader::Bool = false,
                  crop_num_lines_at_beginning::Int = 0,
                  columns_width::Union{Int,AbstractVector{Int}} = 0,
                  equal_columns_width::Bool = false,
                  highlighters::Union{Highlighter,Tuple} = (),
                  hlines::Union{Nothing,Symbol,AbstractVector} = nothing,
                  linebreaks::Bool = false,
                  maximum_columns_width::Union{Int,AbstractVector{Int}} = 0,
                  minimum_columns_width::Union{Int,AbstractVector{Int}} = 0,
                  newline_at_end::Bool = true,
                  overwrite::Bool = false,
                  noheader::Bool = false,
                  nosubheader::Bool = false,
                  row_name_crayon::Crayon = Crayon(bold = true),
                  row_name_header_crayon::Crayon = Crayon(bold = true),
                  row_number_alignment::Symbol = :r,
                  screen_size::Union{Nothing,Tuple{Int,Int}} = nothing,
                  sortkeys::Bool = false,
                  tf::TextFormat = unicode,
                  title_autowrap::Bool = false,
                  title_crayon::Crayon = Crayon(bold = true),
                  title_same_width_as_table::Bool = false,
                  vlines::Union{Nothing,Symbol,AbstractVector} = nothing)

Those arguments depends on the backend (text, LaTeX, HTML). Hence, the API function is declared like this:

pretty_table(data; kwargs...) =
    _pretty_table_select(stdout, data, []; kwargs...)

pretty_table(data, header::AbstractVecOrMat; kwargs...) =
    _pretty_table_select(stdout, data, header; kwargs...)

# This definition is required to avoid ambiguities.
pretty_table(data::AbstractVecOrMat, header::AbstractVecOrMat; kwargs...) =
    _pretty_table_select(stdout, data, header; kwargs...)

# This definition is required to avoid ambiguities.
pretty_table(io::IO, data::AbstractVecOrMat; kwargs...) =
    _pretty_table_select(io, data, String[]; kwargs...)

pretty_table(io::IO, data; kwargs...) =
    _pretty_table_select(io, data, String[]; kwargs...)

pretty_table(io::IO, data, header::AbstractVecOrMat; kwargs...) =
    _pretty_table_select(io, data, header; kwargs...)

and so on.

I am having a problem related to compile time. To print the first table (a matrix), it takes here 1.6s, which is pretty good given the purpose of the package. The problem happens when I try to define a global configuration.

Since there are a lot of things that can be configured, I would like to have a global variable that holds the configuration so that the user can save what they want. Right now, it is stored as a dictionary:

Dict{Symbol,Any} with 2 entries:
  :vlines => :all
  :hlines => :none

Hence, I call the function to print the table by transforming this dictionary into a named tuple:

    dictkeys = (collect(keys(conf.confs))...,)
    dictvals = (collect(values(conf.confs))...,)
    nt = NamedTuple{dictkeys}(dictvals)
    # Print the table.
    pretty_table(args...; nt...)

where conf.confs is that dictionary and conf is a global constant.

In this case, the time to print the first table goes to 5s! Is there any better way to do this so that it can be improved?

Sorry that I have not provided a MWE. I was not able to do this without actually copying a lot of PrettyTables.jl.

2

Curious. Is it slow due to the creation of the nt?

3

I think it is slow because of the global constant. The compile time is 5s, the execution time is very fast.

4

Can’t you splat the dictionary as kwargs directly? I imagine a namedtuple type with a lot of entries could take a long time to compile. Although not 5 seconds, that seems very long.

The other question I had was, why is confs wrapped in conf? If conf is constant, does it store exact type information about confs as well? Otherwise you could have an untyped global in conf.confs

5

Not sure if it will matter but why not just splat the dictionary straight into the call?

julia> f(; kwargs...) = @show kwargs
f (generic function with 1 method)

julia> f(;a=1, b=2)
kwargs = Base.Iterators.Pairs(:a => 1, :b => 2)
pairs(::NamedTuple) with 2 entries:
  :a => 1
  :b => 2

julia> d = Dict(:a=>1, :b=>2);

julia> f(; d...)
kwargs = Base.Iterators.Pairs(:a => 1, :b => 2)
pairs(::NamedTuple) with 2 entries:
  :a => 1
  :b => 2
2 Likes
6

Another approach to reduce compile times if you are on Julia 1.5 or later would be using Base.Experimental.@optlevel to set the optimization level to 0 or 1:

help?> Base.Experimental.@optlevel
  Experimental.@optlevel n::Int

  Set the optimization level (equivalent to the -O command line argument) for code in the current module.
  Submodules inherit the setting of their parent module.

  Supported values are 0, 1, 2, and 3.

  The effective optimization level is the minimum of that specified on the command line and in per-module
  settings.

Runtime might be higher, because the compiler misses out on some opportunities for optimization, but it doesn’t look like that should be a big problem here.

7

Thanks for the suggestions! I tried just splatting the Dict but it became slower:

Using Dict:

julia> @time pretty_table_with_conf(conf,df)
┌───────┬───────┐
│     a │     b │
│ Int64 │ Int64 │
├───────┼───────┤
│     1 │     2 │
│     2 │     4 │
│     3 │     6 │
└───────┴───────┘
  5.332409 seconds (16.27 M allocations: 841.609 MiB, 9.02% gc time)

Using the converted named tuple:

julia> @time pretty_table_with_conf(conf,df)
┌───────┬───────┐
│     a │     b │
│ Int64 │ Int64 │
├───────┼───────┤
│     1 │     2 │
│     2 │     4 │
│     3 │     6 │
└───────┴───────┘
  4.761661 seconds (16.14 M allocations: 835.053 MiB, 8.86% gc time)

PrettyTables.jl can print a Dict. Thus I needed to create a type to store this dictionary so that it can differentiate between the configurations and a dictionary to be printed. The structure is:

struct PrettyTablesConf
    confs::Dict{Symbol, Any}
end

# Global configuration object.
const _pt_conf = PrettyTablesConf()

Thanks for the suggestions, but I am already using @optlevel 1.

8

I think the problem is when I have keyword arguments together with kwargs.... For example, the compilation time decrease a lot if I replace:

    pretty_table(io, df, vcat(names,types);
                 crop                        = crop,
                 nosubheader                 = !eltypes,
                 row_number_column_title     = string(rowlabel),
                 title                       = title,
                 alignment                   = :l,
                 continuation_row_alignment  = :l,
                 crop_num_lines_at_beginning = 2,
                 formatters                  = (_formatter,),
                 highlighters                = (_DF_HIGHLIGHTER,),
                 newline_at_end              = false,
                 row_number_alignment        = :l,
                 show_row_number             = true,
                 vlines                      = [1],
                 tf                          = dataframe,
                 kwargs...)

to

    pretty_table(io, df, vcat(names,types);
                 kwargs...)

Is there anything I can do to improve?

9

You could also try putting @nospecialize around the keyword arguments, but I am not sure whether this helps here.

10

Thanks for the suggestion! But it did not change anything.

1 Like
11

After a lot of tries, I think this is the way Julia compiler works. I was not able to reduce the compilation time by any means. Even when I replaced all the keyword arguments for structures.

Next step is to use SnoopCompile to precompile everything.