[ANN] Announcing ArguMend.jl – for more helpful MethodErrors

Announcing

ArguMend.jl

Dev Build Status Coverage Aqua QA

ArguMend[1] is a small macro that injects a function with logic to help autocorrect mistyped keyword arguments.

@argumend function f(a, b; niterations=10, kw2=2)
    a + b - niterations + kw2
end

This results in “suggestive” MethodErrors that tell you what you mistyped:

julia> f(1, 2; iterations=1)
ERROR: SuggestiveMethodError: in call to `f`, found unsupported
       keyword argument: `iterations`, perhaps you meant `niterations`

This becomes very useful when calling into a large interface with many possible options.

This mechanism has (likely) zero runtime cost, as it relies on adding splatted keyword arguments to the function call, which will re-compile the function if any keyword arguments change, skipping the ArguMend functions altogether.

The core function used for computing candidate keywords is extract_close_matches, which is a clean-room pure-Julia re-implementation of Python’s difflib.get_close_matches, and considers the aggregate length of overlapping subsequences – seems to be well-suited for catching mistyped kwargs.

You can control the suggestions in the macro arguments:

@argumend n_suggestions=1 function g(; abc1=1, abc2=1, abc3=1)
    abc1 + abc2 + abc3
end

which will result in only (up to) one argument getting suggested:

julia> g(abc=2)
ERROR: SuggestiveMethodError: in call to `g`, found unsupported
       keyword argument: `abc`, perhaps you meant `abc1`

You can control the threshold with cutoff which goes from 0.0 (matches everything) to 1.0 (exact match).

Longer Example

I wrote this because SymbolicRegression.jl has a massive number of hyperparameters, and I wanted the error to tell the user (which includes me) what parameter name was mistyped.

The entire list of options in SR is as follows (expand):
function Options(;
    binary_operators=[+, -, /, *],
    unary_operators=[],
    constraints=nothing,
    elementwise_loss::Union{Function,Nothing}=nothing,
    loss_function::Union{Function,Nothing}=nothing,
    tournament_selection_n::Integer=12,
    tournament_selection_p::Real=0.86,
    topn::Integer=12,
    complexity_of_operators=nothing,
    complexity_of_constants::Union{Nothing,Real}=nothing,
    complexity_of_variables::Union{Nothing,Real}=nothing,
    parsimony::Real=0.0032,
    dimensional_constraint_penalty::Union{Nothing,Real}=nothing,
    dimensionless_constants_only::Bool=false,
    alpha::Real=0.100000,
    maxsize::Integer=20,
    maxdepth::Union{Nothing,Integer}=nothing,
    turbo::Bool=false,
    bumper::Bool=false,
    migration::Bool=true,
    hof_migration::Bool=true,
    should_simplify::Union{Nothing,Bool}=nothing,
    should_optimize_constants::Bool=true,
    output_file::Union{Nothing,AbstractString}=nothing,
    node_type=nothing,
    populations::Integer=15,
    perturbation_factor::Real=0.076,
    annealing::Bool=false,
    batching::Bool=false,
    batch_size::Integer=50,
    mutation_weights=NamedTuple(),
    crossover_probability::Real=0.066,
    warmup_maxsize_by::Real=0.0,
    use_frequency::Bool=true,
    use_frequency_in_tournament::Bool=true,
    adaptive_parsimony_scaling::Real=20.0,
    population_size::Integer=33,
    ncycles_per_iteration::Integer=550,
    fraction_replaced::Real=0.00036,
    fraction_replaced_hof::Real=0.035,
    verbosity::Union{Integer,Nothing}=nothing,
    print_precision::Integer=5,
    save_to_file::Bool=true,
    probability_negate_constant::Real=0.01,
    seed=nothing,
    bin_constraints=nothing,
    una_constraints=nothing,
    progress::Union{Bool,Nothing}=nothing,
    terminal_width::Union{Nothing,Integer}=nothing,
    optimizer_algorithm::AbstractString="BFGS",
    optimizer_nrestarts::Integer=2,
    optimizer_probability::Real=0.14,
    optimizer_iterations::Union{Nothing,Integer}=nothing,
    optimizer_f_calls_limit::Union{Nothing,Integer}=nothing,
    optimizer_options=NamedTuple(),
    use_recorder::Bool=false,
    recorder_file::AbstractString="pysr_recorder.json",
    early_stop_condition::Union{Function,Real,Nothing}=nothing,
    timeout_in_seconds::Union{Nothing,Real}=nothing,
    max_evals::Union{Nothing,Integer}=nothing,
    skip_mutation_failures::Bool=true,
    nested_constraints=nothing,
    deterministic::Bool=false,
    # Not search options; just construction options:
    define_helper_functions::Bool=true,
    deprecated_return_state=nothing,
)
    return nothing
end

If I wrap this call with @argumend, I get more useful error messages when I forget the name of a parameter:

julia> Options(; npopulations=3)
ERROR: SuggestiveMethodError: in call to `Options`, found unsupported keyword argument:
      `npopulations`, perhaps you meant `populations` or `population_size`

Here’s what the normal output looks like (scroll to the right)

ERROR: MethodError: no method matching Options(; npopulations::Int64)

Closest candidates are:
  Options(; binary_operators, unary_operators, constraints, elementwise_loss, loss_function, tournament_selection_n, tournament_selection_p, topn, complexity_of_operators, complexity_of_constants, complexity_of_variables, parsimony, dimensional_constraint_penalty, dimensionless_constants_only, alpha, maxsize, maxdepth, turbo, bumper, migration, hof_migration, should_simplify, should_optimize_constants, output_file, node_type, populations, perturbation_factor, annealing, batching, batch_size, mutation_weights, crossover_probability, warmup_maxsize_by, use_frequency, use_frequency_in_tournament, adaptive_parsimony_scaling, population_size, ncycles_per_iteration, fraction_replaced, fraction_replaced_hof, verbosity, print_precision, save_to_file, probability_negate_constant, seed, bin_constraints, una_constraints, progress, terminal_width, optimizer_algorithm, optimizer_nrestarts, optimizer_probability, optimizer_iterations, optimizer_f_calls_limit, optimizer_options, use_recorder, recorder_file, early_stop_condition, timeout_in_seconds, max_evals, skip_mutation_failures, nested_constraints, deterministic, define_helper_functions, deprecated_return_state) got unsupported keyword argument "npopulations"

  1. ArguMend is a play on “argument” and from “mending mistyped keyword arguments” ↩︎

33 Likes

@MilesCranmer this looks neat! I’m hoping to get “did you mean” functionality for all sorts of bits of Julia into a “helpful errors” stdlib of sorts, and hopefully subsume this package as part of that (through improvements to MethodError in general). If you (or anyone else) are interested in collaborating on this, that would be great!

My current thoughts: Julep: Exceptional Errors

5 Likes

That sounds awesome! I’d happily move my code over. It would be lovely to have this sort of thing be built-in behavior.

I’m currently thinking of stealing the fuzzy string matching code I have in DataToolkitBase.jl/src/model/utils.jl at main · tecosaur/DataToolkitBase.jl · GitHub for things like dictionary key errors, etc.

Rust-style error codes could probably do with a bit more thinking.

I’ll try not to derail your ANN thread any further, but if anyone’s interested in helping out with this (and I’m stretched thin enough I could do with some help :sweat_smile:), please DM me on Zulip/Slack. I’ll probably start a thread in #repl over on Zulip for this topic too.

that looks very nice! I am sure many packages can benefit from that, Makie.jl being one on top of my head. On the downside, I always don’t feel at ease when introducing macros in a codebase because the LSP goes bonkers…

1 Like

IIUC, I don’t think Makie (or HTTP.jl, another similarly problematic package) can use it, since it uses vararg kwargs extensively (f(a,b,c; kw...)), and I don’t think ArguMend can help since it doesn’t know the full list of valid keyword arguments from the function definition syntax. And in fact, technically there are often unlimited “valid” kwargs since at some point the trailing args are dropped instead of resulting in a MethodError, so if you make a typo, the kwarg just disappears without a trace, so even a try-catch based approach wouldn’t work.

Perhaps ArguMend could export the function injected into the body by the macro so you could set it up manually?

You would need to define the valid keywords manually (for the reasons @ericphanson explains), but you would otherwise have the same behavior. The complex part is the keyword matching so this would let you access that.

1 Like

@filchristou @ericphanson I just made it easier to call manually. For example:

using ArguMend: suggest_alternative_kws, SuggestiveMethodError

function f(; kws...)
    valid_kws = (:kw1, :kw2, :kw3)
    invalid_kws = filter(k -> !(k in valid_kws), keys(kws))
    if !isempty(invalid_kws)
        msg = suggest_alternative_kws(invalid_kws, valid_kws)
        throw(SuggestiveMethodError(f, msg))
    end

    # Do other stuff
end

I could see this being useful in a Makie context

2 Likes

Hi, I would rather have the valid_kws “registration” independent from the function declaration. A global registry, onda Dict{::MethodSignature, ::ValidKeys}() structure (the type is just an illustation). The macro @argumend could just search for the given function in the registry and do the check/error throwing. Also, given that we are in the checking business, It would be nice to have an automatic test suit for the registered functions/kwargs pairs (ej: test_ArguMend_registry()). This way we help people to keep the registry sync with the function declarations…

All the above in addition to the already automatic engine