Optional argument in macro

General Usage
, ,
1

I’m creating my custom progress bar for my ML related loop. It’s based on ProgressMeter.jl. The macro takes an argument as input that is a dictionary associating names to vectors of data that I’m saving during the computation. At each loop the metrics are updated and the last updated metric is rounded and then put into the progressbar.

I’m creating this macro such that I can have an easy usage such as:
@trainprogress metrics for i in 1:epochs. I would also though add a way to disable the progressbar passing an optional enabled, like this
@trainprogress false metrics for i in 1:epochs. This would disable the progressbar.

Is it possible in general to have a macro with optional arguments? Or even optional keyword arguments? How could I achieve it?

function ml_showvalues(
    epoch::Integer,
    metrics::Dict{String,<:AbstractVector};
    n_digits::Integer=4
)
    vec_metrics = Vector{Tuple{String,Any}}([("Epoch", epoch)])
    for (key, value_arr) in metrics
        value = round(value_arr[epoch], digits=n_digits)
        push!(vec_metrics, (key, value))
    end
    () -> vec_metrics
end

macro trainprogress(metrics_expr, loop_expr; enabled_expr=???)
    loop_assignment = loop_expr.args[1]
    loop_body = loop_expr.args[2]
    iter_var = loop_assignment.args[1]
    iter_range = loop_assignment.args[2]
    return quote
        local metrics_dict = $(esc(metrics_expr))
        local epoch_progress = Progress(
            length($(esc(iter_range))),
            desc="Training: ",
            enabled=$(esc(enabled_expr))
        )
        for $(esc(iter_var)) in $(esc(iter_range))
            $(esc(loop_body))
            next!(
                epoch_progress,
                showvalues=ml_showvalues($(esc(iter_var)), metrics_dict)
            )
        end
    end
end
1 Like
2

In Trixi.jl, we have a workaround to have keyword arguments for macros. You can find the code here: Trixi.jl/test/test_trixi.jl at 214cb71175a53e2bddc38709f98d2e074bc2d1fe · trixi-framework/Trixi.jl · GitHub. The get_kwarg function is defined further down in the file. Maybe this helps.

3 Likes
3

Optional positional arguments for macros and functions both generate multiple methods, and you dispatch to one by providing a specific number of arguments to the call.

julia> macro blah(x, y=:(nothing)) end
@blah (macro with 2 methods)

julia> methods(var"@blah")
# 2 methods for macro "@blah" from Main:
 [1] var"@blah"(__source__::LineNumberNode, __module__::Module, x, y)
     @ REPL[1]:1
 [2] var"@blah"(__source__::LineNumberNode, __module__::Module, x)
     @ REPL[1]:1

As mentioned in the docstring for macro, keyword arguments are disallowed in definitions.

julia> macro blah(x; y=:(nothing)) end
ERROR: syntax: macros cannot accept keyword arguments around REPL[6]:1

That’s unlikely to change. While function calls can distinguish the special syntax of keyword arguments interspersed anywhere among the positional arguments e.g. foo(z = 5, 1, y = 4, 2, 3), a macro call must take the same syntax as an unprocessed Expr(:(=), args...). As demonstrated by JoshuaLampert however, your macro can make any rule about which positional arguments take Expr(:(=), args...) to be internally processed like keyword arguments.

3 Likes
4

So, if I want to method dispatch it giving a default to a middle entry (because, for example, I always want the last expression to be the for loop).

In my case I would want @trainprogress metrics is_enabled for i in 1:epochs or, if not given, @trainprogress metrics for i in 1:epochs.

So this would mean that I need a base macro like this one

macro trainprogress(metrics_expr, enabled_expr, loop_expr)
    ...
end

Then I tried this one as for method dispatch, but it doesn’t work

macro trainprogress(metrics_expr, loop_expr)
    return :(@trainprogress $metrics_expr true $loop_expr)
end

Is it even possible to do this?

5

I suggest looking at @less Threads.@threads :static for i in 1:10 end for inspiration. It’s defined macro threads(args...) and then sorts out which arguments are options and which content.

I wouldn’t. Another common pattern is macro trainprogress(ex...); _ trainprogress(ex...) |> esc; end. Now _trainprogress is an ordinary function, taking and returning expressions. And you can happily write 2- and 3-arg versions of this function which call each other, if you want.

2 Likes
6

Yes, macros can transform into expressions with macro calls, and it is done. However, to elaborate on how it’s far more limited than calling functions that process Expr:

  1. The 3-arg method isn’t actually being called in the 2-arg method. The 2-arg method is just returning an expression that will call the 3-arg method when the parser expands it. Sometimes that’s fine, but sometimes you want to process the expression more. Functions calling each other can do that because they get results right at the call site, when the method body hasn’t returned yet.
  2. You really can’t call the 3-arg method inside the 2-arg method because the parser will execute and expand the macro call during the definition. Function calls only execute after their callers have executed, which is far easier to reason about.
  3. Macro hygiene doesn’t propagate through methods in a convenient way. I’ll abbreviate and adapt your example into a minimal working example. esc makes macro expansions resolve subexpressions in the scope of the call instead of the scope of the macro definition, which we can see by which variables are not qualified with Main.Foo in the expanded expression:
julia> module Foo
       macro tp(m, e, l)
         println("@tp($m, $e, $l) expanded in ", __module__)
         :($m, $e, $(esc(l)))
       end
       end
Main.Foo

julia> @macroexpand Foo.@tp x true y
@tp(x, true, y) expanded in Main
:((Main.Foo.x, true, y))

julia> @eval Foo macro tp(m, l) # can't write Foo.tp like functions yet
         println("@tp($m, $l) expanded in ", __module__)
         :(@tp $m true $l)
       end
@tp (macro with 2 methods)

It’s easy to assume that if we call the 2-arg method, it’d expand to a macro call of the 3-arg method, which then expands in the same module to the esc-aped expression, so the symbols should turn out the same, right?

julia> @macroexpand Foo.@tp x y
@tp(x, y) expanded in Main
@tp(x, true, y) expanded in Main
:((Main.Foo.x, true, Main.Foo.y))

Nope, it only works if esc is repeated through both calls; note how x still doesn’t make it.

julia> @eval Foo macro tp(m, l) # can't write Foo.tp like functions yet
         println("@tp($m, $l) expanded in ", __module__)
         :(@tp $(esc(m)) true $(esc(l)))
       end
@tp (macro with 2 methods)

julia> @macroexpand Foo.@tp x y
@tp(x, y) expanded in Main
@tp($(Expr(:escape, :x)), true, $(Expr(:escape, :y))) expanded in Main
:((Main.Foo.x, true, y))

Luckily we wouldn’t have to struggle with repeating esc across macro calls in practice even if we do opt to chain macro calls like this because it should be apparent which subexpressions are resolved at the call site (especially the input expressions) and which subexpressions are wholly introduced from the definition scope. If you’re involving many macros drawing from different modules, this would be fine. If you’re working from your own module, you really don’t need to type extra esc across several macro calls; you can instead build the expression for one macro call.

1 Like
7

Such a great answer, I’ve learned a lot, thank you. Thanks to also everybody else that answered.

8

As a possible additional note/quirk I’d like to share, while you cannot define a macro with keyword arguments like this as mentioned above:

julia> macro with_kws(ex; kws...)
       @show ex kws
       end
ERROR: syntax: macros cannot accept keyword arguments around REPL[1]:1

You can actually call a macro with keyword arguments, by using its functional form:


julia> macro f(exprs...)
       for (i, expr) in enumerate(exprs)
           println("expression $i:")
           println(expr)
           println()
       end
       return exprs
       end
@f (macro with 1 method)

julia> @f(1+1; a = 3)
expression 1:
$(Expr(:parameters, :($(Expr(:kw, :a, 3)))))

expression 2:
1 + 1

(:($(Expr(:parameters, :($(Expr(:kw, :a, 3)))))), :(1 + 1))

julia> @f a = 3 1+1
expression 1:
a = 3

expression 2:
1 + 1

(:(a = 3), :(1 + 1))

The interesting thing is that because of how Julia parses expressions, the keyword arguments show up as the first argument in the actual macro, with the Expr(:parameters, ...) structure. So this would lead to @f(kw_expr, other_expr) being called in the end.
In principle it is possible to thus support all of these options, although you might want to decide whether or not this is actually worth it, depending on how much you like the syntax of macros in their functional form.
For reference, the @tensor macro from TensorOperations.jl actually does support this, see the keyword parsing here. I’m not sure if anyone actually really uses that, as the typically use would be @tensor kw1 = value kw2 = value tensorexpr

1 Like
9

To elaborate, the ; syntax parsing to a Expr(:tuple, Expr(:parameters, ...), ...) instance is different from a function call’s processing of its keyword arguments. In addition to calls processing specified keywords to local variables in the method body, ; is optional in such function calls, kwargs can gather extra keyword arguments even if some were written before the ;, and it’s invalid to put an argument without a keyword after ;

julia> f(args...; kwargs...) = (args, kwargs)
f (generic function with 1 method)

julia> f(1, a = 3, 2, b = 4)
((1, 2), Base.Pairs(:a => 3, :b => 4))

julia> f(1, a = 3, 2; b = 4)
((1, 2), Base.Pairs(:a => 3, :b => 4))

julia> f(1, a = 3, 2; 4)
ERROR: syntax: invalid keyword argument syntax "4" around REPL[14]:1

A macro call doesn’t do those, so the body may have to do a bit of processing.

julia> @f(1, a = 3, 2; b = 4) # lkdvos' macro f(exprs...)
expression 1:
$(Expr(:parameters, :($(Expr(:kw, :b, 4)))))

expression 2:
1

expression 3:
a = 3

expression 4:
2

(:($(Expr(:parameters, :($(Expr(:kw, :b, 4)))))), 1, :(a = 3), 2)

julia> @f(1, a = 3, 2; 4)
expression 1:
$(Expr(:parameters, 4))

expression 2:
1

expression 3:
a = 3

expression 4:
2

(:($(Expr(:parameters, 4))), 1, :(a = 3), 2)