Weird macro behavior trying to simulate keyword arguments

General Usage
1

DataFramesMeta’s macros are weird, because they want to behave like functions. @transform(df, y = 5), but they are still macros.

If @transform were a function, it would be implemented like transform(df; kwargs...). Indeed, this is how the functions that @transform calls work.

So I thought, in DataFramesMeta #117. So I thought that it would be nice to have pairs of keyword arguments work in @transform as well so you could get run-time keyword argument evaluation @transform(df; x => mean(:x1) as well.

It turns out this isn’t so easy, and definitions like macro foo(args...; kwars...) are disallowed. This is understandable, since macros work with Expr and therefor keyword arguments dont really make sense.

No big deal, one can just inspect the AST inputted into a macro call and make pairs of keyword arguments just work, right? This is where unexpected behavior comes in.

macro foo(args...)
    dump(ars)
end

Returns

julia> @foo(a = 1; b = 2)
Tuple{Expr,Expr}
  1: Expr
    head: Symbol parameters
    args: Array{Any}((1,))
      1: Expr
        head: Symbol kw
        args: Array{Any}((2,))
          1: Symbol b
          2: Int64 2
  2: Expr
    head: Symbol =
    args: Array{Any}((2,))
      1: Symbol a
      2: Int64 1

However

macro bar(x, args...)
    dump(args)
end

returns

julia> @bar(x, a = 1; b = 2)
Tuple{Symbol,Expr}
  1: Symbol x
  2: Expr
    head: Symbol =
    args: Array{Any}((2,))
      1: Symbol a
      2: Int64 1

When you have a macro where everything is splatted, the collected expression includes everything between the parentheses. But when you have a multi-argument macro, where the second argument is splatted, the collected expression only goes until the first semi-colon.

This seems like unexpected and inconsistent behavior. Should I file an issue?

Seems to be some discussion of similar behavior here in MacroTools.jl.

2

All the kwargs get collected in the first argument x, all the rest gets collected in args. This is indeed a bit weird. Maybe worth opening an issue. But this couldn’t be changed until Julia 2.0.

1 Like
3

Not unless it is considered a bug, and the changes don’t break any registered package.

4

What’s the inconsistency here? The keyword argument is always passed in first, which is strange I agree but not inconsistent, so when you splat everything, you get the keyword argument followed by the positional ones and if you supply one normal argument first, that argument gets the, well, first argument, namely the keyword, and the splatting gets the rest, namely the positional ones. Essentially, if you define bar as dump((x, args...)) it should behave exactly the same as foo. This seems to be the case so the two are consistent.

As I said, the only strange thing here is the argument where keyword arguments are passed in. Changing this is highly breaking.

1 Like
5

Thanks for the answer! dump((x, args...)) gets the first argument as the :parameters-annotated keyword arguments.

I will admit that this is a bit of a pain. The relevant code to work with is here:

function transform_helper(x, args...)
    quote
        $transform($x, $(map(args) do kw
            Expr(:kw, kw.args[1], with_anonymous(kw.args[2]))
        end...) )
    end
end

And adding some keyword-argument-type thing would break this as now x becomes the :parameters expression, which is counter-intuitive.

6

That I agree. It even had real consequences that I complained about. Unfortunately, changing this will be very breaking and I’m not sure if there can be a way to give a useful depwarn for macros…

1 Like
7

Is there any way to leverage the way functions automatically parse this so I can avoid doing a bunch of => to = transitions and sorting through the syntax tree?

Or is it all done in lisp?

8

Tangentially related, the way I’ve done keyword arguments with macros in the past is to have the macro create an expression that calls a proper keyword argument function which returns the ‘completed’ set of parameters. See

(The whole package is just 100 lines with 40 lines of docstring, so it should be pretty obvious what’s going on.)

9

This is interesting, thanks!

My use-case is probably something a bit more abstract, since we would just have kwargs... splatted and not know what they will be.

It seems like a MacroKeywordArguments could be a useful package.