Using macros to generate NamedTuples

I am a bit confused by using macros to generate expressions. I would like to have a macro to make NamedTuples as follows:

a = 1
b = 2
@make_nt a b

which should return an expression that evaluates to NamedTuple{(:a, :b)}((a, b)) or (a=1, b=2)

however

macro make_nt_try(args...)
    return :(NamedTuple{$args}($args))
end

returns (a = :a, b = :b)

and

macro make_nt_expr(args...)
    return Expr(:(NamedTuple{$args}), args)
end

raises TypeError: in Expr, expected Symbol, got Expr

and

macro make_nt(args...)
    return :(NamedTuple{$args}(tuple($(args...))))
end

works outside of functions, but not for variables in the local scope of functions.
I have read the metaprogramming section of the documentation, but did not understand when exactly evaluation happens and why it does not happen within the local scope.

@macroexpand @make_nt(a, b)

always gives

:(Main.NamedTuple{(:a, :b)}(Main.tuple(Main.a, Main.b)))

even when it is called inside a function.

Is it possible to make a macro that works properly inside function, or is my mental model for how macros should be used wrong in some way?

1 Like

Looks like like you’re missing an esc. See the macro hygiene section of the documentation.

Does this do what you want?

macro make_nt(args...)
    return :(NamedTuple{$args}(tuple($(esc.(args)...))))
end
4 Likes

It does, thank you for the quick reply! I will look deeper in the escaping and macro hygiene parts to understand it.

Thanks for providing the answer for the question that I’m also thinking about now.

I’m very curious about the two operations ... and tuple here.
It seems like that

macro make_nt(args...)
:( $(esc.(args)) )
end

doesn’t work, as shown below

julia> @make_nt(a, b, c)
(:($(Expr(:escape, :a))), :($(Expr(:escape, :b))), :($(Expr(:escape, :c))))

(Why)?. So the ... is necessary. I think the action first ... then use tuple to collect makes sense, but why can/should we use a $ in between?
Can someone teach me about this? And moreover, if this macro can be rewritten with NamedTuple(k => v for (k, v) = zip(someK, someV)) style? Thanks!


Edit: seems it’s the only available option Metaprogramming · The Julia Language.
The tuple is like a function call, being necessary to contain ....

First, you don’t need a macro for this.

a = 1
b = 2
julia> (; a, b)
(a = 1, b = 2)

Anyway, a macro is very similar to a function, indeed, internally it’s treated almost exactly like a function. The difference is that it runs right after parsing, before the lowering, so its input is symbolic, with Expr, Symbol, number and string literals, but no type information. So in the

@make_nt(a, b, c) # or the equivalent @make_nt a b c

the args will be (:a, :b, :c) (an NTuple{3,Symbol}). The output should also be symbolic, so we can be lazy and make the Expr corresponding to (a=a, b=b, c=c):

julia> Meta.@dump (a=a, b=b, c=c)
Expr
  head: Symbol tuple
  args: Array{Any}((3,))
    1: Expr
      head: Symbol =
      args: Array{Any}((2,))
        1: Symbol a
        2: Symbol a
    2: Expr
      head: Symbol =
      args: Array{Any}((2,))
        1: Symbol b
        2: Symbol b
    3: Expr
      head: Symbol =
      args: Array{Any}((2,))
        1: Symbol c
        2: Symbol c

So, let’s make the macro by crafting together an Expr of the same form:

macro make_nt(args...)
    ex = Expr(:tuple)
    for a in args
        push!(ex.args, Expr(:(=), a, a))
    end
    return ex
end 
julia> @macroexpand @make_nt a b c
:((a = Main.a, b = Main.b, c = Main.c))

Note that the macro hygiene has added Main.. In general it prepends the module in which the macro is defined. Just like in a function which looks up undefined variables as a global in the module. To avoid this, to operate on the caller’s names, the Symbols must be escaped:

macro make_nt(args...)
    ex = Expr(:tuple)
    for a in args
        push!(ex.args, Expr(:(=), a, esc(a)))
    end
    return ex
end 
julia> @macroexpand @make_nt a b c
:((a = a, b = b, c = c))

Now, making Exprs by hand in this way is cumbersome, and so is parsing a complicated input like a for loop or other expressions. The latter is easier with the matching facilities in MacroTools.jl.

Creating expressions is easier with quote, which creates expressions from ordinary julia code, with things interpolated in with $:

macro make_nt(args...)
    quote
        NamedTuple{$args}($(args...))
    end
end
julia> @macroexpand @make_nt a b c
quote
    #= REPL[53]:3 =#
    Main.NamedTuple{(:a, :b, :c)}(Main.a, Main.b, Main.c)
end

We see that the same things happens, they are prefixed by Main.. Each one of them must be escaped:

macro make_nt(args...)
    quote
        NamedTuple{$args}($(esc.(args)...))
    end
end 
julia> @macroexpand @make_nt a b c
quote
    #= REPL[84]:3 =#
    Main.NamedTuple{(:a, :b, :c)}(a, b, c)
end

In short, the hygiene rules, which act on the Expr returned from the macro, are made to make the Expr roughly like a function scope. Variables which are created get a unique name so they won’t interfere with anything in the scope where the macro is called, variables which are only read is prepended with the module name where the macro is defined. If you need to use the caller’s variable names, they must be esced.

You can see them in action here:

julia> macro make_nt(args...)
           ex = quote
               NamedTuple{$args}(($(args...),))
           end
           display(ex)
           return ex
       end
@make_nt (macro with 1 method)
julia> @macroexpand @make_nt a b c
quote
    #= REPL[19]:3 =#
    NamedTuple{(:a, :b, :c)}((a, b, c))
end
quote
    #= REPL[19]:3 =#
    Main.NamedTuple{(:a, :b, :c)}((Main.a, Main.b, Main.c))
end

The first quote is from the display inside the macro. The second is the returned expression which has been sanitized by the hygiene rules.

1 Like

Thank you so much. I’ve learned a lot.

One small issue is that your last code example won’t work

julia> NamedTuple{(:a, :b)}( (1, 2) )
(a = 1, b = 2)

julia> NamedTuple{(:a, :b)}( 1, 2 )
ERROR: MethodError: no method matching

If let me describe a macro definition, it comprises a body part and a return part

macro my(a, b)
    var = _some_fun(a, b) # body part
    return _some_expr     # return part
end

I see this info

help?> esc
  esc(e)
  Only valid in the context of an Expr returned from a macro.

So I’m unsure if it’s valid to use esc within the body part, as

Am I understanding that docstring wrongly? Or is that docstring itself a vague exposition?
I won’t get any warning or Error by writing invalid esc:

julia> macro my(e)
           t = :($(esc(e)))
           return :($e)
       end
@my (macro with 1 method)

julia> @my(identity)
identity (generic function with 1 method)

julia> map(ans, [2, 1])
2-element Vector{Int64}:
 2
 1

You might be interested in my package AddToField.jl.

Thanks. but I think I only need the most basic functions of NamedTuple.

Right. That’s why you need the tuple:

macro make_nt(args...)
    quote
        NamedTuple{$args}(tuple($(esc.(args)...)))
    end
end 
julia> @macroexpand @make_nt a b c
quote
    #= REPL[91]:3 =#
    Main.NamedTuple{(:a, :b, :c)}(Main.tuple(a, b, c))
end

or

macro make_nt(args...)
           quote
               NamedTuple{$args}(($(esc.(args)...),))
           end
       end
@make_nt (macro with 1 method)

julia> @macroexpand @make_nt a b c
quote
    #= REPL[207]:3 =#
    Main.NamedTuple{(:a, :b, :c)}((a, b, c))
end

You can put esc anywhere:

julia> e = esc(:a)
:($(Expr(:escape, :a)))

julia> dump(e)
Expr
  head: Symbol escape
  args: Array{Any}((1,))
    1: Symbol a

julia> e = Expr(:escape, :a)
:($(Expr(:escape, :a)))

julia> eval(e)
ERROR: syntax: "esc(...)" used outside of macro expansion

esc is just an ordinary function:

julia> @less esc(:a)
esc(@nospecialize(e)) = Expr(:escape, e)

But the lowering/compilation step won’t accept such an Expr. The macro hygiene stuff which is run on the returns from macros will remove it.

Many of the “magic” system macros do something similar, it inserts some Expr which is meant for the lowering step:

julia> @macroexpand @inbounds begin; something; end
quote
    $(Expr(:inbounds, true))
    local var"#64#val" = begin
                #= REPL[204]:1 =#
                something
            end
    $(Expr(:inbounds, :pop))
    var"#64#val"
end
1 Like

Btw, Walter,
Here’s an interesting one for you who pondered scopes in loops:

julia> macro myintvec(N)
           N isa Integer || error("argument must be a literal integer")
           v = fill(0, N)
           return v
       end
@myintvec (macro with 1 method)

julia> for i in 1:5
           v = @myintvec(5)
           v[i] = i
           println(v)
       end
[1, 0, 0, 0, 0]
[1, 2, 0, 0, 0]
[1, 2, 3, 0, 0]
[1, 2, 3, 4, 0]
[1, 2, 3, 4, 5]

I don’t understand how you can return a non-expression from a macro.

Oh no, macros shouldn’t interfere in this usage. And I am afraid to make speculations with the existence of macro usage.

Typically I would only write one kind of macro myself in practice which is like

macro get_int_decision(model, expr) return esc(quote
...
end) end

So it fundamentally lacks the body part—only having one return, and one all-including esc—so I can write code “safely” within that—which is familiar to me. An example is Adding set_integer(::AffExpr)? · Issue #4073 · jump-dev/JuMP.jl · GitHub.

The julia doc’s comment about this is

This kind of manipulation of variables should be used judiciously, but is occasionally quite handy.

I only see its handy aspect. Am I missing anything? I want to ask. e.g. Will this style still be of normal performance?

There is not really a “body part” and a “return part” in a macro. A macro is an ordinary function, though its inputs are symbolic, and so is its return value. However, it’s allowed to have julia objects in Exprs, and even a single object not inside an Expr. this is how interpolation with $ works in quote:

julia> v = fill(0,3)
3-element Vector{Int64}:
 0
 0
 0

julia> dump( :(a = $v[1]) )
Expr
  head: Symbol =
  args: Array{Any}((2,))
    1: Symbol a
    2: Expr
      head: Symbol ref
      args: Array{Any}((2,))
        1: Array{Int64}((3,)) [0, 0, 0]    # <-- this is the v vector
        2: Int64 1

While escaping the entire output is possible and sometimes convenient, it can yield surprises since it also escapes all function calls:

julia> macro add(a,b)
           esc(:($a + $b))
       end
@add (macro with 1 method)

julia> let
           x + y = x * y
           @add(2,3)
       end
6
1 Like

This is expected as “expand into the caller’s environment”.


I guess that the variable v is still local for each i-iteration. But you introduced a bridge so that the local v at i = 1 is “passed” successfully to i = 2, 3, 4, 5. But I’m not sure…

The thing is that the macro is run (at least) once when the loop is parsed, not once in every iteration. The macro creates a vector, and returns it. I.e. inside the loop there will be something like
v = <the vector created at parse time>.

If you add another loop or function which uses the macro, it will be run once there too, creating another vector.

julia> f() = @myintvec(0)
f (generic function with 1 method)

julia> v = f()
Int64[]

julia> resize!(v, 2);

julia> v[1] = 1;

julia> v[2] = 2;

julia> push!(v, 5);

julia> f()
3-element Vector{Int64}:
 1
 2
 5

julia> empty!(v)
Int64[]

julia> f()
Int64[]

This is only slightly different from capturing, where the vector is created by the g(0) call, not at parse time by a macro:

julia> g(N) = (v = fill(0,N); () -> v)
g (generic function with 1 method)

julia> f = g(0)
#g##2 (generic function with 1 method)

julia> v = f()
Int64[]

julia> resize!(v, 2);

julia> v[1] = 1;

julia> v[2] = 2;

julia> push!(v, 5);

julia> f()
3-element Vector{Int64}:
 1
 2
 5

Sure, until the @add-macro writer decides to update the @add macro:

macro add(a, b)
    esc(:(sum(($a,$b))))
end
julia> let
           x + y = x * y
           @add(2,3)
       end
5

julia> let a=1, b=2
           sum = a + b
           @add(sum, a)  # this worked before!
       end
ERROR: MethodError: objects of type Int64 are not callable
1 Like

I’m not sure whether someone has considered this before. I wrote a variant macro following the spirit of this topic, to help us construct a struct instance:

struct S
    c::Int
    d::Int
    a::Int
    b::Int
end
macro lazy_instantiate(t...)
    quote
        iv = map(s -> findfirst(x -> x == s, $t), fieldnames(S))
        v = tuple($(map(esc, t)...))
        S(map(i -> v[i], iv)...)
    end
end
a, b, c, d = 2, 5, 4, 7
s = S(c, d, a, b) # use function: subject to the order
s == @lazy_instantiate(a, b, c, d) && println("use macro: permute at will")

Edit: Oh my god, I asked AI, who give me an even better solution

macro lazier_instantiate()
    vals = map(esc, fieldnames(S))
    :( $S($(vals...)) )
end
1 Like

Consider @kwdef struct ..., and you can just do S(; c, d, a, b) afterwards.

1 Like