Nested macros and esc

I have an example of nesting macros that makes me think I’m confused about how hygiene works.

Here’s a minimal example:

macro inner(e)
    println("In @inner")
    dump(e)
    @assert isa(e, Expr) && e.head == :$
    esc(e.args[1])
end

macro outer(e)
    println("In @outer")
    dump(e)
    quote
        @inner($(e))
    end
end

let x = 1
    @inner($x)
end

let x = 1
    @outer($x)
end

This behaves as follows, which confuses me:

julia> macro inner(e)
           println("In @inner")
           dump(e)
           @assert isa(e, Expr) && e.head == :$
           esc(e.args[1])
       end
@inner (macro with 1 method)

julia> macro outer(e)
           println("In @outer")
           dump(e)
           quote
               @inner($(e))
           end
       end
@outer (macro with 1 method)

julia> let x = 1
           @inner($x)
       end
In @inner
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
1

julia> let x = 1
           @outer($x)
       end
In @outer
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
In @inner
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
ERROR: UndefVarError: x not defined
Stacktrace:
 [1] top-level scope at REPL[2]:5
 [2] top-level scope at REPL[4]:2

What I struggle to reconcile in my understanding is why the nested call to @inner seems to reflect the exact same input as the non-nested call sees, but the result differs. My mental model of nested macro calls is that the outer layer is evaluated first and then the inner layer runs as if the outer layer had never existed, but this example makes me question that model.

1 Like

One question I have is whether I essentially need as many calls to esc as I have layers of nesting because the following does work for @outer, but fails for @inner (as I would expect):

macro inner(e)
    @assert isa(e, Expr) && e.head == :$
    esc(esc(e.args[1]))
end

macro outer(e)
    quote
        @inner($(e))
    end
end

let x = 1
    @inner($x)
end

let x = 1
    @outer($x)
end

I gave up on hygiene a long time ago and I have been much happier with manual @gensyms. What are you trying to do with these macros?

1 Like

The use case I have is simplifying the code in Volcanito, where I’d like to have an outer macro generate repeated calls to an inner macro to make the sequence of macro expansions easier to debug.

In that example, you want to do something like the following:

@select(df, a, b, d = $x + a + b)
# ==>
Volcanito.Projection(
    df,
    (
        @expression(a),
        @expression(b),
        @expression(d = $x + a + b),
    )
)

I’m not sure I can make this work with gensyms given that I’m intentionally trying to capture local variables.

One thing that’s also weird about this is that it’s happening at parse time, right?

julia> macro outer(e)
           println("In @outer")
           dump(e)
           quote
               @show x
               @inner($e)
           end
       end
@outer (macro with 1 method)

julia> let x = 1
           @outer($x)
       end
In @outer
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
In @inner
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
ERROR: UndefVarError: x not defined
Stacktrace:
 [1] top-level scope at show.jl:641
 [2] top-level scope at REPL[35]:5
 [3] top-level scope at REPL[36]:2

shouldn’t the @show x in @outer run here?

Very interested in this, because I am having a hard time justifying switching DataFramesMeta to x instead of :x in light of all the macro escaping headaches this will cause.

1 Like

How do you disable the hygiene rewriter? A practical example would be nice.

1 Like

The issue you’re hitting makes more sense to me because the x shouldn’t be defined in the scope that exists while evaluating @outer, no?

To point a fine point on it, what really confuses me is the way this all interacts with @macroexpand1 since @macroexpand1 behaves like there’s no path dependence on a call to @outer having ever existed, but the need for double calling esc suggests there is path dependence:

julia> macro inner(e)
           @assert isa(e, Expr) && e.head == :$
           esc(e.args[1])
       end
@inner (macro with 1 method)

julia> macro outer(e)
           quote
               @inner($(e))
           end
       end
@outer (macro with 1 method)

julia> let x = 1
           @inner($x)
       end
1

julia> @macroexpand1 let x = 1
           @outer($x)
       end
:(let x = 1
      #= REPL[4]:2 =#
      begin
          #= REPL[2]:3 =#
          #= REPL[2]:3 =# @inner $(Expr(:$, :x))
      end
  end)

Why is that?

My mental model is that

macro outer(e)
    quote
        @show x
        @inner($(e))
    end
end

should be the same as

let x = 1
    @show x 
    @inner(x)
end

barring your weird escaping problem.

For clarification


julia> macro foo(e)
           quote
              @show z
              $e
          end
       end
@foo (macro with 1 method)

julia> let z = 5
       @foo y = 5
       end
z = 9
5
1 Like

You’re right, I misread your example and thought the @show wasn’t nested inside the quote block.

Don’t you need something @show $(esc(x)) to avoid a gensym operation on x though?

1 Like

Yes, that was just an example to show that I think the behavior is odd, fewer chance for bugs using an x literal. You would definitely need that to make it a workable macro.

1 Like

This example doesn’t work for me. Did you have a global z = 9 somewhere?

You are totally right. My mental model for macros needs work.

1 Like

I suspect this issue comes down to distinguishing the following two possibilities.

(1) Nested macros are partially expanded outer-to-inner, but the expansions occur without any hygiene steps taking place until all partial expansions are finished. So you get a sequence like:

  • Expansion of @outer without hygiene
  • Expansion of @inner without hygiene
  • Hygiene for @inner
  • Hygiene for @outer

(2) Nested macros are fully expanded outer-to-inner including hygiene:

  • Expansion of @outer without hygiene
  • Hygiene for @outer
  • Expansion of @inner without hygiene
  • Hygiene for @inner

I had believed (2) is what happens, but I think this example suggests (1) is what happens.

1 Like

esc by default “captures the local variables”.

julia> module Volcanito
       using MacroTools

       macro select(df, exprs...)
           get_var(e) = @capture(e, lhs_ = _) ? lhs : e
           vars = map(get_var, exprs)
           esc(quote 
           $Volcanito.Projection($df, tuple($([:($Volcanito.@expression($e, $(vars...))) for e in exprs]...)))
           end)
       end

       macro expression(eq, vars...)
           esc(Expr(:let, Expr(:block, [Expr(:(=), v, 55) for v in vars]...), eq))
       end
       end
Main.Volcanito

julia> MacroTools.striplines(@macroexpand1 Volcanito.@select df a b d = x + a + b)
quote
    (Main.Volcanito).Projection(df, 
    tuple((Main.Volcanito).@expression(a, a, b, d), 
          (Main.Volcanito).@expression(b, a, b, d), 
          (Main.Volcanito).@expression(d = x + a + b, a, b, d)))
end

julia> MacroTools.striplines(@macroexpand1 Volcanito.@expression(d = x + a + b, a, b, d))
:(let a = 55, b = 55, d = 55  # could be `a = df.a` etc.
      d = x + a + b   # thus the "local" x will be used
  end)

looks like along the lines of what you need, but I don’t know what semantics you want from $x and @expression .

2 Likes

We already had that conversation :slightly_smiling_face: Looking for good macro examples

Is it unclear?

1 Like

You do need some escaping at each level, @outer2 and @outer3 are different ways you might think to do it. But @outer3 doesn’t work, because it passes @inner an expression which doesn’t have the $ nakedly visible. There is a helpful issue about this: https://github.com/JuliaLang/julia/issues/37691.

The problem with @outer2 is that that everything is in the caller’s scope, including the inner macro. When it’s in another module, it needs Two.@inner2. (Or better, you could interpolate, which is just $ for functions, but is apparently tricker for macros.)

julia> macro inner(e)
           println("In @inner")
           dump(e)
           @assert isa(e, Expr) && e.head == :$
           esc(e.args[1])
       end
@inner (macro with 1 method)

julia> macro outer2(e)
           println("In @outer")
           dump(e)
           quote
               @inner($(e))
           end |> esc
       end
@outer2 (macro with 1 method)

julia> let x = 1
           @inner($x)
       end
In @inner
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
1

julia> let x = 1
           @outer2($x)
       end
In @outer
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
In @inner
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
1

julia> macro outer3(e)
           println("In @outer")
           dump(e)
           quote
               @inner($(esc(e)))
           end
       end
@outer3 (macro with 1 method)

julia> let x = 1
           @outer3($x)
       end
In @outer
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
In @inner
Expr
  head: Symbol escape
  args: Array{Any}((1,))
    1: Expr
      head: Symbol $
      args: Array{Any}((1,))
        1: Symbol x
ERROR: LoadError: AssertionError: e isa Expr && e.head == :$
Stacktrace:
 [1] @inner(::LineNumberNode, ::Module, ::Any) at ./REPL[4]:4
in expression starting at REPL[8]:6245:

julia> module Two
       macro inner2(e)
           println("In Two.@inner2")
           dump(e)
           @assert isa(e, Expr) && e.head == :$
           esc(e.args[1])
       end
       function inner2()
         2
       end

       macro outer2(e)
           println("Expanding Two.@outer2")
           dump(e)
           quote
               println("Runing outer2's code")
               @show $inner2()
               Two.@inner2($(e))
           end |> esc
       end
       end;

julia> let x = 1
           Two.@outer2($x)
       end
Expanding Two.@outer2
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
In Two.@inner2
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
Runing outer2's code
(Main.Two.inner2)() = 2
1
1 Like

This is very helpful and I had not appreciated how long a history these issues have had.

No, it’s just that I ran into some issue with composability using that approach. I can’t recall the exact issue now, it may be fixed now. I will try it out again, thanks for the reminder!