# Nested macros and esc

**URL:** <https://discourse.julialang.org/t/nested-macros-and-esc/47857>\
**Category:** General Usage\
**Created:** [October 6, 2020, 1:45pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857 "2020-10-06T13:45:07Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [October 6, 2020, 1:45pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/1 "2020-10-06T13:45:07Z")

</div>

I have an example of nesting macros that makes me think I’m confused about how hygiene works.

Here’s a minimal example:

```julia
macro inner(e)
    println("In @inner")
    dump(e)
    @assert isa(e, Expr) && e.head == :$
    esc(e.args[1])
end

macro outer(e)
    println("In @outer")
    dump(e)
    quote
        @inner($(e))
    end
end

let x = 1
    @inner($x)
end

let x = 1
    @outer($x)
end

```

This behaves as follows, which confuses me:

```julia
julia> macro inner(e)
           println("In @inner")
           dump(e)
           @assert isa(e, Expr) && e.head == :$
           esc(e.args[1])
       end
@inner (macro with 1 method)

julia> macro outer(e)
           println("In @outer")
           dump(e)
           quote
               @inner($(e))
           end
       end
@outer (macro with 1 method)

julia> let x = 1
           @inner($x)
       end
In @inner
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
1

julia> let x = 1
           @outer($x)
       end
In @outer
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
In @inner
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
ERROR: UndefVarError: x not defined
Stacktrace:
 [1] top-level scope at REPL[2]:5
 [2] top-level scope at REPL[4]:2

```

What I struggle to reconcile in my understanding is why the nested call to `@inner` seems to reflect the exact same input as the non-nested call sees, but the result differs. My mental model of nested macro calls is that the outer layer is evaluated first and then the inner layer runs as if the outer layer had never existed, but this example makes me question that model.

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [October 6, 2020, 1:57pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/2 "2020-10-06T13:57:05Z")

</div>

One question I have is whether I essentially need as many calls to `esc` as I have layers of nesting because the following does work for `@outer`, but fails for `@inner` (as I would expect):

```julia
macro inner(e)
    @assert isa(e, Expr) && e.head == :$
    esc(esc(e.args[1]))
end

macro outer(e)
    quote
        @inner($(e))
    end
end

let x = 1
    @inner($x)
end

let x = 1
    @outer($x)
end

```

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [October 6, 2020, 2:01pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/3 "2020-10-06T14:01:26Z")

</div>

I gave up on hygiene a long time ago and I have been much happier with manual `@gensym`s. What are you trying to do with these macros?

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [October 6, 2020, 2:04pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/4 "2020-10-06T14:04:13Z")

</div>

The use case I have is simplifying the code in Volcanito, where I’d like to have an outer macro generate repeated calls to an inner macro to make the sequence of macro expansions easier to debug.

In that example, you want to do something like the following:

```julia
@select(df, a, b, d = $x + a + b)
# ==>
Volcanito.Projection(
    df,
    (
        @expression(a),
        @expression(b),
        @expression(d = $x + a + b),
    )
)

```

I’m not sure I can make this work with gensyms given that I’m intentionally trying to capture local variables.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 6, 2020, 2:05pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/5 "2020-10-06T14:05:09Z")

</div>

One thing that’s also weird about this is that it’s happening at parse time, right?

```julia
julia> macro outer(e)
           println("In @outer")
           dump(e)
           quote
               @show x
               @inner($e)
           end
       end
@outer (macro with 1 method)

julia> let x = 1
           @outer($x)
       end
In @outer
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
In @inner
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
ERROR: UndefVarError: x not defined
Stacktrace:
 [1] top-level scope at show.jl:641
 [2] top-level scope at REPL[35]:5
 [3] top-level scope at REPL[36]:2

```

shouldn’t the `@show x` in `@outer` run here?

Very interested in this, because I am having a hard time justifying switching DataFramesMeta to `x` instead of `:x` in light of all the macro escaping headaches this will cause.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 6, 2020, 2:06pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/6 "2020-10-06T14:06:31Z")

</div>

> [@cstjean](#):
>
> gave up on hygiene a long time ago and I have been much happier with manual `@gensym` s.

How do you disable the hygiene rewriter? A practical example would be nice.

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [October 6, 2020, 2:08pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/7 "2020-10-06T14:08:02Z")

</div>

The issue you’re hitting makes more sense to me because the `x` shouldn’t be defined in the scope that exists while evaluating `@outer`, no?

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [October 6, 2020, 2:10pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/8 "2020-10-06T14:10:59Z")

</div>

To point a fine point on it, what really confuses me is the way this all interacts with `@macroexpand1` since `@macroexpand1` behaves like there’s no path dependence on a call to `@outer` having ever existed, but the need for double calling `esc` suggests there is path dependence:

```julia
julia> macro inner(e)
           @assert isa(e, Expr) && e.head == :$
           esc(e.args[1])
       end
@inner (macro with 1 method)

julia> macro outer(e)
           quote
               @inner($(e))
           end
       end
@outer (macro with 1 method)

julia> let x = 1
           @inner($x)
       end
1

julia> @macroexpand1 let x = 1
           @outer($x)
       end
:(let x = 1
      #= REPL[4]:2 =#
      begin
          #= REPL[2]:3 =#
          #= REPL[2]:3 =# @inner $(Expr(:$, :x))
      end
  end)

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 6, 2020, 2:17pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/9 "2020-10-06T14:17:57Z")

</div>

Why is that?

My mental model is that

```julia
macro outer(e)
    quote
        @show x
        @inner($(e))
    end
end

```

should be the same as

```julia
let x = 1
    @show x 
    @inner(x)
end

```

barring your weird escaping problem.

For clarification

```julia

julia> macro foo(e)
           quote
              @show z
              $e
          end
       end
@foo (macro with 1 method)

julia> let z = 5
       @foo y = 5
       end
z = 9
5

```

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [October 6, 2020, 2:23pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/10 "2020-10-06T14:23:37Z")

</div>

You’re right, I misread your example and thought the `@show` wasn’t nested inside the quote block.

Don’t you need something `@show $(esc(x))` to avoid a gensym operation on `x` though?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 6, 2020, 2:24pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/11 "2020-10-06T14:24:55Z")

</div>

Yes, that was just an example to show that I think the behavior is odd, fewer chance for bugs using an `x` literal. You would definitely need that to make it a workable macro.

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [October 6, 2020, 2:25pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/12 "2020-10-06T14:25:21Z")

</div>

> [@pdeffebach](#):
>
> For clarification
> 
> ```julia
> julia> macro foo(e)
> quote
> @show z
> $e
> end
> end
> @foo (macro with 1 method)
> 
> julia> let z = 5
> @foo y = 5
> end
> z = 9
> 5
> 
> ```

This example doesn’t work for me. Did you have a global `z = 9` somewhere?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 6, 2020, 2:27pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/13 "2020-10-06T14:27:10Z")

</div>

You are totally right. My mental model for macros needs work.

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [October 6, 2020, 2:34pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/14 "2020-10-06T14:34:08Z")

</div>

I suspect this issue comes down to distinguishing the following two possibilities.

(1) Nested macros are partially expanded outer-to-inner, but the expansions occur without any hygiene steps taking place until all partial expansions are finished. So you get a sequence like:

- Expansion of `@outer` without hygiene
- Expansion of `@inner` without hygiene
- Hygiene for `@inner`
- Hygiene for `@outer`

(2) Nested macros are fully expanded outer-to-inner including hygiene:

- Expansion of `@outer` without hygiene
- Hygiene for `@outer`
- Expansion of `@inner` without hygiene
- Hygiene for `@inner`

I had believed (2) is what happens, but I think this example suggests (1) is what happens.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [October 6, 2020, 2:36pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/15 "2020-10-06T14:36:16Z")

</div>

`esc` by default “captures the local variables”.

```julia
julia> module Volcanito
       using MacroTools

       macro select(df, exprs...)
           get_var(e) = @capture(e, lhs_ = _) ? lhs : e
           vars = map(get_var, exprs)
           esc(quote 
           $Volcanito.Projection($df, tuple($([:($Volcanito.@expression($e, $(vars...))) for e in exprs]...)))
           end)
       end

       macro expression(eq, vars...)
           esc(Expr(:let, Expr(:block, [Expr(:(=), v, 55) for v in vars]...), eq))
       end
       end
Main.Volcanito

julia> MacroTools.striplines(@macroexpand1 Volcanito.@select df a b d = x + a + b)
quote
    (Main.Volcanito).Projection(df, 
    tuple((Main.Volcanito).@expression(a, a, b, d), 
          (Main.Volcanito).@expression(b, a, b, d), 
          (Main.Volcanito).@expression(d = x + a + b, a, b, d)))
end

julia> MacroTools.striplines(@macroexpand1 Volcanito.@expression(d = x + a + b, a, b, d))
:(let a = 55, b = 55, d = 55 # could be `a = df.a` etc.
      d = x + a + b # thus the "local" x will be used
  end)

```

looks like along the lines of what you need, but I don’t know what semantics you want from `$x` and `@expression` .

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [October 6, 2020, 2:40pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/16 "2020-10-06T14:40:11Z")

</div>

We already had that conversation 🙂[Looking for good macro examples - #9 by cstjean](https://discourse.julialang.org/t/looking-for-good-macro-examples/450/9)

Is it unclear?

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [October 6, 2020, 4:24pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/17 "2020-10-06T16:24:58Z")

</div>

You do need some escaping at each level, `@outer2` and `@outer3` are different ways you might think to do it. But `@outer3` doesn’t work, because it passes `@inner` an expression which doesn’t have the `$` nakedly visible. There is a helpful issue about this: [https://github.com/JuliaLang/julia/issues/37691](https://github.com/JuliaLang/julia/issues/37691).

The problem with `@outer2` is that that everything is in the caller’s scope, including the inner macro. When it’s in another module, it needs `Two.@inner2`. (Or better, you could interpolate, which is just `$` for functions, but is apparently tricker for macros.)

```julia
julia> macro inner(e)
           println("In @inner")
           dump(e)
           @assert isa(e, Expr) && e.head == :$
           esc(e.args[1])
       end
@inner (macro with 1 method)

julia> macro outer2(e)
           println("In @outer")
           dump(e)
           quote
               @inner($(e))
           end |> esc
       end
@outer2 (macro with 1 method)

julia> let x = 1
           @inner($x)
       end
In @inner
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
1

julia> let x = 1
           @outer2($x)
       end
In @outer
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
In @inner
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
1

julia> macro outer3(e)
           println("In @outer")
           dump(e)
           quote
               @inner($(esc(e)))
           end
       end
@outer3 (macro with 1 method)

julia> let x = 1
           @outer3($x)
       end
In @outer
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
In @inner
Expr
  head: Symbol escape
  args: Array{Any}((1,))
    1: Expr
      head: Symbol $
      args: Array{Any}((1,))
        1: Symbol x
ERROR: LoadError: AssertionError: e isa Expr && e.head == :$
Stacktrace:
 [1] @inner(::LineNumberNode, ::Module, ::Any) at ./REPL[4]:4
in expression starting at REPL[8]:6245:

julia> module Two
       macro inner2(e)
           println("In Two.@inner2")
           dump(e)
           @assert isa(e, Expr) && e.head == :$
           esc(e.args[1])
       end
       function inner2()
         2
       end

       macro outer2(e)
           println("Expanding Two.@outer2")
           dump(e)
           quote
               println("Runing outer2's code")
               @show $inner2()
               Two.@inner2($(e))
           end |> esc
       end
       end;

julia> let x = 1
           Two.@outer2($x)
       end
Expanding Two.@outer2
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
In Two.@inner2
Expr
  head: Symbol $
  args: Array{Any}((1,))
    1: Symbol x
Runing outer2's code
(Main.Two.inner2)() = 2
1

```

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [October 6, 2020, 4:35pm UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/18 "2020-10-06T16:35:39Z")

</div>

This is very helpful and I had not appreciated how long a history these issues have had.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 7, 2020, 8:10am UTC](https://discourse.julialang.org/t/nested-macros-and-esc/47857/19 "2020-10-07T08:10:33Z")

</div>

No, it’s just that I ran into some issue with composability using that approach. I can’t recall the exact issue now, it may be fixed now. I will try it out again, thanks for the reminder!