I have an example of nesting macros that makes me think I’m confused about how hygiene works.
Here’s a minimal example:
macro inner(e)
println("In @inner")
dump(e)
@assert isa(e, Expr) && e.head == :$
esc(e.args[1])
end
macro outer(e)
println("In @outer")
dump(e)
quote
@inner($(e))
end
end
let x = 1
@inner($x)
end
let x = 1
@outer($x)
end
This behaves as follows, which confuses me:
julia> macro inner(e)
println("In @inner")
dump(e)
@assert isa(e, Expr) && e.head == :$
esc(e.args[1])
end
@inner (macro with 1 method)
julia> macro outer(e)
println("In @outer")
dump(e)
quote
@inner($(e))
end
end
@outer (macro with 1 method)
julia> let x = 1
@inner($x)
end
In @inner
Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
1
julia> let x = 1
@outer($x)
end
In @outer
Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
In @inner
Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
ERROR: UndefVarError: x not defined
Stacktrace:
[1] top-level scope at REPL[2]:5
[2] top-level scope at REPL[4]:2
What I struggle to reconcile in my understanding is why the nested call to @inner seems to reflect the exact same input as the non-nested call sees, but the result differs. My mental model of nested macro calls is that the outer layer is evaluated first and then the inner layer runs as if the outer layer had never existed, but this example makes me question that model.
1 Like
One question I have is whether I essentially need as many calls to esc as I have layers of nesting because the following does work for @outer, but fails for @inner (as I would expect):
macro inner(e)
@assert isa(e, Expr) && e.head == :$
esc(esc(e.args[1]))
end
macro outer(e)
quote
@inner($(e))
end
end
let x = 1
@inner($x)
end
let x = 1
@outer($x)
end
I gave up on hygiene a long time ago and I have been much happier with manual @gensyms. What are you trying to do with these macros?
1 Like
The use case I have is simplifying the code in Volcanito, where I’d like to have an outer macro generate repeated calls to an inner macro to make the sequence of macro expansions easier to debug.
In that example, you want to do something like the following:
@select(df, a, b, d = $x + a + b)
# ==>
Volcanito.Projection(
df,
(
@expression(a),
@expression(b),
@expression(d = $x + a + b),
)
)
I’m not sure I can make this work with gensyms given that I’m intentionally trying to capture local variables.
One thing that’s also weird about this is that it’s happening at parse time, right?
julia> macro outer(e)
println("In @outer")
dump(e)
quote
@show x
@inner($e)
end
end
@outer (macro with 1 method)
julia> let x = 1
@outer($x)
end
In @outer
Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
In @inner
Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
ERROR: UndefVarError: x not defined
Stacktrace:
[1] top-level scope at show.jl:641
[2] top-level scope at REPL[35]:5
[3] top-level scope at REPL[36]:2
shouldn’t the @show x in @outer run here?
Very interested in this, because I am having a hard time justifying switching DataFramesMeta to x instead of :x in light of all the macro escaping headaches this will cause.
1 Like
How do you disable the hygiene rewriter? A practical example would be nice.
1 Like
The issue you’re hitting makes more sense to me because the x shouldn’t be defined in the scope that exists while evaluating @outer, no?
To point a fine point on it, what really confuses me is the way this all interacts with @macroexpand1 since @macroexpand1 behaves like there’s no path dependence on a call to @outer having ever existed, but the need for double calling esc suggests there is path dependence:
julia> macro inner(e)
@assert isa(e, Expr) && e.head == :$
esc(e.args[1])
end
@inner (macro with 1 method)
julia> macro outer(e)
quote
@inner($(e))
end
end
@outer (macro with 1 method)
julia> let x = 1
@inner($x)
end
1
julia> @macroexpand1 let x = 1
@outer($x)
end
:(let x = 1
#= REPL[4]:2 =#
begin
#= REPL[2]:3 =#
#= REPL[2]:3 =# @inner $(Expr(:$, :x))
end
end)
Why is that?
My mental model is that
macro outer(e)
quote
@show x
@inner($(e))
end
end
should be the same as
let x = 1
@show x
@inner(x)
end
barring your weird escaping problem.
For clarification
julia> macro foo(e)
quote
@show z
$e
end
end
@foo (macro with 1 method)
julia> let z = 5
@foo y = 5
end
z = 9
5
1 Like
You’re right, I misread your example and thought the @show wasn’t nested inside the quote block.
Don’t you need something @show $(esc(x)) to avoid a gensym operation on x though?
1 Like
Yes, that was just an example to show that I think the behavior is odd, fewer chance for bugs using an x literal. You would definitely need that to make it a workable macro.
1 Like
This example doesn’t work for me. Did you have a global z = 9 somewhere?
You are totally right. My mental model for macros needs work.
1 Like
I suspect this issue comes down to distinguishing the following two possibilities.
(1) Nested macros are partially expanded outer-to-inner, but the expansions occur without any hygiene steps taking place until all partial expansions are finished. So you get a sequence like:
- Expansion of
@outer without hygiene
- Expansion of
@inner without hygiene
- Hygiene for
@inner
- Hygiene for
@outer
(2) Nested macros are fully expanded outer-to-inner including hygiene:
- Expansion of
@outer without hygiene
- Hygiene for
@outer
- Expansion of
@inner without hygiene
- Hygiene for
@inner
I had believed (2) is what happens, but I think this example suggests (1) is what happens.
1 Like
esc by default “captures the local variables”.
julia> module Volcanito
using MacroTools
macro select(df, exprs...)
get_var(e) = @capture(e, lhs_ = _) ? lhs : e
vars = map(get_var, exprs)
esc(quote
$Volcanito.Projection($df, tuple($([:($Volcanito.@expression($e, $(vars...))) for e in exprs]...)))
end)
end
macro expression(eq, vars...)
esc(Expr(:let, Expr(:block, [Expr(:(=), v, 55) for v in vars]...), eq))
end
end
Main.Volcanito
julia> MacroTools.striplines(@macroexpand1 Volcanito.@select df a b d = x + a + b)
quote
(Main.Volcanito).Projection(df,
tuple((Main.Volcanito).@expression(a, a, b, d),
(Main.Volcanito).@expression(b, a, b, d),
(Main.Volcanito).@expression(d = x + a + b, a, b, d)))
end
julia> MacroTools.striplines(@macroexpand1 Volcanito.@expression(d = x + a + b, a, b, d))
:(let a = 55, b = 55, d = 55 # could be `a = df.a` etc.
d = x + a + b # thus the "local" x will be used
end)
looks like along the lines of what you need, but I don’t know what semantics you want from $x and @expression .
2 Likes
We already had that conversation 
Looking for good macro examples - #9 by cstjean
Is it unclear?
1 Like
You do need some escaping at each level, @outer2 and @outer3 are different ways you might think to do it. But @outer3 doesn’t work, because it passes @inner an expression which doesn’t have the $ nakedly visible. There is a helpful issue about this: https://github.com/JuliaLang/julia/issues/37691.
The problem with @outer2 is that that everything is in the caller’s scope, including the inner macro. When it’s in another module, it needs Two.@inner2. (Or better, you could interpolate, which is just $ for functions, but is apparently tricker for macros.)
julia> macro inner(e)
println("In @inner")
dump(e)
@assert isa(e, Expr) && e.head == :$
esc(e.args[1])
end
@inner (macro with 1 method)
julia> macro outer2(e)
println("In @outer")
dump(e)
quote
@inner($(e))
end |> esc
end
@outer2 (macro with 1 method)
julia> let x = 1
@inner($x)
end
In @inner
Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
1
julia> let x = 1
@outer2($x)
end
In @outer
Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
In @inner
Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
1
julia> macro outer3(e)
println("In @outer")
dump(e)
quote
@inner($(esc(e)))
end
end
@outer3 (macro with 1 method)
julia> let x = 1
@outer3($x)
end
In @outer
Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
In @inner
Expr
head: Symbol escape
args: Array{Any}((1,))
1: Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
ERROR: LoadError: AssertionError: e isa Expr && e.head == :$
Stacktrace:
[1] @inner(::LineNumberNode, ::Module, ::Any) at ./REPL[4]:4
in expression starting at REPL[8]:6245:
julia> module Two
macro inner2(e)
println("In Two.@inner2")
dump(e)
@assert isa(e, Expr) && e.head == :$
esc(e.args[1])
end
function inner2()
2
end
macro outer2(e)
println("Expanding Two.@outer2")
dump(e)
quote
println("Runing outer2's code")
@show $inner2()
Two.@inner2($(e))
end |> esc
end
end;
julia> let x = 1
Two.@outer2($x)
end
Expanding Two.@outer2
Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
In Two.@inner2
Expr
head: Symbol $
args: Array{Any}((1,))
1: Symbol x
Runing outer2's code
(Main.Two.inner2)() = 2
1
1 Like
This is very helpful and I had not appreciated how long a history these issues have had.
No, it’s just that I ran into some issue with composability using that approach. I can’t recall the exact issue now, it may be fixed now. I will try it out again, thanks for the reminder!