Escaping of nested blocks

#1

Hi,

I am trying to learn to work with macros and ASTs, but am running into a bit of confusion regarding escaping. I haven’t found a lot of documentation regarding this, but if there is an authorative source, I would appreciate any pointers.

I wrote some example code which looks similar to this:

``````function op_helper(ee::Expr)
...
end

macro op(ee::Expr)
\$(op_helper(ee))
end
``````

Inside of `op_helper`, I recursively walk the AST and do some very simple transformations (I am effectively trying to just return identity) on the expressions.

I am running into an interesting conundrum, which is challenging my understanding of `esc` in recursive contexts.

Consider the following test function:

``````function test(a)
x = @op  a + 1 #1
@op y = a + 2 #2

@op begin  #3
z = a + 3 #4
end
end
``````

I am finding the following:

1. is straightforward, and my code simply recursively traverses the AST, only escaping
``````# ...
if typeof(ee) == Symbol
esc(ee)
elseif #...
``````
1. Seems only a little bit more work is required (note for simplicity, I’m not recursing on the LHS)
``````if typeof(ee) == Expr && ee.head == :(=)
esc(:(\$(ee.args[1]) = \$(op_helper(ee.args[2])))
elseif #...
``````
1. It is this case which is intriguing.
``````if typeof(ee) == Expr && ee.head == :block
parts = [op_helper(elem) for elem in ee.args]
Expr(:block, parts...) #5
elseif #...
``````

It appears that for this to work, I actually need to write `esc(Expr(:block, parts...))` in #5; however, this then causes problems with #4, which now doesn’t want to be escaped, unlike the recursive code path for #2.

Is what I’m describing correct, or am I missing something?

Thank you,

Tom

#2

This looks wrong. I assume you mean `op_helper(ee)`.

You must escape the LHS.

What do you mean by not working?

#3

Yes. Typed the wrong thing, sorry.

I think it is escaped; I just said I didn’t recurse.

The compiler complains:

Anyway, I think I’ve figured it out. I just generate the AST using op_helper(), and then just `esc`'ape the resulting expression in `op`.

Thanks for the help,

Tom

#4

At least the code you paste didn’t.

Do note that this turns off macro hygiene and you can have accidental reference to user local variable that’s not intended.

#5

Hi Yuyichao,

Thank you for your time. I was wondering if you could clarify your comment for me, because I’m afraid I may not be understanding something correctly.

I wrote

``````if typeof(ee) == Expr && ee.head == :(=)
esc(:(\$(ee.args[1]) = \$(op_helper(ee.args[2])))
elseif #...
``````

Say, in the case of #2, I would have expected this to expand to something like

esc(:(y = a + 2))

You assert that the code is not escaped. Would you be able to explain? I’ve copied the entire example code below for reference:

``````function op_helper(ee::Expr)
if typeof(ee) == Symbol
esc(ee)
elseif typeof(ee) == Expr && ee.head == :(=)
esc(:(\$(ee.args[1]) = \$(op_helper(ee.args[2]))))
elseif typeof(ee) == Expr && ee.head == :block
parts = [op_helper(elem) for elem in ee.args]
Expr(:block, parts...) #5
elseif typeof(ee) == Expr && ee.head == :call
ee
# Nothing
else
error("Not handled", ee)
end
end

macro op(ee::Expr)
res = op_helper(ee)
println(res)
res
end

function test(a)
x = @op a + 1 #1
@op y = a + 2 #2

@op begin  #3
z = a + 3 #4
end

return (x, y, z)
end

println(test(3))
``````

Yes, that was partially my worry. It appears that my new code is able to avoid it to some degree, at least.

Thanks,

Tom

#6

I missed the `esc` around the whole thing. That does escape the LHS but is wrong. In general, you must escape each piece of user input once and exactly once. A slightly better implementation would be

``````julia> function op_helper(ee::ANY)
if isa(ee, Symbol)
esc(ee)
elseif isa(ee, Expr) && ee.head == :(=)
:(\$(esc(ee.args[1])) = \$(op_helper(ee.args[2])))
elseif isa(ee, Expr) && ee.head == :block
parts = [op_helper(elem) for elem in ee.args]
Expr(:block, parts...) #5
elseif isa(ee, Expr) && ee.head == :call
esc(ee)
elseif isa(ee, Expr) && ee.head == :line
esc(ee)
else
error("Not handled", ee)
end
end
op_helper (generic function with 1 method)

julia> macro op(ee::Expr)
res = op_helper(ee)
println(res)
res
end
@op (macro with 1 method)

julia> function test(a)
x = @op a + 1 #1
@op y = a + 2 #2

@op begin  #3
z = a + 3 #4
end

return (x, y, z)
end
\$(Expr(:escape, :(a + 1)))
\$(Expr(:escape, :y)) = \$(Expr(:escape, :(a + 2)))
begin
\$(Expr(:escape, :( # REPL[3], line 6:)))
\$(Expr(:escape, :z)) = \$(Expr(:escape, :(a + 3)))
end
test (generic function with 1 method)

julia> println(test(3))
(4,5,6)
``````

Note that handling `Symbol`s but not other literals is generally problematic. (although actually neither is handled in either version)

#7

Hi yuyichao,

Thank you very much for the example code; that helped a lot. I actually had something almost identical to what you wrote to begin with, but couldn’t get it to work. The main difference was my initial attempt of the assignment, which I couldn’t get working

``````:(esc(\$(ee.args[1])) = \$(op_helper(ee.args[2])))
``````

``````:(\$(esc(ee.args[1])) = \$(op_helper(ee.args[2])))
``````

This was actually the reason why I wrapped the whole expression in `esc`, instead of just the destination.

Let me see if I can my code going now.

Thank you for your help, I really appreciate it.

Kind regards,

Tom

#8

I wouldn’t disagree that expression interpolation can easily give you multiple levels of parenthesis. At least you can always assign it to a variable (e.g. `earg1 = esc(ee.args[1])`) and in this sense it’s not too different from an expression with nested function calls.

#9

Hi yuyichao,

I wasn’t really referring the parenthesis, but more to the

`\$(esc(<expr>))` vs `esc(\$(<expr>))`.

I can’t say I fully see the difference (other than one working and the other not).

Thanks,

Tom

#10

I know. I’m just saying that the many level of parentheses makes it harder to tell the difference.

`\$` is the interpolation so in `\$(esc(<expr>))` it interpolated an escaped expression (what you want) and in `esc(\$(<expr>))` it interpolate the original expression into a function call of `esc`.