Calling a macro from within a macro, revisited

General Usage
1

Hi all,

I have a question: am i seeing a bug in the macro expander, or am I misunderstanding its usage?

I’m implementing a macro which delegates (most of) its work to another macro, and I’m running into some issues around escaping. Here is a minimal version which I think should work, based on the discussion in this related issue, but isn’t:

module Retest

using Test

macro retestset1(args...)
    :(@testset($((esc(a) for a in args)...)))  # should work?
end

macro retestset2(args...)
    esc(:($Test.@testset($(args...))))         # works but eww?
end

end

Note that this is a completely generic problem—the fact that @testset is the macro being delegated to is just a detail, and we should be able to “wrap” any macro we want without changes to that macro’s implementation.

Here’s the error trying to invoke @retesetse1:

julia> Retest.@retestset1 begin end
ERROR: LoadError: Expected begin/end block or for loop as argument to @testset
Stacktrace:
 [1] @testset(::LineNumberNode, ::Module, ::Vararg{Any,N} where N) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1041
in expression starting at /Volumes/GoogleDrive/My Drive/Retest/src/Retest.jl:6

A little investigation indicates that the issue is that the expressions passed to @testset still contain escape nodes, which it chokes on. Here’s an even simpler example illustrating the phenomenon:

julia> macro foo(arg)
           @show arg
       end
@foo (macro with 1 method)

julia> macro bar(arg)
           :(@foo($(esc(arg))))
       end
@bar (macro with 1 method)

julia> @bar 1+1
arg = :($(Expr(:escape, :(1 + 1))))
2

Is this how the macroexpander is supposed to operate? If so, how do I reconcile this with the advice from @Tamas_Papp in Call a macro within a macro - #6 by findmyway?

Invoking @retestset2 on the other hand works fine, including picking up variables from the calling environment:

julia> let x=1
       Retest.@retestset2 begin @assert x == 1 end
       end
Test Summary: |
test set      | No tests
Test.DefaultTestSet("test set", Any[], 0, false)

But it seems like a bit of a kludge. Any help appreciated.

2

I suggest writing the macros as just wrappers around functions (that take an expression as argument and return an expression), and calling the respective functions instead.

2 Likes
3

In general, I completely agree. In this case, though, the OP’s goal was to call @testset; is there a good function equivalent to that?

Either way, I think as OP says, the escape nodes should’ve been removed by the macro expander before calling the macro returned in the quoted code. Seeing :($(Expr(:escape, :(1 + 1))) being provided as the input to @foo() does seem like it’s not behaving as expected, and might be a bug in the macro expander?

2 Likes
4

Thanks @dpsanders, but I don’t quite follow, can you show me a working example of what you mean? Also, can anyone comment on the @foo/@bar behavior, is that working as expected?

5

@NHDaly is right that I spoke a bit too fast, since you’re wrapping @testset.

But in my own code, I would define something like

julia> macro hello(ex)
           return hello(ex)
       end
@hello (macro with 1 method)

julia> hello(ex) = ex
hello (generic function with 1 method)

julia> @hello 3+4
7

Here the macro exists just to pick up what the user types into an expression, and pass this expression over to a function that actually does the hard work.

If this is the case then you can just call the corresponding function instead inside your other macro.
The case of @testset seems to be a bit harder, though.

1 Like
6

esc please…

2 Likes
7

(@yuyichao, do you mean @dpsanders’s example should include esc? The OP, @tjgreen,'s examples all have appropriate escaping, I think?)

1 Like
8

Correct. ( I thought I pressed the button to reply to that post) edit: hmmm seems that it might not be working in my phone.

Also, my point is that esc is the whole reason nesting is a problem at all so in additional to the normal reason of not putting out misleading info it s especially important for this issue to include it.

1 Like
9

Right, my situation is different, and I’m really asking a more general question: given a macro @f in some other library, how do you write a macro @g which behaves exactly like @f (and is implemented in terms of @f)? And, if my first attempt at this for the case of @testset (@retestset1) isn’t working, is this because of a bug in the macroexpander, or because I wrote it wrong, or something else?

1 Like
10

What you want is to define it like this

macro retestset1(arg)
    :(@testset$(Expr(:block,esc.(arg.args)...))
end

then

julia> @retestset1 begin
           @test 1 == 1.0
       end
Test Summary: | Pass  Total
test set      |    1      1
Test.DefaultTestSet("test set", Any[], 1, false)

Your issue is that you are not using the Julia abstract syntax tree correctly.

11

@chakravala Hmm, so this works, but only for one form of the arguments to @testset (which can also take a for-loop instead of a block, and also has an optional name parameter). Can you think of a generic implementation that doesn’t make any assumptions about the shape of the underlying macro’s arguments?

12

Yes,

macro retestset1(arg)
    :(@testset$(Expr(arg.head,esc.(arg.args)...))
end

simply replace :block with arg.head

1 Like
13

There is no solution to this problem, it’s still an open issue: macro hygine escape stripped late for macros calling macros · Issue #23221 · JuliaLang/julia · GitHub

IIRC the only completely general workaround is for your macros to understand that they may see Expr(:escape) and in some cases to partially understand lowering :frowning:

See also FastClosures.jl/FastClosures.jl at c4c8cf2d46024a875d6ab967b207709218015dd0 · c42f/FastClosures.jl · GitHub

5 Likes
14

Awesome, thanks for the authoritative answer @c42f!

15

I know this is an old thread, but I’ve ended up back here a few times, so I think it might be useful to add on. I saw that the @simd macro uses an esc() outside the entire expression like this: https://github.com/JuliaLang/julia/blob/1b93d53fc4bb59350ada898038ed4de2994cce33/base/simdloop.jl#L127-L129

Following a similar pattern seems to work for wrapping @simd. I have a macro

macro innerloop(ex)
    return esc( :( @simd ivdep $ex ) )
end

so I can change the macros applied to inner loops throughout my code. It seems to be working at first glance. Not sure if it’s a general solution, and I confess I’m fairly new to Julia and not really comfortable with how macros/metaprogramming work - should this work? Anyone have comments?

2 Likes
16

This “works” by disabling all macro hygiene. In this case you’re assuming that the module which uses @innerloop ex also has the macro @simd available. But this might not always be the case.

Disabling hygiene can be an acceptable workaround for macros which aren’t meant to be reused in other modules, or generally where the macro author is also the user of the macro so they can manually ensure hygiene at the use site.

But in general this isn’t a great idea because it leads to symbols being looked up in the wrong scope which can be very confusing for macro users.

4 Likes