Calling a macro from within a macro, revisited

question

#1

Hi all,

I have a question: am i seeing a bug in the macro expander, or am I misunderstanding its usage?

I’m implementing a macro which delegates (most of) its work to another macro, and I’m running into some issues around escaping. Here is a minimal version which I think should work, based on the discussion in this related issue, but isn’t:

module Retest

using Test

macro retestset1(args...)
    :(@testset($((esc(a) for a in args)...)))  # should work?
end

macro retestset2(args...)
    esc(:($Test.@testset($(args...))))         # works but eww?
end

end

Note that this is a completely generic problem—the fact that @testset is the macro being delegated to is just a detail, and we should be able to “wrap” any macro we want without changes to that macro’s implementation.

Here’s the error trying to invoke @retesetse1:

julia> Retest.@retestset1 begin end
ERROR: LoadError: Expected begin/end block or for loop as argument to @testset
Stacktrace:
 [1] @testset(::LineNumberNode, ::Module, ::Vararg{Any,N} where N) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1041
in expression starting at /Volumes/GoogleDrive/My Drive/Retest/src/Retest.jl:6

A little investigation indicates that the issue is that the expressions passed to @testset still contain escape nodes, which it chokes on. Here’s an even simpler example illustrating the phenomenon:

julia> macro foo(arg)
           @show arg
       end
@foo (macro with 1 method)

julia> macro bar(arg)
           :(@foo($(esc(arg))))
       end
@bar (macro with 1 method)

julia> @bar 1+1
arg = :($(Expr(:escape, :(1 + 1))))
2

Is this how the macroexpander is supposed to operate? If so, how do I reconcile this with the advice from @Tamas_Papp in Call a macro within a macro?

Invoking @retestset2 on the other hand works fine, including picking up variables from the calling environment:

julia> let x=1
       Retest.@retestset2 begin @assert x == 1 end
       end
Test Summary: |
test set      | No tests
Test.DefaultTestSet("test set", Any[], 0, false)

But it seems like a bit of a kludge. Any help appreciated.


#2

I suggest writing the macros as just wrappers around functions (that take an expression as argument and return an expression), and calling the respective functions instead.


#3

In general, I completely agree. In this case, though, the OP’s goal was to call @testset; is there a good function equivalent to that?

Either way, I think as OP says, the escape nodes should’ve been removed by the macro expander before calling the macro returned in the quoted code. Seeing :($(Expr(:escape, :(1 + 1))) being provided as the input to @foo() does seem like it’s not behaving as expected, and might be a bug in the macro expander?


#4

Thanks @dpsanders, but I don’t quite follow, can you show me a working example of what you mean? Also, can anyone comment on the @foo/@bar behavior, is that working as expected?


#5

@NHDaly is right that I spoke a bit too fast, since you’re wrapping @testset.

But in my own code, I would define something like

julia> macro hello(ex)
           return hello(ex)
       end
@hello (macro with 1 method)

julia> hello(ex) = ex
hello (generic function with 1 method)

julia> @hello 3+4
7

Here the macro exists just to pick up what the user types into an expression, and pass this expression over to a function that actually does the hard work.

If this is the case then you can just call the corresponding function instead inside your other macro.
The case of @testset seems to be a bit harder, though.


#6

esc please…


#7

(@yuyichao, do you mean @dpsanders’s example should include esc? The OP, @tjgreen,'s examples all have appropriate escaping, I think?)


#8

Correct. ( I thought I pressed the button to reply to that post) edit: hmmm seems that it might not be working in my phone.

Also, my point is that esc is the whole reason nesting is a problem at all so in additional to the normal reason of not putting out misleading info it s especially important for this issue to include it.


#9

Right, my situation is different, and I’m really asking a more general question: given a macro @f in some other library, how do you write a macro @g which behaves exactly like @f (and is implemented in terms of @f)? And, if my first attempt at this for the case of @testset (@retestset1) isn’t working, is this because of a bug in the macroexpander, or because I wrote it wrong, or something else?


#10

What you want is to define it like this

macro retestset1(arg)
    :(@testset$(Expr(:block,esc.(arg.args)...))
end

then

julia> @retestset1 begin
           @test 1 == 1.0
       end
Test Summary: | Pass  Total
test set      |    1      1
Test.DefaultTestSet("test set", Any[], 1, false)

Your issue is that you are not using the Julia abstract syntax tree correctly.


#11

@chakravala Hmm, so this works, but only for one form of the arguments to @testset (which can also take a for-loop instead of a block, and also has an optional name parameter). Can you think of a generic implementation that doesn’t make any assumptions about the shape of the underlying macro’s arguments?


#12

Yes,

macro retestset1(arg)
    :(@testset$(Expr(arg.head,esc.(arg.args)...))
end

simply replace :block with arg.head


#13

There is no solution to this problem, it’s still an open issue: https://github.com/JuliaLang/julia/issues/23221

IIRC the only completely general workaround is for your macros to understand that they may see Expr(:escape) and in some cases to partially understand lowering :frowning:

See also https://github.com/c42f/FastClosures.jl/blob/c4c8cf2d46024a875d6ab967b207709218015dd0/src/FastClosures.jl#L138


#14

Awesome, thanks for the authoritative answer @Chris_Foster!