@generated inside macro: who's interpolating?

General Usage
1

I have a macro to generate code for “many” binary operations. The macro generates several functions, including a @generated function so that I can generate code for any combination of subtypes. A crash reproduction of my code looks like this

macro Op2(OP,A,B)
    return quote
        @generated function $OP(a::MyType{Ta},b::MyType{Tb}) where{Ta,Tb}
            return quote 
                MyType{Ta}($OP(a.x*$A,b.x*$B))  
            end    
        end
    end
end

struct MyType{T}
    x::Float64
end

and it makes a big fireball

@Op2(Base.:(+),1.,2.)
x = MyType{1}(3.)
y = MyType{2}(2.)
x+y
ERROR: LoadError: UndefVarError: A not defined

I am surprised: I would expect the macro’s outermost interpolated quote to replace OP, A and B, return code to the effect of

@generated function Base.:(+)(a::MyType{Ta},b::MyType{Tb}) where{Ta,Tb}
    return quote 
        MyType{Ta}(Base.:(+)(a.x*1.,b.x*2.))  
    end    
end

but this is not what seems to happen: I can’t say I can cut through the fog returned by @macroexpand, but it still mentions OP, A and B. What did I miss?

Code that generates code is bad for the brain, but code that generates code that generates code? Is it really wise? :roll_eyes: :smile:

2

You need MyType{Ta}($$OP(a.x*$$A,b.x*$$B)), and to esc what the macro returns. But might be simpler just to loop over @eval statements.

1 Like
3

FWIW after wasting too many hours debugging nested quotes, I now use Expr in those case. Eg.

return quote
    @generate function ...
         return Expr(:call, ...)
    end
end

No more surprises.

4

CSTJean, I can see why you say that !!!

5

Sorry, I do not understand

  • why double interpolate
  • I have a faint grasp of esc, but escaping what the macro return?

And yes, loop over eval, or any other way of not nesting @generate into macro, might be the smart way!

6

Cedric,

You helped me on the track. I rewrote my macro to avoid nested quotes:

macro Op2(OP,A,B)
    ex = quote
        MyType{Ta}($OP(a.x*$A,b.x*$B))
    end
    return quote
        @generated function $OP(a::MyType{Ta},b::MyType{Tb}) where{Ta,Tb}
            return $ex
        end
    end
end

and now

@macroexpand @Op2(Base.:(+),1.,2.)

returns (with my simplifying the output for readability)

    function (Main.Base).:+(a::Main.MyType{Ta}, b::Main.MyType{Tb}) where {Ta, Tb}
            return begin
                    Main.MyType{Ta}((Main.Base).:+(a.x * 1.0, b.x * 1.0))
                end
    end

which is what I aimed for - except I don’t see “@generated”. And the next puzzle is that

x = MyType{1}(3.)
y = MyType{2}(2.)
x+y

returns

ERROR: MethodError: no method matching +(::MyType{1}, ::MyType{2})

I will look at how to avoid having, not only quote within quote, but @generated within quote.

A little later. I suspect the problem is that my generated function, returns $ex. ex is an expression I want to interpolate in an expression (@generated function…) quoted in a macro. :($ex) reduces to ex, and the generated function then is compiled with the code ‘ex’.

Direct manipulation of Expr seems the only way out - or workarounds.

Thank you all!

7

You need an extra level of quoting around ex, so ex is interpolated as a quoted expression, not as regular code that gets run as part of the function. Since since the interpolation inside ex already gets resolved in the macro, you can just use QuoteNode for that, if ex still contained interpolation that is supposed to be resolved in the scope of the generated function body, you should use Meta.quot. You still need to esc at least the arguments Ta, a and b of the generated function, because macro hygiene will try to scramble those argument names, but leaves quoted expressions alone, so @generated doesn’t know any arguments Ta, a and b and therefore will throw an UndefVarError. In this case, you can also just esc the whole expression, since Ta, a and b are local variables anyways. Putting this all together, we get:

macro Op2(OP,A,B)
    ex = quote
        MyType{Ta}($OP(a.x*$A,b.x*$B))
    end
    return esc(quote
        @generated function $OP(a::MyType{Ta},b::MyType{Tb}) where{Ta,Tb}
            return $(QuoteNode(ex))
        end
    end)
end

, which should do what you want.

8

“Just works”. I would never have made that up myself, thank you. And I have to think hard and long about your explanation.

9

I must admit I still do not understand Simeon’s explanation - no worry, I will just have to spend some quality time on this (and examples help a lot!). But of course, now I want more: compared to the previous “exercise”, I now want the body of the generated function to make a computation and interpolate the result T in the code it returns. My code doesn’t work (I have $T and $OP side by side which I intend to be interpolated by the @generated function and the macro respectively), but you get the gist:

macro Op2(OP,A,B)
    ex = quote
        MyType{$T}($OP(a.x*$A,b.x*$B))
    end
    return esc(quote
        @generated function $OP(a::MyType{Ta},b::MyType{Tb}) where{Ta,Tb}
            T = max(Ta,Tb)
            return $(QuoteNode(ex))
        end
    end)
end

Simeon mentionned Meta.quot, and I imagine he was foreseeing exactly my situation. Still my attempts failed. If anyone can give me a nudge…

Logging out for the night.

10

I think the easiest solution here is to not define ex separately, but inside the generated function. You can then use $$ to interpolate twice, meaning the interpolated variable lives in the scope of Op2, like OP, A and B, or just interpolate once with $ to interpolate variables from the generated function’s scope, like T:

julia> macro Op2(OP,A,B)
           return esc(quote
               @generated function $OP(a::MyType{Ta},b::MyType{Tb}) where{Ta,Tb}
                   T = max(Ta,Tb)
                   return :(MyType{$T}($$OP(a.x*$$A,b.x*$$B)))
               end
           end)
       end
@Op2 (macro with 1 method)

julia> struct MyType{T} x end

julia> @Op2 Base.:+ 1 2

julia> MyType{1}(1) + MyType{2}(2)
MyType{2}(5)
2 Likes
11

Oh but this code is much easier for me to read! And I think this is exactly what McAbbot was talking about.

Simeon, McAbbot and Cedric, thank you for your patient help. This makes a great difference.

:grinning:

Philippe

2 Likes
12

Still more on the same subject of generated inside macro.

I want my macro to evaluate both a @generated function “jenny” (requiring double interpolation) and plain old function “foo”

macro mac(A)
    ex = esc(quote
        @generated function jenny(a,b) 
            ex = :(a+b*$$A)
            display(ex)
            return ex
        end
        foo(a,b) = a+b*$A
    end)
    display(ex)
    return ex
end

@mac(:a)
jenny(1.,2.)
@mac(a)
foo(1.,2.)

You see that to get “jenny” to work I must call the macro with :a, while to get “foo” to work, I must call it with a.

I need to be able to have the same macro do both, with either a or :a. But for the sake of learning - how would I do a macro taking a and generate workable jenny and foo, and a macro that takes :a and does the same?

13

The difference is this following:


julia> macro mac(A)
           @show A
           @show typeof(A)
           nothing
       end
@mac (macro with 1 method)

julia> @mac(a)
A = :a
typeof(A) = Symbol

julia> @mac(:a)
A = :(:a)
typeof(A) = QuoteNode

So if the input is a QuoteNode, the macro call was with a Symbol as the argument (a “double-symbol” sort of thing), and if it’s a symbol then it was a normal expression like @mac(a).
If you meant that you want the macro to be able to accept either one, then you just need to convert a QuoteNode to the appropriate Symbol to proceed as normal (or vice versa, doesn’t matter)

1 Like
14

Well, I think in the end I must bow to Cedric cstjean’s hard earned wisdom.

As I see it the main issue is that with a @generated -in-a-macro, where some of the interpolation is intended to occur at macro time, some at generation time, the code I was trying to write is not explicit enough on this decision: even if we got the code to work (and I still struggle with this :a vs. :(:a) business in spite of Tom’s good remark), it’s not readable. At least I get a headache reading my own code. So I got it all to work with the readable, is somewhat verbose, code:

Base.replace(ex::Symbol,a::Symbol,b) = ex==a ? deepcopy(b) : ex
Base.replace(ex::Expr  ,a::Symbol,b) = Expr(ex.head,replace.(ex.args,a,Ref(b))...)
Base.replace(ex        ,a::Symbol,b) = ex
macro mac(A)
    ex = quote
        @generated jenny(a,b) = :(a+b*øa)
        foo(a,b) = a+b*øb
    end
    ex = replace(ex,:øa,A)
    ex = replace(ex,:øb,A)
    return esc(ex)
end

@mac(a)
jenny(2.,3.)
foo(1.,2.)

Here I explicitely prescribe the interpolations that are to occur at macro time - and I could have done the same within the @generate function. Flexible, explicit, and so far, works.

A bit of syntax sweetening on the replace function and the job is done.

Neat thing with a Scandinavian keyboard: øReplaceMe :wink:

1 Like