Questions about macro interpolation inside @generated

[matthias314]

I would like to have a macro @m f g that produces

@generated function f(x)
    quote g(x) end
end

where f and g are the macro parameters. This has essentially been discussed in

@generated inside macro: who's interpolating?

I have a macro to generate code for “many” binary operations. The macro generates several functions, including a @generated function so that I can generate code for any combination of subtypes. A crash reproduction of my code looks like this macro Op2(OP,A,B) return quote @generated function $OP(a::MyType{Ta},b::MyType{Tb}) where{Ta,Tb} return quote MyType{Ta}($OP(a.x*$A,b.x*$B)) end end end end struct MyType{T} x::Fl…

@simeonschaub’s solution, adapted to my example, was

macro m(f, g)
    esc(quote
        @generated function $f(x)
            quote $$g(x)end
        end
    end)
end

I have two questions about this:

Firstly, I can’t get this approach working with “callable” types. For example, the following doesn’t work:

struct A{G} g::G end
@m a::A a.g

julia> a = A(identity);
julia> a(1)
ERROR: type DataType has no field g
Stacktrace:
 [1] getproperty(x::Type, f::Symbol)
   @ Base ./Base.jl:32
 [2] #s2#5
   @ Main ./REPL[1]:4 [inlined]
 [3] var"#s2#5"(a::Any, x::Any)
   @ Main ./none:0
 [4] (::Core.GeneratedFunctionStub)(::UInt64, ::LineNumberNode, ::Any, ::Vararg{Any})
   @ Core ./boot.jl:600
 [5] top-level scope
   @ REPL[5]:1

whereas

@generated function (a::A)(x)
    quote a.g(x) end
end

works without problems.

Secondly, the esc in the @m macro means that code is resolved in the macro call environment. If I want to define such a macro in a module and use many module functions inside the generated code, this becomes a bit awkward. How would one write the macro such that code is resolved in the module environment?

[fatteneder]

Nested quotes can be tricky. I tend to avoid working with $$ because it produces weird Core._expr things and such that (when looked at with @macroexpand1) which I don’t understand. Instead, I try to construct function bodies separately from function definitions inside macros.

I would approach your example as follows: Consider a rewrite like

julia> macro m(f,g)
  body = quote
    $g(x)
  end
  esc(quote
        @generated function $f(x)
          $body
        end
      end)
end

julia> struct A{G} g::G end

julia> @macroexpand1 @m a::G a.g # I manually stripped the LineNumberNodes from the console output
quote
    @generated function (a::A)(x)
            begin
                a.g(x)
            end
        end
end

So in this case the macro generates a @generated function definition that does not return an expression, but instead tries to access a field member of a, which is disallowed by @generated.

[ Side question for others: Is the rewrite equivalent to what OP gave? At least running a(1) gives the same ERROR: type DataType has no field g, but the outputs of @macroexpand1 @m a::G a.g differ. ]

To get the extra quote around a.g(x), you need to wrap the body into a QuoteNode so that the expression produced by the macro contains a quote that does not get expanded before @generated is expanded, e.g.

julia> macro m(f,g)
  body = quote
    $g(x)
  end
  esc(quote
        @generated function $f(x)
          $(QuoteNode(body))
        end
      end)
end

julia> @macroexpand1 @m a::A a.g
quote
    @generated function (a::A)(x)
       $(QuoteNode(
           quote
              a.g(x)
           end)
        )
     end
end

@matthias314

Secondly, the esc in the @m macro means that code is resolved in the macro call environment. If I want to define such a macro in a module and use many module functions inside the generated code, this becomes a bit awkward. How would one write the macro such that code is resolved in the module environment?

Not sure I understand. The macro output should be an expression that, when copied into the REPL (or the respective) module, just works as you would have typed it.
Why is that awkward? Can you maybe give an example?

[matthias314]

Here is an example for my second question. @m is defined in a module and uses some other function from the same module. Then I have to write something like the following:

module MyModule

export @m

h(x) = 2*x

macro m(f, g)
    esc(quote
        @generated function $f(x)
            quote $$g(x) + MyModule.h(x) end
        end
    end)
end

end

If there is more than one function from MyModule that I want to use, then writing the code gets awkward. It would be more convenient if I didn’t have to write every function h I use as MyModule.h. Can this be done?

[fatteneder]

The least intrusive ideas that come to my mind would be to

1. just use the functions that the module also exports,
2. or declare an alias inside the macro and use that to avoid ‘boilerplate’ in the code generation (not tested), e.g.

macro m(f, g)
    h = :(MyModule.h)
    esc(quote
        @generated function $f(x)
            quote $$g(x) + $$h(x) end
        end
    end)
end

But I advise against any more advanced trickery, e.g. like messing with using or import inside the macro’s returned expression, because a macro is already very opaque by default and so makes it even more vulnerable to annoying bugs.

[matthias314]

I would also like to avoid trickery. However, exporting all functions that one needs inside the @generated function doesn’t seem like an attractive solution.

Also, I have noticed that both my proposed macro involving MyModule.h and yours don’t work in all cases: Suppose I save the code to some MyModule.h. Then

using MyModule
g(x) = -x
@m f g

works, but

import MyModule as M
g(x) = -x
M.@m f g

gives

julia> f(1)
ERROR: UndefVarError: `MyModule` not defined

It would be great to have a clean solution that works in all cases (without exporting functions one doesn’t want to export).

[matthias314]

Here is a simpler case that I don’t understand, either. Why does the following not work?

module MyModule2

export @m

h(x) = x

macro m(f)
    quote
        @generated $(esc(f))(x) = quote h(x) end  
    end
end

end

It gives:

julia> using MyModule2
julia> @m f
julia> f(1)
ERROR: UndefVarError: `h` not defined

[fatteneder]

@matthias314

I would also like to avoid trickery. However, exporting all functions that one needs inside the @generated function doesn’t seem like an attractive solution.

Are you really sure an exotic combination of a macro constructing @generated functions that export methods from a module is the right solution to whatever problem you have?
@generated functions are usually only needed in very special cases, because multiple dispatch and aggressive optimization by the compiler is often enough.
Perhaps if you share the problem you want to have solved I (or someone else) could provide a simpler and more convenient solution.

Anyways, regarding your questions:

@matthias314

import MyModule as M
g(x) = -x
M.@m f g

This does not work, presumably because the macro generates code with MyModule.<method>, but you imported the module under the name M. But the macro cannot know that how the user imports a package, unless you pass that info also to the macro.

@matthias314

module MyModule2

export @m

h(x) = x

macro m(f)
    quote
        @generated $(esc(f))(x) = quote h(x) end  
    end
end

end

Same problem as your first post: You need a QuoteNode around the quote inside the @generated function. This here works for me

# mwe.jl
module MyModule2

export @m

h(x) = x

macro m(f)
  body = quote
    MyModule2.h(x)
  end
  esc(quote
    @generated function f(x)
      $(QuoteNode(body))
    end
  end)
end

end

julia> include("mwe.jl")

julia> using .MyModule2

julia> @m f

julia> f(1)
1

I advise you to take a look at @macroexpand1 and @macroexpand to debug this issues yourself.

[jules]

Not sure if I’ve followed the thread correctly, maybe this was already covered. Is this a solution to the problem?

julia> macro m(f, g)
           quote
               @generated function $(esc(f))($(esc(:x)))
                   quote
                       $$g(x)
                   end
               end
           end
       end

       struct A{G}
           g::G
       end

       @m a::A a.g

       a = A(identity)
       a(1)
1

[matthias314]

This is nice, but I don’t get it working inside a module:

module MyModule

export @m

macro m(f, g)
    quote
        @generated function $(esc(f))($(esc(:x)))
            quote $$g(x) end
        end
    end
end

end

gives (after loading the module etc.)

julia> a(1)
ERROR: UndefVarError: `a` not defined

Regarding my previous post: When I asked why

h(x) = x

macro m(f)
    quote
        @generated $(esc(f))(x) = quote h(x) end  
    end
end

doesn’t work inside a module, I wanted to understand the scoping rules that are applied. The documentation says that variables are resolved within the macro definition environment. But this doesn’t seem to be the case here because the function h defined in the same module is ignored. Why is this?

(Maybe that would be a topic for a second posting. Also, we do have a method that partially works around this. However, for me having an approach B that works doesn’t explain why approach A doesn’t work. I’m seeing this discussion as an opportunity to better understand Julia.)

[jules]

I also don’t 100% get it yet, but I trial-and-errored my way to this:

module MyModule

macro m2(f, g)
    q = Expr(:quote, :(
        $g(x)
    ))
    quote
        @generated function $(esc(f))($(esc(:x)))
            $q
        end
    end
end

end

struct A{G}
    g::G
end

MyModule.@m2 a::A a.g

a = A(x -> x + 2)
a(1)

It seems the double quote does something unexpected with scoping, which I avoid here by using a Expr(:quote which seems to play by different rules. I got the idea by looking what Meta.@dump quote something end resolved to.

[fatteneder]

@matthias314

The documentation says that variables are resolved within the macro definition environment. But this doesn’t seem to be the case here because the function h defined in the same module is ignored. Why is this?

I guess you are referring to the second paragraph after the code block here: Metaprogramming · The Julia Language
The paragraph that follows this statement gives a hint on the solution you are looking for.
That is, one needs to wrap an escape around the function to call.
But because in your example there are two quotes involved, one needs to use a $$(esc(h)) to make it work, e.g. this works now without the nasty MyModule2. prefix:

# mwe.jl
module MyModule2

export @m

h(x) = x

macro m(f)
    quote
      @generated $(esc(f))($(esc(:x))) = quote $$(esc(h))(x) end
    end
end

@m a

end


using .MyModule2

@m b

julia> include("mwe.jl")

julia> MyModule2.a(1)
1

julia> b(1)
1

I gotta admit I did not know that this was possible and I am happy I learned something from this

[matthias314]

Very interesting, thanks! This seem to work except for the following special case: The macro argument g is not allowed to be an x,

julia> x(t) = t
julia> MyModule.@m2 f x
julia> f(1)
ERROR: MethodError: objects of type Int64 are not callable

EDIT: The following seems to work:

module MyModule3

macro m3(f, g)
    x = gensym()
    q = Expr(:quote, :($g($x)))
    quote
        @generated $(esc(f))($(esc(x))) = $q
    end
end

end

[matthias314]

Great, thanks!! This seems to work for all ways of loading the module (and also if the macro argument f is an x). So I would say that the case of @m f (without second argument g) is solved.

But do you understand why the $$(esc(h)) does not lead to the h being resolved in the macro call environment? It’s obviously resolved in the environment where the macro is defined. If you have a single quote end, then $(esc(h)) would bring you to the macro call environment. Here we have two nested quote end.