Problem about `quote` in macro definition

KDr2 · April 18, 2020, 2:00pm

I wrote the code below:


module T

function f1 end

macro m1(sym)
    :(f1(::typeof($(esc(sym)))) = 1)
end

macro m2(sym)
    quote
        f1(::typeof($(esc(sym)))) = 1
    end
end

macro m3(sym)
    quote
        $(esc(f1))(::typeof($(esc(sym)))) = 1
    end
end

end

function test end

@show @macroexpand T.@m1(test)
@show "-" ^ 12
@show @macroexpand T.@m2(test)
@show "-" ^ 12
@show @macroexpand T.@m3(test)

It turns out that:

m1 expands as Main.T.f1(::Main.T.typeof(test)) = ...
m2 expands as var"#5#f1"(::Main.T.typeof(test)) = ...
m3 expands as (Main.T.f1)(::Main.T.typeof(test)) = ...

My problems:

Why m2 doesn’t resolve f1 correctly as m1 does?
Why Main.T.f1 is put between parenthesizes in the m3 expansion? T think f(a...) = and (f)(a...) = do NOT have the same meaning, and, how to remove the parenthesizes?

Thanks.

ffevotte · April 18, 2020, 7:13pm

There are many things going on here. I’m not sure I can explain everything, but let’s at least try.

The relevant part of the documentation is the section about macro hygiene. We see there that a symbol appearing in a macro is renamed (gensymed, like var"#5#f1" in your example) if the corresponding variable is considered local.

In macro m1, the name f1 is considered to refer to a global variable. But in macro m2, it is detected as being local and is renamed. My guess is that this change in behavior comes from the implicit begin...end block generated by quote...end. But I’m actually not sure whether this is intended or not.

The reason why macro m3 does not work is a bit tricky to explain. It is not really because of the presence of parentheses: as you can easily verify, parentheses surrounding the function name are allowed.

julia> (f)(x) = 2x
f (generic function with 1 method)

julia> f(4)
8

Rather, the presence of parentheses is a subtle indicator that the generated code is not really what you think it is. Let’s inspect it further:

julia> using MacroTools
julia> dump(MacroTools.striplines(@macroexpand T.@m3 foo))
Expr
  head: Symbol block
  args: Array{Any}((1,))
    1: Expr
      head: Symbol =
      args: Array{Any}((2,))
        1: Expr
          head: Symbol call
          args: Array{Any}((2,))
            1: f1 (function of type typeof(Main.T.f1)) # The function itself; not its name
            2: Expr
              head: Symbol ::
              args: Array{Any}((1,))
                1: Expr
        2: Expr
          head: Symbol block
          args: Array{Any}((1,))
            1: Int64 1

(For comparison, the same output for `@m1`, which works)

julia> dump(MacroTools.striplines(@macroexpand T.@m1 foo))
Expr
  head: Symbol =
  args: Array{Any}((2,))
    1: Expr
      head: Symbol call
      args: Array{Any}((2,))
        1: GlobalRef            # (qualified) name Main.T.f1
          mod: Module Main.T
          name: Symbol f1
        2: Expr
          head: Symbol ::
          args: Array{Any}((1,))
            1: Expr
              head: Symbol call
              args: Array{Any}((2,))
                1: GlobalRef
                2: Symbol foo
    2: Expr
      head: Symbol block
      args: Array{Any}((1,))
        1: Int64 1

What we see is that, instead of the name of f1, we got the function itself. Let’s try and understand what happens with this whole $(esc(...)) syntax. In the case of $(esc(sym)), which you got right:

sym is evaluated in the context of the macro expansion, yielding the symbol :foo, because that’s what the macro argument was named
this symbol :foo is escaped, meaning that it is marked to not be changed for hygiene purposes → it therefore reaches the macro expansion without modification
when the macro expansion gets evaluated (in the context of the macro call), :foo therefore refers to the function named foo in the current scope

Let’s follow the same steps in the case of $(esc(f1)):

f1 is evaluated in the context of the macro expansion, yielding the function T.f1 (not its name: the function itself)
escaping the function doesn’t mean much; nothing is done about it regarding hygiene (EDIT: to be clear, and as noted by @mcabbott, esc has no effect whatsoever here, and can be removed without changing anything)
when the macro expansion gets evaluated, Julia tries to interpret function T.f1 as a function name, which does not make sense => error
```
julia> T.@m3 foo
ERROR: syntax: invalid function name "typeof(T.f1)()"
```

All in all, if you surround your code in a quote...end block (which I would advise to do), you’ll have to make it so that Julia understands f1 as being a global variable. There are (at least) two ways of doing this:

mark it with the global keyword (-> @m2 in the example below)
qualify the function name with the module name (-> @m3 in the example below)

module T

function f1 end

macro m1(sym)
    :(f1(::typeof($(esc(sym)))) = 1)
end

macro m2(sym)
    quote
        global f1
        f1(::typeof($(esc(sym)))) = 1
    end
end

macro m3(sym)
    quote
        T.f1(::typeof($(esc(sym)))) = 1
    end
end

end

We can check that all three macros work:

julia> using .T: @m1, @m2, @m3, f1

julia> foo1() = 1;
julia> @m1(foo1)
f1 (generic function with 1 method)

julia> foo2() = 1;
julia> @m2(foo2)
f1 (generic function with 2 methods)

julia> foo3() = 1;
julia> @m3(foo3)

julia> methods(T.f1)
# 3 methods for generic function "f1":
[1] f1(::typeof(foo3)) in Main at REPL[1]:18
[2] f1(::typeof(foo2)) in Main at REPL[1]:12
[3] f1(::typeof(foo1)) in Main at REPL[1]:6

mcabbott · April 18, 2020, 8:34pm

Thanks, that’s a nice writeup. I still find f1 a bit surprising, but indeed :( ) differs from quote by a begin block, which is what seems to matter. Two more cases:

macro m4(sym) # like original m3, not a function definition
    quote
        $f1(::typeof($(esc(sym)))) = 1
    end
end

macro m5(sym) # :(begin  end) is like quote, local f1
    :(begin
        f1(::typeof($(esc(sym)))) = 1
    end)
end

julia> @m4(foo4)
ERROR: syntax: invalid function name "typeof(T.f1)()" around REPL[1]:24
julia> @m5(foo5)
#22#f1 (generic function with 1 method)

julia> methods(T.f1) # nothing for foo5
# 3 methods for generic function "f1":
[1] f1(::typeof(foo3)) in Main at REPL[1]:18
[2] f1(::typeof(foo2)) in Main at REPL[1]:12
[3] f1(::typeof(foo1)) in Main at REPL[1]:6

ffevotte · April 18, 2020, 8:43pm

So do I.

I really wonder whether this is a bug, but in any case I wouldn’t rely on this corner case to avoid problems here…

Thanks: I think these help clarify the explanation. I edited my answer above to be more explicit about the behavior you exhibit in m4: that esc has absolutely no effect here, and can be removed without changing anything to the issue (which is rather related to $).

KDr2 · April 19, 2020, 1:31am

I tried to add an expression that reads f1 to make the macro recognize f1 as a non-local var, like this:

macro m4(sym)
    quote
        print(f1)
        f1(::typeof($(esc(sym)))) = 1
    end
end

But it doesn’t work, global f1 works,thanks.

And “qualify the function name with the module name” like this

macro m3(sym)
    quote
        T.f1(::typeof($(esc(sym)))) = 1
    end
end

resolves f1 as (Main.T.T).f1 which is a little weird but works.