Macro behaves differently than manually running the output of macroexpand

General Usage
1

I’m trying to write a macro and found a problem that I fail to debug, since macroexpand gives code that runs just fine.
MWE (on Julia 0.6.3):
after defining

function _b(iter_vars, ranges)
  iter_exprs = [:($(esc(v)) = $(esc(r))) for (v,r) in zip(iter_vars,ranges)]
  var_array_ex = Expr(:vect, (esc(v) for v in iter_vars)...)
  quote
      $(Expr(:comprehension, Expr(:generator,
                          quote
                            $var_array_ex .+ 1
                          end,
                          iter_exprs...
                       )))
  end
end

macro _b(iters...)
  iter_vars = [e.args[1] for e in iters]
  ranges = [e.args[2] for e in iters]
  _b(iter_vars, ranges)
end

running the macro _b gives a warning

julia> @_b( i=1:1, j=1:1 )
WARNING: .+ is no longer a function object; use broadcast(+, ...) instead
Stacktrace:
 [1] depwarn(::String, ::Symbol) at .\deprecated.jl:70
 [2] (::Base.##719#720)(::Array{Int64,1}, ::Int64) at .\deprecated.jl:355
 [3] macro expansion at .\REPL[9]:0 [inlined]
 [4] (::##7#8)(::Tuple{Int64,Int64}) at .\REPL[9]:7
 [5] collect(::Base.Generator{Base.Iterators.Prod2{UnitRange{Int64},UnitRange{Int64}},##7#8}) at .\array.jl:470
 [6] eval(::Module, ::Any) at .\boot.jl:235
 [7] eval_user_input(::Any, ::Base.REPL.REPLBackend) at .\REPL.jl:66
 [8] macro expansion at .\REPL.jl:97 [inlined]
 [9] (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at .\event.jl:73
while loading no file, in expression starting on line 5
1×1 Array{Array{Int64,1},2}:
 [2, 2]

However, running the result of macroexpand “manually” yields no warning:

julia> @macroexpand @_b( i=1:1, j=1:1 )
quote  # REPL[9], line 5:
    [begin  # REPL[9], line 7:
        [i, j] .+ 1
    end for i = 1:1, j = 1:1]
end

julia> [begin
         [i,j] .+ 1
         end for i=1:1, j=1:1]
1×1 Array{Array{Int64,1},2}:
 [2, 2]

I can work around this by using a loop, but would like to understand what is happening here.

2

This macro is entirely ran at compile-time and does not output an expression. I think you meant:

macro _b(iters...)
  iter_vars = [e.args[1] for e in iters]
  ranges = [e.args[2] for e in iters]
  quote
    _b(iter_vars, ranges)
  end
end
3

Well, you would at least need to interpolate the two variables…

I don’t think so. _b operate on expressions.

I think (not easy for me to check right now) the warning won’t be raised unless the expression is evaluated (lowered at least) so @macroexpand won’t have any issue.

4

Oh that’s my bad. I didn’t see that the inner function was returning a quoted expression. Ignore my first comment.

5

I’m not sure I understand what you mean. To clarify, the warning is not raised when copy-pasting and running the expression returned by @macroexpand. I expect this to be equivalent to running the macro, and also to eval(@macroexpand ...), but both of these do show the warning.

6

It’s fixed if you follow the issue and do:

  quote
      $(Expr(:comprehension, Expr(:generator,
                          quote
                            broadcast(+,$var_array_ex,1)
                          end,
                          iter_exprs...
                       )))
  end

The issue is likely due to the fact that on v0.6 broadcast is implemented in the parser, see More Dots: Syntactic Loop Fusion in Julia for details. At parse time broadcasted expressions are changed to broadcast(...) calls in order to fuse, however that must not be happening inside of the quoted expression.

However, on v0.7 the implementation of broadcast changed to be done lazily through the type system as discussed in this blog: Extensible broadcast fusion . When I check on v0.7 I also get the same warning, so there must be more to it.

7

I mean macroexpand the expression but not evaluate it. It only evaluate the macro body. The warning is raised in the step(s) in between macro expansion and evaluation which is why it’s not raised when you only do macro expansion. More about copy-paste below.

And no. It’s implemented in the frontend but not the parser (i.e. lowering not parsing).

I don’t believe the dot to “broadcast” conversion has moved in 0.7 so it’s not surprising that the warning is still there. The only difference is the form of the “boardcast”.

That actually shows you that the copy paste isn’t accurate, or more like the printing doesn’t give you full information:

julia> macro m(a, b)
           :($(esc(a)) .* $(esc(b)))
       end
@m (macro with 1 method)

julia> @macroexpand @m a b
:(a .* b)

julia> Meta.show_sexpr(@macroexpand @m a b)
(:call, :(Main..*), :a, :b)
julia> :(a .* b)
:(a .* b)

julia> Meta.show_sexpr(:(a .* b))
(:call, :.*, :a, :b)

This is a bug in macro hygiene. It should probably just keep .* intact since it’s not a name anymore. It’s very hard to do before the depwarn is removed though since that changes the semantics.

8

Wouldn’t that also be a bug in the printing of expressions, since (:call, :(Main..*), :a, :b) and (:call, :.*, :a, :b) are printed identically? Or is it unrealistic to expect the pretty-printing of expressions to be unambiguous?
In the latter case, perhaps the documentation should stress the use of Meta.show_sexpr more. It’s briefly mentioned in the Metaprogramming section, but not as a something that provides more detailed information, and not in the context of macroexpand.

9

It’s a trade off. It’ll be pretty bad if everything in the macro expansion looks like M.:+(a, b) instead of a + b…

Actually that’s not correct. The * in .* should still need proper scope so the macro expander isn’t doing anything wrong and the proper fix is impossible without the removal of the deprecation.