Is there a better way to write this macro?

General Usage
, ,
1

I’m working on a VM, of the classic dispatch-on-value sort, and the last step (which I’m bearing down on) will be to make it type stable. The current implementation uses a multimethod for dispatch, but the Vector holding instructions is boxed and the type is resolved at runtime. I believe I have a handle on how to get all of that working, this is just by way of introduction.

The current runvm! function is very small, and I have two of them currently, and plan to add a third. There’s no getting around unrolling the dispatch, which I would then have to repeat and keep in sync between all VM instances.

This strikes me as a good candidate for a macro. I can assure that each opcode is compared against in ascending order, to help LLVM emit it as a computed goto, it will always include every opcode, if any of the functions are missing I’ll get an immediate error, and I can just call @dispatchinstruction, meaning that the function body stays the same size as it is now.

I’ve implemented this macro successfully, but wow, does it not follow recommended style. It has both Meta.parse and eval in it, I ended up resorting to building up strings to encode an Expr. Reason being that I couldn’t find a way to express part of an if statement as an expression, which makes sense, because it’s not, it’s just a fragment of one.

Stripped of irrelevancies it looks like this:

macro dispatchinstruction()
    fragments = [":("]
    first = true
    for opcode in instances(Opcode)
        opname = "MyModule." * string(opcode)
        if first
            first = false
            push!(fragments, """if \$(esc(:inst)).op  == \$(esc($opname))\n""")
        else
            push!(fragments, """elseif \$(esc(:inst)).op  == \$(esc($opname))\n""")
        end
        fname = "on" * string(opcode)
        push!(fragments, """    $fname(\$(esc(:inst)),\$(esc(:vm))) """)
    end
    push!(fragments, "else error(\"illegal value\")\nend)\n")
    eval(Meta.parse(join(fragments)))
end

Which does the job, no complaints. But it leaves me wondering if I missed a trick, given the emphasis placed on not using eval in macros, and building expressions rather than doing string-based codegen, which I’ve seen in both the Julia manual and here on the discourse.

I’m not displeased with the result and am content to ship it as-is, but does anyone see a way to do this without building up a program string and evaluating it? In Common Lisp it would be easy enough to write this macro without strings, and I have a nagging feeling that this is possible in Julia as well.

2

Can’t you first define the condition and branch expressions, and then put them into an if afterwards?

3

@dispatchinstruction takes no expression as inputs and does not return an expression to be evaled. It may do what you want in a standalone call, but it is not designed how macros are intended to be and is not capable of transforming source code like macros do. It might as well be a function.

Typically the strings being Meta.parsed don’t have :() in them, and parsing it introduces an unwanted layer of quoting.

This is just bizarre to me, it looks like you’re trying to produce esced expression interpolation in the pre-parse String, and it does not work that way.

You don’t need to metaprogram by building strings, which makes parse errors easier to encounter. You don’t need to metaprogram with macros, the docs shows examples where you just eval.

1 Like
4

The macro compiles the expected expression correctly.

5

I can only take your word for that because I don’t have the full context in which you call the macro. The reason why the \$ in a string doesn’t throw an error is the atypical second layer of quoting.

julia> text = ":(\$(esc(:inst)))"
":(\$(esc(:inst)))"

julia> expression = Meta.parse(text)
:($(Expr(:quote, :($(Expr(:$, :(esc(:inst))))))))

julia> evaluate1 = eval(expression)
:($(Expr(:escape, :inst)))

julia> evaluate2 = eval(evaluate1) # this somehow works in your case
ERROR: syntax: invalid syntax (escape (outerref inst))

That second eval is being done automatically after your macro body returns. It should be apparent that evaling an expression to another expression is not standard practice, and none of the macros in the docs needs to do this. You layered several atypical practices until you got a functioning product, and that will be brittle.

2 Likes
6

Macros return expressions, that’s their whole jam.

I would also describe building up a string of source code containing a quasiquoted expression, and using parse/eval to return the expression, as ‘atypical’. Hence the topic of this thread.

7

I am obviously aware that macros are intended to return expressions, I was clearly referring to the manual eval call to generate that expression. If you find a documented example, feel free to link it.

Something else I should warn you about: the macro uses Opcode as a global variable, but the string suggests the enum and its global names actually belong to a module MyModule. You’re relying on 2 modules and an import between them for the macro to work, that’s another brittle condition. Refactoring to a typical metaprogramming practice will likely look like code that runs once and only in one of the modules. Use a local scope (let, for, while) to make the temporary variables go away, eval finds its way to the global scope anyway.

8

The documentation does not provide complete examples of everything one might do with macros. The eval is there because I needed the string, which I needed because :(if somestuff) is a parse error.

What I’m doing with it is simply:

julia> :(1 + 2)
:(1 + 2)

julia> eval(Meta.parse(":(1 + 2)"))
:(1 + 2)

And the reason is because strings don’t have syntax errors, which gave me the flexibility to build the full expression.

The code is a stripped down version of the actual code, meant to illustrate the domain and the solution.

Atypical code cannot be rendered typical in full generality.

9

Yes, there is definitely a better way. Instead of trying to jam it all into quote syntax, just build up the Exprs manually. See Meta.show_sexpr for inspiration on how to convert syntax to the Expr equivalent.

7 Likes
10

You can build expressions with if-statements, the docs expands @assert as an example. You’re right that you don’t build it word by word because that’s not how syntactic macros manipulate ASTs. As dense as it is, the Metaprogramming page clearly explains the structure of Expr and how to mutate them. Julia isn’t as convenient as a Lisp where source code is exactly the expression structure, you’ll have to use dump to see the actual tree structure (and remember to change the default recursive limit sometimes), and use quotes to write expressions in the source code format.

The only syntax error you’re working around is truncated expressions. You can piece strings together instead of learning how to interpolate or otherwise mutate expressions, sure, but you’re additionally risking a whole host of other syntax errors. The parser isn’t as liberal as we’d think, that’s why all documented metaprogramming examples start at expressions instead of strings.

Cool, but writing atypical code without exhausting typical practices is a misguided approach. Your first post shows you suspect this, and I’m confirming all of your intuition was good.

11

Manipulating if-elseif is a little bit hard, but you can do that with Expr though.

macro dispatcher()
    conditions = [:(inst.op == $op) for op in instances(Opcode)]
    actions = [:($(Symbol("on"*string(op)))(inst, vm)) for op in instances(Opcode)]
    body = Expr(:if, conditions[1], actions[1])
    last = body
    for i in 2:lastindex(conditions)
        e = Expr(:elseif, conditions[i], actions[i])
        push!(last.args, e)
        last = e
    end
    push!(last.args, :(error("illegal value")))
    esc(body)
end

You can test it with:

@enum Opcode add sub mul
struct Inst
    op::Opcode
end
function foo(inst, vm)
    @dispatcher
end
onadd(_, _) = nothing

@code_lowered foo(Inst(add), 1)
#=
CodeInfo(
1 ─ %1  = Base.getproperty(inst, :op)
│   %2  = %1 == add
└──       goto #3 if not %2
2 ─ %4  = Main.onadd(inst, vm)
└──       return %4
3 ─ %6  = Base.getproperty(inst, :op)
│   %7  = %6 == sub
└──       goto #5 if not %7
4 ─ %9  = Main.onsub(inst, vm)
└──       return %9
5 ─ %11 = Base.getproperty(inst, :op)
│   %12 = %11 == mul
└──       goto #7 if not %12
6 ─ %14 = Main.onmul(inst, vm)
└──       return %14
7 ─ %16 = Main.error("illegal value")
└──       return %16
)
=#
2 Likes
12

The whole string/eval thing is definitely not the recommended way to do it. There’s lots of ways to write this, but my preferred way would be the following:

macro dispatchinstruction(inst, vm)
    let vm = esc(vm), inst = esc(inst)
        foldr(instances(Opcode); init=:(error("Illegal value"))) do opcode, else_branch
            cond = :($inst.op == $opcode)
            fname = Symbol(:on, opcode)
            body = :($fname($inst, $vm))
            Expr(:if, cond, body, else_branch)
        end
    end
end

The important changes I made were

  • inst and vm are arguments to the macro, rather than implicitly assuming that the caller has variables defined in their local scope named inst and vm.
  • No use of string metaprogramming, we instead build up a series of if statements by folding over them
  • No use of eval.
  • Don’t esc the function fname since that should be defined in the same namespace that this macro and Opcode were defined in, not necessarily in the module where the macro is called.

Here’s the generated code:

julia> @enum Opcode a b c d

julia> @macroexpand1 @dispatchinstruction(inst, vm)
:(if inst.op == a
      Main.ona(inst, vm)
  else
      if inst.op == b
          Main.onb(inst, vm)
      else
          if inst.op == c
              Main.onc(inst, vm)
          else
              if inst.op == d
                  Main.ond(inst, vm)
              else
                  Main.error("Illegal value")
              end
          end
      end
  end)

Note that the Mains you see in the above code will become whatever module the macro was defined in, not whatever module the macro was called in, since julia macros are hygenic by default.

7 Likes
13

I figured you’d have said if there was, but is there a dual to show_sexpr which lets the user cons up expressions in that format?

Building it out of raw Exprs is, hmm, ideologically correct, and probably not that bad since I can use quasiquotes to build the parts which are themselves valid expressions, but it would be handy if that shorthand were exposed directly.

14

Isn’t that just Expr?

julia> Meta.show_sexpr(:(1 + 1 / 2))
(:call, :+, 1, (:call, :/, 1, 2))

julia> Expr(:call, :+, 1, Expr(:call, :/, 1, 2))
:(1 + 1 / 2)
5 Likes
15

Or quote

julia> ex = quote
           x + y
       end
quote
    #= REPL[2]:2 =#
    x + y
end

julia> dump(ex)
Expr
  head: Symbol block
  args: Array{Any}((2,))
    1: LineNumberNode
      line: Int64 2
      file: Symbol REPL[2]
    2: Expr
      head: Symbol call
      args: Array{Any}((3,))
        1: Symbol +
        2: Symbol x
        3: Symbol y
1 Like
16

That’s a nice bit of code, thanks for pointing out that the elseif has to cons to the latest branch, the parser doesn’t flatten them down to one level. That would explain the problem which lead me to give up and build it out of strings. There’s some escaping I’d need to add but that’s not the hard part.

This is downright eloquent.

Thanks for your help @melonedo, @Mason, @mbauman!

Edit:

Yeah I guess it is, huh. Dumping an Expr includes a lot of derived info and I hadn’t thought about the fact that show_sexpr is more-or-less what you get from calling the constructor, just with the call itself elided.

That’ll help, a tiny bit of visual noise over “pure” s-expr format isn’t an actual problem here.

17

If you don’t want to type Expr, you can also just write a quick macro for that:

julia> macro s(args...)
           :(Expr($(esc.(args)...),))
       end;

julia> (@s :call :+ 1 (@s :call :/ 1 2))
:(1 + 1 / 2)
3 Likes
18

Beautiful. Now I just need to add some functions for cons/car/cdr and I’m back in the zone!

2 Likes
19

Decided to play around with sexpr-izing tuples and came up with this quick sketch/MVP.

I’m sure it would need a bit of refinement to handle everything correctly but it has the right spirit.

import MacroTools: postwalk 

function sexpr_walk(expr)
    if expr isa Expr
        if expr.head == :call && expr.args[1] == :~
            e = Expr(expr.args[2], expr.args[3:end]...)
            return QuoteNode(e)
        elseif expr.head == :tuple
            if expr.args[1] isa QuoteNode
                car = expr.args[1].value
            else
                car = expr.args[1]
            end
            return Expr(car, expr.args[2:end]...)
        else
            return expr
        end
    end
    return expr
end

macro sexpr(expr)
    transform = postwalk(sexpr_walk, expr)
    return QuoteNode(transform)
end

Which lets me write this:

julia> @sexpr (:if, (call, <, a, b), (call, f, a, b), (call, g, a, b))
:(if a < b
      f(a, b)
  else
      g(a, b)
  end)

If the head of the Expr isn’t a keyword you don’t have to use :head, which is handy.

Pity about all the commas but truly can’t complain there, only way around it is a string macro with a built-in parser, which seems like a lot of work just to fight the language.

The ~ for QuoteNode is in there because canonical Lisps have quote as well as quasiquote. Don’t know where that would be useful but it felt like an affordance to supply.

For example, if it weren’t for the circularity issue, I could write the return line of the macro @sexpr as @sexpr ~($transform). Maybe. This is an under-tested sketch, remember.

20

These were pretty easy as it turns out.

It bugs me that the semantics aren’t identical, but I don’t see how they can be.

function car(expr::Expr)
    expr.head
end

function cdr(expr::Expr)
    expr.args
end

function cons(e1::Expr, e2)
    push!(e1.args, e2)
    e1
end

julia> ifdef = @sexpr (:if, (call, <, a, b), (call, f, a, b))
:(if a < b
      f(a, b)
  end)

julia> elseifdef = @sexpr (:elseif, (call, isa, a, Symbol), (call, g, a, b, :c))
:(elseif a isa Symbol
      g(a, b, :c)
  end)

julia> cons(ifdef, elseifdef)
:(if a < b
      f(a, b)
  elseif a isa Symbol
      g(a, b, :c)
  end)