Skipping LineNumberNodes in v0.7 macros

General Usage
1 
ex = :(begin
    p1, (X,Y,Z) --> 0
    hill(X,p2,100.,-4), 0 --> Y
    hill(Y,p3,100.,-4), 0 --> Z
    hill(Z,p4,100.,-4), 0 --> X
    hill(X,p5,100.,6), 0 --> R
    hill(Y,p6,100.,4)*0.002, R --> 0
    p7, 0 --> S
    R*p8, S --> SP
    p9, SP + SP --> SP2
    p10, SP2 --> 0
end)

julia> ex.args
20-element Array{Any,1}:
 :(#= REPL[5]:2 =#)
 :((p1, $(Expr(:-->, :((X, Y, Z)), 0))))
 :(#= REPL[5]:3 =#)
 :((hill(X, p2, 100.0, -4), $(Expr(:-->, 0, :Y))))
 :(#= REPL[5]:4 =#)
 :((hill(Y, p3, 100.0, -4), $(Expr(:-->, 0, :Z))))
 :(#= REPL[5]:5 =#)
 :((hill(Z, p4, 100.0, -4), $(Expr(:-->, 0, :X))))
 :(#= REPL[5]:6 =#)
 ⋮
 :((hill(Y, p6, 100.0, 4) * 0.002, $(Expr(:-->, :R, 0))))
 :(#= REPL[5]:8 =#)
 :((p7, $(Expr(:-->, 0, :S))))
 :(#= REPL[5]:9 =#)
 :((R * p8, $(Expr(:-->, :S, :SP))))
 :(#= REPL[5]:10 =#)
 :((p9, $(Expr(:-->, :(SP + SP), :SP2))))
 :(#= REPL[5]:11 =#)
 :((p10, $(Expr(:-->, :SP2, 0))))

julia> ex.args[2]
:((p1, $(Expr(:-->, :((X, Y, Z)), 0))))

julia> ex.args[1]
:(#= REPL[5]:2 =#)

julia> typeof(ex.args[1])
LineNumberNode

In v0.6 I used to for a in ex.args but now that gives these extra LineNumberNodes in there. Is there a canonical way to remove them or skip them when iterating? Are they always the odd values?

1 Like
2

You can use MacroTools.prettify:

julia> using MacroTools: prettify

julia> ex = quote
           p1, (X,Y,Z) --> 0
           hill(X,p2,100.,-4), 0 --> Y
           hill(Y,p3,100.,-4), 0 --> Z
           hill(Z,p4,100.,-4), 0 --> X
           hill(X,p5,100.,6), 0 --> R
           hill(Y,p6,100.,4)*0.002, R --> 0
           p7, 0 --> S
           R*p8, S --> SP
           p9, SP + SP --> SP2
           p10, SP2 --> 0
       end |> prettify
quote
    (p1, $(Expr(:-->, :((X, Y, Z)), 0)))
    (hill(X, p2, 100.0, -4), $(Expr(:-->, 0, :Y)))
    (hill(Y, p3, 100.0, -4), $(Expr(:-->, 0, :Z)))
    (hill(Z, p4, 100.0, -4), $(Expr(:-->, 0, :X)))
    (hill(X, p5, 100.0, 6), $(Expr(:-->, 0, :R)))
    (hill(Y, p6, 100.0, 4) * 0.002, $(Expr(:-->, :R, 0)))
    (p7, $(Expr(:-->, 0, :S)))
    (R * p8, $(Expr(:-->, :S, :SP)))
    (p9, $(Expr(:-->, :(SP + SP), :SP2)))
    (p10, $(Expr(:-->, :SP2, 0)))
end

julia>

You also don’t need :(begin; ...; end), for multi-line expressions you could simply use quote; ...; end.

3

You can also use Reduce.linefilter, which removes all comments like that from a piece of code.

julia> Reduce.linefilter(ex)
quote 
    (p1, $(Expr(:-->, :((X, Y, Z)), 0)))
    (hill(X, p2, 100.0, -4), $(Expr(:-->, 0, :Y)))
    (hill(Y, p3, 100.0, -4), $(Expr(:-->, 0, :Z)))
    (hill(Z, p4, 100.0, -4), $(Expr(:-->, 0, :X)))
    (hill(X, p5, 100.0, 6), $(Expr(:-->, 0, :R)))
    (hill(Y, p6, 100.0, 4) * 0.002, $(Expr(:-->, :R, 0)))
    (p7, $(Expr(:-->, 0, :S)))
    (R * p8, $(Expr(:-->, :S, :SP)))
    (p9, $(Expr(:-->, :(SP + SP), :SP2)))
    (p10, $(Expr(:-->, :SP2, 0)))
end
4

But presumably one should keep them for better back-traces, right? So maybe

out = []
for a in ex.args
  if a isa LineNumberNode
    push!(out, a)
  else
    # do something
    push!(out, somthing)
  end
end
...
5

A for loop won’t suffice, you need to go recursively into the ASTree to access all possible comments. This is why Reduce.linefilter is recursively defined with while loops

https://github.com/chakravala/Reduce.jl/blob/master/src/parser.jl#L423-L443

This is backwards compatible with 0.6 and 0.7 simultaneously.

6

Note that prettify does multiple things and is mainly aimed at showing code in the REPL. For macros you probably want striplines.

prewalk makes the recursion a lot nicer here :slight_smile:

3 Likes
7

MacroTools.prettify does more than just removing line nodes, so just take that into account:

https://github.com/MikeInnes/MacroTools.jl/blob/master/src/utils.jl#L387-L393

There is also the @q macro:

https://github.com/MikeInnes/MacroTools.jl/blob/master/src/utils.jl#L18-L24

@mauro3 macro tools also have prewalk and postwalk that are used to traverse expressions recursively instead of iterating on them.

8

However, it turns out that my implementation in Reduce is significantly faster than striplines from MacroTools, as the benchmark shows

julia> @btime Reduce.linefilter(ex)
  21.442 μs (0 allocations: 0 bytes)
quote 
    (p1, $(Expr(:-->, :((X, Y, Z)), 0)))
    (hill(X, p2, 100.0, -4), $(Expr(:-->, 0, :Y)))
    (hill(Y, p3, 100.0, -4), $(Expr(:-->, 0, :Z)))
    (hill(Z, p4, 100.0, -4), $(Expr(:-->, 0, :X)))
    (hill(X, p5, 100.0, 6), $(Expr(:-->, 0, :R)))
    (hill(Y, p6, 100.0, 4) * 0.002, $(Expr(:-->, :R, 0)))
    (p7, $(Expr(:-->, 0, :S)))
    (R * p8, $(Expr(:-->, :S, :SP)))
    (p9, $(Expr(:-->, :(SP + SP), :SP2)))
    (p10, $(Expr(:-->, :SP2, 0)))
end

julia> @btime MacroTools.striplines(ex)
  427.192 μs (443 allocations: 20.70 KiB)
quote 
    (p1, $(Expr(:-->, :((X, Y, Z)), 0)))
    (hill(X, p2, 100.0, -4), $(Expr(:-->, 0, :Y)))
    (hill(Y, p3, 100.0, -4), $(Expr(:-->, 0, :Z)))
    (hill(Z, p4, 100.0, -4), $(Expr(:-->, 0, :X)))
    (hill(X, p5, 100.0, 6), $(Expr(:-->, 0, :R)))
    (hill(Y, p6, 100.0, 4) * 0.002, $(Expr(:-->, :R, 0)))
    (p7, $(Expr(:-->, 0, :S)))
    (R * p8, $(Expr(:-->, :S, :SP)))
    (p9, $(Expr(:-->, :(SP + SP), :SP2)))
    (p10, $(Expr(:-->, :SP2, 0)))
end
9

prettify broke something on v0.6, Reduce is far too large of a dependency to even consider here, and striplines worked well with no hassle, so striplines it is.

https://github.com/JuliaDiffEq/DiffEqBiological.jl/commit/792e7b98cf0dc9389765c597c040b77fcdb6d658

10

created a new package called SyntaxTree that now contains linefilter method

11

I would generally discourage using mutation with Exprs, given the safety you get (and performance usually matters less when from macro code that runs once).

But you could no doubt implement an in-place prewalk! for MacroTools and get the best of both versions :slight_smile:

3 Likes
12

In my case, I needed something with high performance, since I need to apply this method many times, it would take up a significant chunk of processing time if it was too slow, so I opted for the fastest way possible by making a single purposed method for this task. I just simply don’t need a generalized walk method for this, I really needed a special purpose method that does it as fast as possible, that’s why I made it.

So in my special situations, the performance does make a big difference for something like this.

I think you could also write a code generator that writes special purpose code for parsing through syntax trees based on what operation it is, such as the linefilter method, then you can also gain performance.