Converting symbols to types and splatting inside @eval

General Usage
,
1

I am trying to wrap my head around metaprogramming and macros and I struggle to figure this out and I think I am on the wrong path. I have the feeling that I need to build up an expression tree programmatically instead of the current approach, but let’s see.

To avoid an XY problem I’d like to provide a not so MWE (see the end) since there might be a better way to do what I am up to.

My specific problem is that I want to generate a function call with an arbitrary number of arguments. In my macro I have access to a list of types in form of a Vector{Symbol} like this:

types = Symbol[:Int32, :Float32]  # the length of this varies

which I’d like to turn into (dummy example):

foo() = Foo(Int32, Float32)

I managed to do it but the problem is that I am calling eval() which is then part of the function definition and I lose a lot of performance. It’s looking something like this inside my macro definition:

types = Symbol[:Int32, :Float32] 
@eval foo() = Foo([eval(t) for t in $(types)]...)

This is of course less performant then writing

@eval foo() = Foo($(types[1]), $(types[2]), etc.)

Long story short, here is my code:

using BenchmarkTools

macro io(data, parameters...)
    eval(data)
    struct_name = data.args[2]
    fields = filter(f->isa(f, Expr), data.args[3].args)
    types = [f.args[2] for f in fields]
    function_args = [:(:call)]
    @eval unpack(io, ::Type{$(struct_name)}) = begin
        # hardcoded version as proof of concept, this works and is super fast
        $(struct_name)(ntoh(read(io, $(types[1]))), ntoh(read(io, $(types[2]))))
        # this works too but "bakes in" the `eval()` into the function definition -> super slow
        #$(struct_name)([ntoh(read(io, eval(t))) for t in $(types)]...)
    end
end

@io struct Foo  # this struct could have any number of fields
    a::Int32
    b::Float32
end

# a huge buffer as random data pool for testing
buf = IOBuffer(rand(UInt8, sizeof(Foo)*100000000))  

@btime unpack($buf, Foo) # gives ~3ns for the hardcoded and ~800ns for the dynamic `eval`

The @io macro essentially takes a struct, evaluates it and additionally it creates a function

unpack(io, ::Type{StructName}) = StructName(ntoh(read(io, TypeOfField1)), ntoh(read(io, TypeOfField2)), etc.)

I hope it’s clear.

I also tried to build up the entire call tree using nested Expr but it seemed too complicated to be true :wink:

A little bit of background: I am doing this function generation since according to my performance studies, this is by far the fastest way of parsing big endian data. I thought calling ntoh() and read() multiple times would have a huge impact on the performance but it seems that Julia and LLVM do some magic there. I also tried StructIO from Keno but it’s way slower (around 1us).

Thanks in advance for your time!

2

Well, first, if you use a macro, you should usually not use eval. It’s not as bad in your case since the macro can only be called in global scope anyway. However, you do still need to call the eval for the correct module, i.e. you need eval(__module__, ...) and @eval __module__ ....

But you can just generate this? You just need to generate the AST that is Foo($(types[1]), $(types[2])). As long as such an AST exists that statisfy your requirement, you should never start to think about eval.

You are just talking about constructing a call that takes a variable number of argumnt, and you can see how such an object can be constructed by just looking at what’s in it:

julia> dump(:(Foo(Int32, Float32)))
Expr
  head: Symbol call
  args: Array{Any}((3,))
    1: Symbol Foo
    2: Symbol Int32
    3: Symbol Float32

Now if you know how to construct a Expr in general you’ll know that you can construct this via,

julia> dump(Expr(:call, :Foo, :Int32, :Float32))
Expr
  head: Symbol call
  args: Array{Any}((3,))
    1: Symbol Foo
    2: Symbol Int32
    3: Symbol Float32

julia> Expr(:call, :Foo, :Int32, :Float32)
:(Foo(Int32, Float32))

Finally, in order to use the splicing syntax you need to know how to splice in a variable number of arguments to an expression:

julia> types = [:Int32, :Float32]
2-element Array{Symbol,1}:
 :Int32
 :Float32

julia> :(Foo($(types...)))
:(Foo(Int32, Float32))

julia> dump(:(Foo($(types...))))
Expr
  head: Symbol call
  args: Array{Any}((3,))
    1: Symbol Foo
    2: Symbol Int32
    3: Symbol Float32
4 Likes
3

And for an improved version of your macro including correct escaping and minimum input validation.

macro io(data)
    struct_name = data.args[2]
    types = []
    for f in data.args[3].args
        isa(f, LineNumberNode) && continue
        isa(f, Symbol) && error("Untyped field")
        Meta.isexpr(f, :(::)) || error("")
        push!(types, f.args[2])
    end
    quote
        $(esc(data))
        $(@__MODULE__).unpack(io, ::Type{$(esc(struct_name))}) = $(esc(struct_name))($([:(ntoh(read(io, $(esc(t))))) for t in types]...))
        nothing
    end
end
2 Likes
4

Awesome, I was already trying to figure out how to connect the individual pieces which you described very well and now you even provide a full solution to the initial. Many thanks!

Now I am going to dive into the details… :wink:

5

On and you also don’t need to completely throw away splicing just becauses you don’t know how to construct one expression. You can use Expr to construct the part that you don’t know how to construct with splicing. You can construct the arguments to it using other methods and you can also splice the result into the final/parent expression.

i.e. you can do,

args = [:(ntoh(....)) for t in types]
call = Expr(:call, esc(struct_name), args...) # :($(esc(struct_name))($(args...)))
quote
....
unpack(...) = $(call)
end
1 Like
6

OK thanks! It’s really hard to get into that with all these “layers” of expressions, quotes and escapes, anyways, learning by doing with expert comments is probably the best method.

7

… I agree it get’s very LISPy … I’d recommend just assign intermediate results to variables when you get too many layers of parenthesis. i.e. from

e = :(f($([:(g($(esc(t)))) for t in types]...)))

to

function call_g(t)
    et = esc(t)
    :(g($et))
end
args = [call_g(t) for t in types]
e = :(f($(args...)))

until you are more familiar with some patterns…

1 Like
8

Yep, that’s a really nice method actually, glad to know about it now! Also the trick with the dump to reverse engineer things is, well “obvious” but I didn’t think about it :wink:

9

I have two problems though which I still do not understand.

The $(@__MODULE__).unpack line causes a UndefVarError: unpack not defined. I was able to circumvent this by defining an unpack = nothing in the global scope (I am prototyping in a Jupyter notebook). I also tried inside the module but I get the same message.

The second problem is that I get ~63 ns (1 allocation: 16 bytes) instead of the ~3ns 0 allocations of the hardcoded version.
I went through your code and it seems to me that it generates the exact same AST, so wondering where the difference is coming from?

10

Your version is defining a function in the module using the macro. My version is extending a function defined in the module where the macro is defined. The latter is usually what you want since the caller of the macro will then be extending the same function which is how multiple dispatch works. You need function unpack end in the macro’s module if it is not already defined.

I see no allocation.

If you “fixed” the problem by defining unpack = nothing, you are actually adding a method to nothing instead. This isn’t good or what you want to do but it’s not where the performance issue come from. The performance issue is because unpack is not a constant so the compiler cannot infer the called function. In your current session, using nothing(buf, Foo) should work (since the name nothing is a constant) and in a new session, define unpack as a function and it should be OK.

1 Like
11

Alright, it all makes sense. I already suspected the hack will interfere with the inference…

Many thanks for your help again I learned a lot in this thread!