# Converting symbols to types and splatting inside @eval
**URL:** https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728
**Category:** General Usage
**Tags:** macros, metaprogramming
**Created:** [December 1, 2019, 10:39pm UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728 "2019-12-01T22:39:27Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@tamasgal](https://discourse.julialang.org/u/tamasgal)
#### Post date: [December 1, 2019, 10:39pm UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728/1 "2019-12-01T22:39:28Z")
I am trying to wrap my head around metaprogramming and macros and I struggle to figure this out and I think I am on the wrong path. I have the feeling that I need to build up an expression tree programmatically instead of the current approach, but let’s see.
To avoid an XY problem I’d like to provide a not so MWE (see the end) since there might be a better way to do what I am up to.
My specific problem is that I want to generate a function call with an arbitrary number of arguments. In my macro I have access to a list of types in form of a `Vector{Symbol}` like this:
```julia
types = Symbol[:Int32, :Float32] # the length of this varies
```
which I’d like to turn into (dummy example):
```julia
foo() = Foo(Int32, Float32)
```
I managed to do it but the problem is that I am calling `eval()` which is then part of the function definition and I lose a lot of performance. It’s looking something like this inside my macro definition:
```julia
types = Symbol[:Int32, :Float32]
@eval foo() = Foo([eval(t) for t in $(types)]...)
```
This is of course less performant then writing
```julia
@eval foo() = Foo($(types[1]), $(types[2]), etc.)
```
Long story short, here is my code:
```julia
using BenchmarkTools
macro io(data, parameters...)
eval(data)
struct_name = data.args[2]
fields = filter(f->isa(f, Expr), data.args[3].args)
types = [f.args[2] for f in fields]
function_args = [:(:call)]
@eval unpack(io, ::Type{$(struct_name)}) = begin
# hardcoded version as proof of concept, this works and is super fast
$(struct_name)(ntoh(read(io, $(types[1]))), ntoh(read(io, $(types[2]))))
# this works too but "bakes in" the `eval()` into the function definition -> super slow
#$(struct_name)([ntoh(read(io, eval(t))) for t in $(types)]...)
end
end
@io struct Foo # this struct could have any number of fields
a::Int32
b::Float32
end
# a huge buffer as random data pool for testing
buf = IOBuffer(rand(UInt8, sizeof(Foo)*100000000))
@btime unpack($buf, Foo) # gives ~3ns for the hardcoded and ~800ns for the dynamic `eval`
```
The `@io` macro essentially takes a struct, evaluates it and additionally it creates a function
```julia
unpack(io, ::Type{StructName}) = StructName(ntoh(read(io, TypeOfField1)), ntoh(read(io, TypeOfField2)), etc.)
```
I hope it’s clear.
I also tried to build up the entire call tree using nested `Expr` but it seemed too complicated to be true 😉
A little bit of background: I am doing this function generation since according to my performance studies, this is by far the fastest way of parsing big endian data. I thought calling `ntoh()` and `read()` multiple times would have a huge impact on the performance but it seems that Julia and LLVM do some magic there. I also tried `StructIO` from Keno but it’s way slower (around 1us).
Thanks in advance for your time!
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [December 1, 2019, 11:48pm UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728/2 "2019-12-01T23:48:34Z")
Well, first, if you use a macro, you should usually not use `eval`. It’s not as bad in your case since the macro can only be called in global scope anyway. However, you do still need to call the `eval` for the correct module, i.e. you need `eval( __module__ , ...)` and `@eval __module__...`.
> [@tamasgal](#):
>
> ```julia
> @eval foo() = Foo($(types[1]), $(types[2]), etc.)
>
> ```
But you can just generate this? You just need to generate the AST that is `Foo($(types[1]), $(types[2]))`. As long as such an AST exists that statisfy your requirement, you should **never** start to think about `eval`.
You are just talking about constructing a `call` that takes a variable number of argumnt, and you can see how such an object can be constructed by just looking at what’s in it:
```julia
julia> dump(:(Foo(Int32, Float32)))
Expr
head: Symbol call
args: Array{Any}((3,))
1: Symbol Foo
2: Symbol Int32
3: Symbol Float32
```
Now if you know how to construct a `Expr` in general you’ll know that you can construct this via,
```julia
julia> dump(Expr(:call, :Foo, :Int32, :Float32))
Expr
head: Symbol call
args: Array{Any}((3,))
1: Symbol Foo
2: Symbol Int32
3: Symbol Float32
julia> Expr(:call, :Foo, :Int32, :Float32)
:(Foo(Int32, Float32))
```
Finally, in order to use the splicing syntax you need to know how to splice in a variable number of arguments to an expression:
```julia
julia> types = [:Int32, :Float32]
2-element Array{Symbol,1}:
:Int32
:Float32
julia> :(Foo($(types...)))
:(Foo(Int32, Float32))
julia> dump(:(Foo($(types...))))
Expr
head: Symbol call
args: Array{Any}((3,))
1: Symbol Foo
2: Symbol Int32
3: Symbol Float32
```
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [December 2, 2019, 12:03am UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728/3 "2019-12-02T00:03:38Z")
And for an improved version of your macro including correct escaping and minimum input validation.
```julia
macro io(data)
struct_name = data.args[2]
types = []
for f in data.args[3].args
isa(f, LineNumberNode) && continue
isa(f, Symbol) && error("Untyped field")
Meta.isexpr(f, :(::)) || error("")
push!(types, f.args[2])
end
quote
$(esc(data))
$(@ __MODULE__ ).unpack(io, ::Type{$(esc(struct_name))}) = $(esc(struct_name))($([:(ntoh(read(io, $(esc(t))))) for t in types]...))
nothing
end
end
```
---
### Author:  [@tamasgal](https://discourse.julialang.org/u/tamasgal)
#### Post date: [December 2, 2019, 12:07am UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728/4 "2019-12-02T00:07:32Z")
Awesome, I was already trying to figure out how to connect the individual pieces which you described very well and now you even provide a full solution to the initial. Many thanks!
Now I am going to dive into the details… 😉
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [December 2, 2019, 12:09am UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728/5 "2019-12-02T00:09:28Z")
> [@tamasgal](#):
>
> I also tried to build up the entire call tree using nested `Expr` but it seemed too complicated to be true 😉
On and you also don’t need to completely throw away splicing just becauses you don’t know how to construct one expression. You can use `Expr` to construct the part that you don’t know how to construct with splicing. You can construct the arguments to it using other methods and you can also splice the result into the final/parent expression.
i.e. you can do,
```julia
args = [:(ntoh(....)) for t in types]
call = Expr(:call, esc(struct_name), args...) # :($(esc(struct_name))($(args...)))
quote
....
unpack(...) = $(call)
end
```
---
### Author:  [@tamasgal](https://discourse.julialang.org/u/tamasgal)
#### Post date: [December 2, 2019, 12:14am UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728/6 "2019-12-02T00:14:37Z")
OK thanks! It’s really hard to get into that with all these “layers” of expressions, quotes and escapes, anyways, learning by doing with expert comments is probably the best method.
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [December 2, 2019, 12:18am UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728/7 "2019-12-02T00:18:53Z")
> [@tamasgal](#):
>
> OK thanks! It’s really hard to get into that with all these “layers” of expressions, quotes and escapes,
… I agree it get’s very LISPy … I’d recommend just assign intermediate results to variables when you get too many layers of parenthesis. i.e. from
```julia
e = :(f($([:(g($(esc(t)))) for t in types]...)))
```
to
```julia
function call_g(t)
et = esc(t)
:(g($et))
end
args = [call_g(t) for t in types]
e = :(f($(args...)))
```
until you are more familiar with some patterns…
---
### Author:  [@tamasgal](https://discourse.julialang.org/u/tamasgal)
#### Post date: [December 2, 2019, 12:20am UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728/8 "2019-12-02T00:20:07Z")
Yep, that’s a really nice method actually, glad to know about it now! Also the trick with the `dump` to reverse engineer things is, well “obvious” but I didn’t think about it 😉
---
### Author:  [@tamasgal](https://discourse.julialang.org/u/tamasgal)
#### Post date: [December 2, 2019, 12:36am UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728/9 "2019-12-02T00:36:35Z")
I have two problems though which I still do not understand.
The `$(@ __MODULE__ ).unpack` line causes a `UndefVarError: unpack not defined`. I was able to circumvent this by defining an `unpack = nothing` in the global scope (I am prototyping in a Jupyter notebook). I also tried inside the module but I get the same message.
The second problem is that I get ` ~63 ns (1 allocation: 16 bytes)` instead of the `~3ns 0 allocations` of the hardcoded version.
I went through your code and it seems to me that it generates the exact same AST, so wondering where the difference is coming from?
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [December 2, 2019, 12:52am UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728/10 "2019-12-02T00:52:30Z")
> [@tamasgal](#):
>
> The `$(@ __MODULE__ ).unpack` line causes a `UndefVarError: unpack not defined` . I was able to circumvent this by defining an `unpack = nothing` in the global scope (I am prototyping in a Jupyter notebook). I also tried inside the module but I get the same message.
Your version is defining a function in the module using the macro. My version is extending a function defined in the module where the macro is defined. The latter is usually what you want since the caller of the macro will then be extending the same function which is how multiple dispatch works. You need `function unpack end` in the macro’s module if it is not already defined.
> [@tamasgal](#):
>
> The second problem is that I get ` ~63 ns (1 allocation: 16 bytes)` instead of the `~3ns 0 allocations` of the hardcoded version.
I see no allocation.
> [@tamasgal](#):
>
> I went through your code and it seems to me that it generates the exact same AST, so wondering where the difference is coming from?
If you “fixed” the problem by defining `unpack = nothing`, you are actually adding a method to `nothing` instead. This isn’t good or what you want to do but it’s not where the performance issue come from. The performance issue is because `unpack` is not a constant so the compiler cannot infer the called function. In your current session, using `nothing(buf, Foo)` should work (since the name `nothing` is a constant) and in a new session, define `unpack` as a function and it should be OK.
---
### Author:  [@tamasgal](https://discourse.julialang.org/u/tamasgal)
#### Post date: [December 2, 2019, 7:52am UTC](https://discourse.julialang.org/t/converting-symbols-to-types-and-splatting-inside-eval/31728/11 "2019-12-02T07:52:48Z")
Alright, it all makes sense. I already suspected the hack will interfere with the inference…
Many thanks for your help again I learned a lot in this thread!