# Interpolate expression into macro
**URL:** https://discourse.julialang.org/t/interpolate-expression-into-macro/24204
**Category:** General Usage
**Tags:** metaprogramming
**Created:** [May 14, 2019, 11:26am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204 "2019-05-14T11:26:29Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [May 14, 2019, 11:26am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/1 "2019-05-14T11:26:29Z")
I create some expressions during runtime, which I would like to end up in the expression returned by a macro. I can’t seem to get this to work, and wonder if it is a fundamental limitation? Example:
```julia
function makeexpr(a)
return :($a+1)
end
macro useexpr(ex)
# :($(esc(ex)))
Expr(:quote, :($(Expr(:$, :ex))))
# :($(esc(QuoteNode(ex))))
# :($(QuoteNode(ex)))
# :($(QuoteNode(esc(ex))))
# Expr(:$, :ex)
end
function runit(a)
ex = makeexpr(a)
@useexpr ex
end
julia> runit(2)
:(2 + 1)
```
The returned value is an expression, not the number 3. I tried some other ways of inserting the code of the expression into the macro expression without success. Could a generated function solve this? I tried to `@eval` my expressions into functions and call those, but then I run into world age problems instead:
```julia
function makefun(a)
return @eval () -> $a+1
end
macro usefun(ex)
:($(esc(ex))())
end
function runitfun(a)
ex = makefun(a)
@usefun ex
end
julia> runitfun(2)
ERROR: MethodError: no method matching (::getfield(Main, Symbol("##328#329")))()
The applicable method may be too new: running in world age 26513, while current world is 26514.
```
---
### Author:  [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)
#### Post date: [May 14, 2019, 11:40am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/2 "2019-05-14T11:40:37Z")
I do not completely understand the purpose of function userexpr(ex) but if you change function runit(a) as follows, it gives the required result
```julia
julia> function runit(a)
ex = eval(makeexpr(a))
end
runit (generic function with 1 method)
julia> runit(2)
3
```
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [May 14, 2019, 11:45am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/3 "2019-05-14T11:45:38Z")
Thanks for your reply! If I do that, I have to call `eval` at every invocation of `runit` which I would like to avoid.
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [May 14, 2019, 11:47am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/4 "2019-05-14T11:47:50Z")
Some more background
In the real application, the generated expressions are accessing deeply nested fields in an object. I can either try to interpolate these expressions into a bigger expression, or turn them into functions that update the fields of interest in-place. The function approach works well, but I have to divide it into two so that the world age have time to update 😕
These are my real-world expressions, that are constructed during runtime
```julia
set1expr = quote
vecpartind2vec!((P2.matrix[1]).num.a, (P.matrix[1]).num.a, partind)
vecpartind2vec!((P2.matrix[1]).den.a, (P.matrix[1]).den.a, partind)
end
set2expr = quote
setindex!.((Pres.matrix[1]).num.a, (P2res.matrix[1]).num.a, partind)
setindex!.((Pres.matrix[1]).den.a, (P2res.matrix[1]).den.a, partind)
end
```
If I make them into functions by
```julia
@eval set1fun = (P,P2,partind)-> $set1expr
@eval set2fun = (Pres,P2res,partind)-> $set2expr
```
I may not use the functions until the world age is updated
---
### Author:  [@jonathanBieler](https://discourse.julialang.org/u/jonathanBieler)
#### Post date: [May 14, 2019, 11:47am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/5 "2019-05-14T11:47:57Z")
Macro goes from expression to expression. Here it looks like you are trying to take a value (`a`) and feed it to a macro by first transforming it into an expression, which is still conceptually wrong. If you want to go from value to value just use a function.
Your example would just be:
```julia
makefun(a) = () -> a+1
runit(f) = f()
f = makefun(1)
runit(f)
```
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [May 14, 2019, 11:55am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/6 "2019-05-14T11:55:39Z")
The function approach does not work that well in my case since the expression `a+1` is very complicated and generated at runtime. I updated my answer [above](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/4) with more background
---
### Author:  [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)
#### Post date: [May 14, 2019, 3:52pm UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/7 "2019-05-14T15:52:41Z")
> [@jonathanBieler](#):
>
> just use a function.
I completely agree with you and must admit that my understanding of the issue is flawed.
---
### Author:  [@Vasily\_Pisarev](https://discourse.julialang.org/u/Vasily_Pisarev)
#### Post date: [May 14, 2019, 10:31pm UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/8 "2019-05-14T22:31:06Z")
Macros treat the arguments as a list of expressions and do not care of what those expressions evaluate to. So, `@useexpr ex` expands into `:ex` and `runit()` is equivalent to
```julia
function runit(a)
ex = makeexpr(a)
ex
end
```
Can’t you just use `makeexpr()` as a macro? Then the expression it returns gets “pasted” into the code and compiled.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [May 14, 2019, 10:56pm UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/9 "2019-05-14T22:56:49Z")
Why are you generating complex symbolic expressions at runtime? Some context would be helpful.
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [May 15, 2019, 7:00am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/10 "2019-05-15T07:00:04Z")
The full background is the following: I have implemented a type `Particles <: Real` which wraps a vector for `Float64`. I can define lots of functions for this type, but some functions are hard to define in a meaningful way. In those situations, I want to fall back to propagate each float of the wrapped vector through the function one by one and collect the result in a new `Particles` at the output. If the `particles` appear as an argument to the top-level function, this is really straightforward. The problem appears when my type `Particles` appears as a field nested somewhere in a struct I have no control over. As an example, consider this structure
```julia
julia> P
TransferFunction{ControlSystems.SisoRational{StaticParticles{Float64,100}}}
0.998 ± 0.1
-----------------
1.0*s + 1.0 ± 0.1
Continuous-time transfer function model
julia> dump(P)
TransferFunction{ControlSystems.SisoRational{StaticParticles{Float64,100}}}
matrix: Array{ControlSystems.SisoRational{StaticParticles{Float64,100}}}((1, 1))
1: ControlSystems.SisoRational{StaticParticles{Float64,100}}
num: Polynomials.Poly{StaticParticles{Float64,100}}
a: Array{StaticParticles{Float64,100}}((1,)) StaticParticles{Float64,100}[0.998 ± 0.1]
var: Symbol x
den: Polynomials.Poly{StaticParticles{Float64,100}}
a: Array{StaticParticles{Float64,100}}((2,)) StaticParticles{Float64,100}[1.0 ± 0.1, 1.0]
var: Symbol x
Ts: Float64 0.0
nu: Int64 1
ny: Int64 1
```
The particles appear deeply nested in the object `P` which is the input to my function, and can be accessed by `P.matrix[1].num.a[:]` and `P.matrix[1].den.a[:]`. which are the expressions I am generating.
I want to create an object identical to `P`, let’s call it `P2`, but in all places where `Particles` appear, I want to have `Float64` instead. I then want to populate `P2` with particle index `i` for `eachindex(particles)`. I then call my function with `P2` which is completely free of weird particle types and put the result of the computation into an output object with the desired `Particles` fields.
The real code to generate the expressions is [here](https://github.com/baggepinnen/MonteCarloMeasurements.jl/blob/281a59fb03bfc02fd7894889d2073b14e2a6b878/src/deconstruct.jl#L94)
I turn these expressions into functions that get and set fields in the generated intermediate buffers `P2` and `Pres` to store the result. The buffer is autogenerated [here](https://github.com/baggepinnen/MonteCarloMeasurements.jl/blob/deconstruct/src/deconstruct.jl#L35) and it mostly works okay at the moment. The only problem is with the world age, causing me to not be able to use the getter and setter functions until the world age is updated. My current strategy is to make the user create a `Workspace` object which creates the internal buffer `P2` and the `eval`ed functions, and then call this workspace object with the desired function in a later stage:
```julia
P = tf(1 ∓ 0.1, [1, 1∓0.1]) # The P shown above
w = Workspace(P) # Creates work buffer `P2` and some getter/setter functions
f = x->c2d(x,0.1) # This is the computation I want to perform on P but which fails if P contains Particles
Pd = w(f) # This works
```
However, if created and called in the same function:
```julia
tt = function (P)
w = Workspace(P)
f = x->c2d(x,0.1)
@time Pd = w(f)
end
@test_throws MethodError tt(P) # This causes a world-age problem. If this tests suddenly break, it would be nice and we can get rid of the intermediate workspace object.
```
---
### Author:  [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)
#### Post date: [May 15, 2019, 10:37am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/11 "2019-05-15T10:37:20Z")
Why not define all the functions you want to act on the vector wrapped by `Particles` element-wise using meta-programming?
```julia
for f in all_funcs
@eval $f(p::Particles) = Particles(f.(p.vec))
end
```
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [May 15, 2019, 10:45am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/12 "2019-05-15T10:45:32Z")
That is how most functions are [defined](https://github.com/baggepinnen/MonteCarloMeasurements.jl/blob/53cfc769581bf496b7db1ce87a1796386e4c6925/src/register_primitive.jl#L15).
The problem only arises when there is a third-party function that takes third-party objects as inputs, where `Particles` are nested inside the object.
---
### Author:  [@Vasily\_Pisarev](https://discourse.julialang.org/u/Vasily_Pisarev)
#### Post date: [May 15, 2019, 10:48am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/13 "2019-05-15T10:48:00Z")
In your `get_setter_funs()`, are elements of `paths` known at compile-time or have to be determined at runtime?
If they are known at compile-time, then you can make two macros out of the function, I guess. One would generate an equivalent of `(P,P2,partind)-> $set1expr`, the other one `(Pres,P2res,partind)-> $set2expr`, and then you can bind the lambdas to variables as
```julia
set1fun = @set1expr (path1, path2, path3)
set2fun = @set2expr (path1, path2, path3)
```
---
### Author:  [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)
#### Post date: [May 15, 2019, 10:52am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/14 "2019-05-15T10:52:06Z")
> [@baggepinnen](#):
>
> The problem only arises when there is a third-party function that takes third-party objects as inputs, where `Particles` are nested inside the object.
Hmm but at the end of the day, only a known set of functions can be called on the `Particles` struct hidden inside, right?
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [May 15, 2019, 11:15am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/15 "2019-05-15T11:15:57Z")
The elements of `paths` are known at compile time provided that all fields of all objects leading down to the `Particles` are concretely typed. I started out writing a `get_setter_funs` that operated on types rather than instances, but abandoned it since I can not control whether or not a third-party type has concretely typed fields and this would be a limitation. I could have two implementations and fall back to the instances-based implementation if the type-based fails.
I’ll also have a look at and benchmark `FunctionWrappers` and `invokelatest` to see if the performance hit is small enough for my usecase.
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [May 15, 2019, 11:17am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/16 "2019-05-15T11:17:45Z")
That is correct, and my goal is to cover all such from `Base` etc. User defined and third-party functions can be made to work with `Particles` quite easily, provided that the particles appear as a direct function argument, not nested in another object.
---
### Author:  [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)
#### Post date: [May 15, 2019, 11:33am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/17 "2019-05-15T11:33:33Z")
I am not sure we have the same understanding here. Here is what I think you are doing:
```julia
struct Particles
x::Float64
end
Base.rand(::Type{<:Particles}) = Particles(rand())
Base.:+(p1::Particles, p2::Particles) = Particles(p1.x + p2.x)
struct Wrapper{T}
p::T
end
third_party(w::Wrapper) = rand(typeof(w.p)) + w.p
# Works because rand and + are defined
third_party(Wrapper(rand(Particles)))
```
Instead of just `rand` and `+`, if _all_ the relevant `Base` functions were defined properly, any “third party” function should _just work_ if dispatch works correctly at all the right points, much like how `ForwardDiff.Dual` works by just being defined on a finite set of functions. I don’t know if we are talking about the same thing, so I am trying to clarify.
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [May 15, 2019, 11:41am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/18 "2019-05-15T11:41:41Z")
You have more or less the right understanding. The problem is that there are base functions for which I cannot define a method acting on `Particles`. One such function is `eigvals(m::Matrix{Particles})`. `eigvals` produces a sorted list of eigenvalues (the sorting is a discontinuous operation in a really nasty sense) that any subsequent function makes use of, e.g.,
```julia
function third_party_fun(obj::ThirdParty)
e = eigvals(obj.M)
do_something_with_eigvals(e)
end
```
in this case, `eigvals(obj.M)` will fail because it is ill-defined for matrices with `Particles`. I would therefore like to construct one `obj` for each particle, calculate the whole function `third_party_fun` using each particle-free `obj` and then assemble the result in the end.
If the situation was `third_party_fun(M::Matrix{Particles})` there would be no problem, defining this dispatch and to disassemble and reassemble the particle matrix is easy. When `M::Matrix{Particles}` appears inside `obj` things get difficult.
---
### Author:  [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)
#### Post date: [May 15, 2019, 11:52am UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/19 "2019-05-15T11:52:55Z")
Hmm, I assume that `M isa Matrix{Particles}`. I have 2 comments:
1. If `eigvals` makes sense for `Matrix{Particles}` but in a way that is not supported by the generic fallbacks in Julia, can’t you define the method `eigvals(m::Matrix{Particles})` in your package instead?
2. If `eigvals` doesn’t make sense for `Matrix{Particles}`, then an error is the correct behavior, but you can still overload it to throw your own error.
I assume the first is the case or you wouldn’t be using it as an example. I am still not sure why you want to work on `third_party_fun` directly though as opposed to the finite set of functions it is calling on a finite set of `Particles`-based types, e.g. `Matrix{Particles}` IIUC.
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [May 15, 2019, 12:01pm UTC](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204/20 "2019-05-15T12:01:13Z")
Defining `eigvals(m::Matrix{Particles})` is not a good idea (I’ve tried), it really is ill-defined. Currently, the user gets an error if that method is called. The outer `third_party_fun` is well defined for each individual `obj` the `obj` with particles represent. Consider this very simple case
```julia
function t(p)
p > 0 ? 1 : 0
end
```
This is ill-defined for uncertain `p`. If half of the particles are positive and half negative, the boolean `p>0` is 50% true, which is not something julia accepts. for each particle individually, the function `t` is well defined, and half the answers are 1 and the other half 0. I can assemble that into a `p2` with half particles =1 and half=2.
Overloading `<` is not an option since booleans are required in boolean contexts.
Note: in my example `t` we have the simple situation of particles appearing directly as input to the function, this situation is easy to handle.
[Next page](https://discourse.julialang.org/t/interpolate-expression-into-macro/24204.md?page=2)