Interpolate expression into macro

I create some expressions during runtime, which I would like to end up in the expression returned by a macro. I can’t seem to get this to work, and wonder if it is a fundamental limitation? Example:

function makeexpr(a)
    return :($a+1)
end

macro useexpr(ex)
    # :($(esc(ex)))
    Expr(:quote, :($(Expr(:$, :ex))))
    # :($(esc(QuoteNode(ex))))
    # :($(QuoteNode(ex)))
    # :($(QuoteNode(esc(ex))))
    # Expr(:$, :ex)
end

function runit(a)
    ex = makeexpr(a)
    @useexpr ex
end

julia> runit(2)
:(2 + 1)

The returned value is an expression, not the number 3. I tried some other ways of inserting the code of the expression into the macro expression without success. Could a generated function solve this? I tried to @eval my expressions into functions and call those, but then I run into world age problems instead:

function makefun(a)
    return @eval () -> $a+1
end

macro usefun(ex)
    :($(esc(ex))())
end

function runitfun(a)
    ex = makefun(a)
    @usefun ex
end

julia> runitfun(2)
ERROR: MethodError: no method matching (::getfield(Main, Symbol("##328#329")))()
The applicable method may be too new: running in world age 26513, while current world is 26514.
2 Likes

I do not completely understand the purpose of function userexpr(ex) but if you change function runit(a) as follows, it gives the required result

julia> function runit(a)
           ex = eval(makeexpr(a))
       end
runit (generic function with 1 method)

julia> runit(2)
3

Thanks for your reply! If I do that, I have to call eval at every invocation of runit which I would like to avoid.

Some more background

In the real application, the generated expressions are accessing deeply nested fields in an object. I can either try to interpolate these expressions into a bigger expression, or turn them into functions that update the fields of interest in-place. The function approach works well, but I have to divide it into two so that the world age have time to update :confused:

These are my real-world expressions, that are constructed during runtime

set1expr = quote
    vecpartind2vec!((P2.matrix[1]).num.a, (P.matrix[1]).num.a, partind)
    vecpartind2vec!((P2.matrix[1]).den.a, (P.matrix[1]).den.a, partind)
end
set2expr = quote
    setindex!.((Pres.matrix[1]).num.a, (P2res.matrix[1]).num.a, partind)
    setindex!.((Pres.matrix[1]).den.a, (P2res.matrix[1]).den.a, partind)
end

If I make them into functions by

@eval set1fun = (P,P2,partind)-> $set1expr
@eval set2fun = (Pres,P2res,partind)-> $set2expr

I may not use the functions until the world age is updated

Macro goes from expression to expression. Here it looks like you are trying to take a value (a) and feed it to a macro by first transforming it into an expression, which is still conceptually wrong. If you want to go from value to value just use a function.

Your example would just be:

makefun(a) = () -> a+1
runit(f) = f()

f = makefun(1)
runit(f) 

The function approach does not work that well in my case since the expression a+1 is very complicated and generated at runtime. I updated my answer above with more background

I completely agree with you and must admit that my understanding of the issue is flawed.

Macros treat the arguments as a list of expressions and do not care of what those expressions evaluate to. So, @useexpr ex expands into :ex and runit() is equivalent to

function runit(a)
    ex = makeexpr(a)
    ex
end

Can’t you just use makeexpr() as a macro? Then the expression it returns gets “pasted” into the code and compiled.

Why are you generating complex symbolic expressions at runtime? Some context would be helpful.

2 Likes

The full background is the following: I have implemented a type Particles <: Real which wraps a vector for Float64. I can define lots of functions for this type, but some functions are hard to define in a meaningful way. In those situations, I want to fall back to propagate each float of the wrapped vector through the function one by one and collect the result in a new Particles at the output. If the particles appear as an argument to the top-level function, this is really straightforward. The problem appears when my type Particles appears as a field nested somewhere in a struct I have no control over. As an example, consider this structure

julia> P
TransferFunction{ControlSystems.SisoRational{StaticParticles{Float64,100}}}
   0.998 ± 0.1
-----------------
1.0*s + 1.0 ± 0.1

Continuous-time transfer function model

julia> dump(P)
TransferFunction{ControlSystems.SisoRational{StaticParticles{Float64,100}}}
  matrix: Array{ControlSystems.SisoRational{StaticParticles{Float64,100}}}((1, 1))
    1: ControlSystems.SisoRational{StaticParticles{Float64,100}}
      num: Polynomials.Poly{StaticParticles{Float64,100}}
        a: Array{StaticParticles{Float64,100}}((1,)) StaticParticles{Float64,100}[0.998 ± 0.1]
        var: Symbol x
      den: Polynomials.Poly{StaticParticles{Float64,100}}
        a: Array{StaticParticles{Float64,100}}((2,)) StaticParticles{Float64,100}[1.0 ± 0.1, 1.0]
        var: Symbol x
  Ts: Float64 0.0
  nu: Int64 1
  ny: Int64 1

The particles appear deeply nested in the object P which is the input to my function, and can be accessed by P.matrix[1].num.a[:] and P.matrix[1].den.a[:]. which are the expressions I am generating.

I want to create an object identical to P, let’s call it P2, but in all places where Particles appear, I want to have Float64 instead. I then want to populate P2 with particle index i for eachindex(particles). I then call my function with P2 which is completely free of weird particle types and put the result of the computation into an output object with the desired Particles fields.

The real code to generate the expressions is here
I turn these expressions into functions that get and set fields in the generated intermediate buffers P2 and Pres to store the result. The buffer is autogenerated here and it mostly works okay at the moment. The only problem is with the world age, causing me to not be able to use the getter and setter functions until the world age is updated. My current strategy is to make the user create a Workspace object which creates the internal buffer P2 and the evaled functions, and then call this workspace object with the desired function in a later stage:

P = tf(1 ∓ 0.1, [1, 1∓0.1]) # The P shown above
w = Workspace(P)   # Creates work buffer `P2` and some getter/setter functions
f = x->c2d(x,0.1) # This is the computation I want to perform on P but which fails if P contains Particles
Pd = w(f) # This works

However, if created and called in the same function:

    tt = function (P)
        w = Workspace(P)
        f = x->c2d(x,0.1)
        @time Pd = w(f)
    end
    @test_throws MethodError tt(P) # This causes a world-age problem. If this tests suddenly break, it would be nice and we can get rid of the intermediate workspace object.

Why not define all the functions you want to act on the vector wrapped by Particles element-wise using meta-programming?

for f in all_funcs
    @eval $f(p::Particles) = Particles(f.(p.vec))
end

That is how most functions are defined.

The problem only arises when there is a third-party function that takes third-party objects as inputs, where Particles are nested inside the object.

In your get_setter_funs(), are elements of paths known at compile-time or have to be determined at runtime?

If they are known at compile-time, then you can make two macros out of the function, I guess. One would generate an equivalent of (P,P2,partind)-> $set1expr, the other one (Pres,P2res,partind)-> $set2expr, and then you can bind the lambdas to variables as

set1fun = @set1expr (path1, path2, path3)
set2fun = @set2expr (path1, path2, path3)

Hmm but at the end of the day, only a known set of functions can be called on the Particles struct hidden inside, right?

The elements of paths are known at compile time provided that all fields of all objects leading down to the Particles are concretely typed. I started out writing a get_setter_funs that operated on types rather than instances, but abandoned it since I can not control whether or not a third-party type has concretely typed fields and this would be a limitation. I could have two implementations and fall back to the instances-based implementation if the type-based fails.

I’ll also have a look at and benchmark FunctionWrappers and invokelatest to see if the performance hit is small enough for my usecase.

That is correct, and my goal is to cover all such from Base etc. User defined and third-party functions can be made to work with Particles quite easily, provided that the particles appear as a direct function argument, not nested in another object.

I am not sure we have the same understanding here. Here is what I think you are doing:

struct Particles
    x::Float64
end
Base.rand(::Type{<:Particles}) = Particles(rand())
Base.:+(p1::Particles, p2::Particles) = Particles(p1.x + p2.x)

struct Wrapper{T}
    p::T
end
third_party(w::Wrapper) = rand(typeof(w.p)) + w.p

# Works because rand and + are defined
third_party(Wrapper(rand(Particles)))

Instead of just rand and +, if all the relevant Base functions were defined properly, any “third party” function should just work if dispatch works correctly at all the right points, much like how ForwardDiff.Dual works by just being defined on a finite set of functions. I don’t know if we are talking about the same thing, so I am trying to clarify.

1 Like

You have more or less the right understanding. The problem is that there are base functions for which I cannot define a method acting on Particles. One such function is eigvals(m::Matrix{Particles}). eigvals produces a sorted list of eigenvalues (the sorting is a discontinuous operation in a really nasty sense) that any subsequent function makes use of, e.g.,

function third_party_fun(obj::ThirdParty)
    e = eigvals(obj.M)
    do_something_with_eigvals(e)
end

in this case, eigvals(obj.M) will fail because it is ill-defined for matrices with Particles. I would therefore like to construct one obj for each particle, calculate the whole function third_party_fun using each particle-free obj and then assemble the result in the end.
If the situation was third_party_fun(M::Matrix{Particles}) there would be no problem, defining this dispatch and to disassemble and reassemble the particle matrix is easy. When M::Matrix{Particles} appears inside obj things get difficult.

Hmm, I assume that M isa Matrix{Particles}. I have 2 comments:

  1. If eigvals makes sense for Matrix{Particles} but in a way that is not supported by the generic fallbacks in Julia, can’t you define the method eigvals(m::Matrix{Particles}) in your package instead?
  2. If eigvals doesn’t make sense for Matrix{Particles}, then an error is the correct behavior, but you can still overload it to throw your own error.

I assume the first is the case or you wouldn’t be using it as an example. I am still not sure why you want to work on third_party_fun directly though as opposed to the finite set of functions it is calling on a finite set of Particles-based types, e.g. Matrix{Particles} IIUC.

1 Like

Defining eigvals(m::Matrix{Particles}) is not a good idea (I’ve tried), it really is ill-defined. Currently, the user gets an error if that method is called. The outer third_party_fun is well defined for each individual obj the obj with particles represent. Consider this very simple case

function t(p)
    p > 0 ? 1 : 0
end

This is ill-defined for uncertain p. If half of the particles are positive and half negative, the boolean p>0 is 50% true, which is not something julia accepts. for each particle individually, the function t is well defined, and half the answers are 1 and the other half 0. I can assemble that into a p2 with half particles =1 and half=2.

Overloading < is not an option since booleans are required in boolean contexts.

Note: in my example t we have the simple situation of particles appearing directly as input to the function, this situation is easy to handle.