Macro to create variables from arrays of names and values

Usage
#1

Hi,

I am stuck writing a macro to create variables from an array of names (as strings) and an array of values to be assigned.

So this is what I start with:

Names = ["R", "k", "y"]
Values = [1.01, 4.7, 1.3]

what I would like to have is a macro which translates to the following (or equivalent) in the local scope:

R, k, y = [1.01, 4.7, 1.3]

I read the documentation section on metaprogramming, went through tutorials, and searched on the forum but it is still not clear to me how to solve this. Parameters.jl does not handle and unknown number of variables as I understand it.

The best I could come up with is this:

macro assign_loop(nms,vls)
    expr = quote 
        local ii = 1
        for NMS in $nms
            NMS = $vls[ii]
            ii += 1
        end
    end
    return esc(expr)
end

but when I test it, the “assigned” variable is not recognised.

Test code:

function test(a, b)
    @assign_loop(a, b)
    print(y)
end

test(Symbol.(Names), Values)

Any help would be much appreciated.

1 Like
#2

There is no general way to do this in Julia. A macro cannot help you here because the @assign_loop macro just sees the literal symbols :a and :b, not whatever values they happen to contain in some particular case. The input to a macro is syntax; in your case the names :a and :b. What value those happen to be bound to inside test is not something that macro can or should know about.

Furthermore, there is no way to dynamically create local variables in Julia–this is important for performance reasons, since it makes the code much easier to reason about for the compiler.

Fortunately, there are a lot of other good tools to make life easier in situations like this, like named tuples and @unpack macros. Maybe if you can tell us more about what you’re actually trying to achieve in your code, then we can help point you in the right direction.

Edit: Also, hi! Welcome!

4 Likes
#3

Could you tell more about what you want to do with those local variables? Given that it’s impossible to create local variables from runtime information maybe there are other ways you can get to your goal

1 Like
#4

Rather than generating variables (which is almost always not what you should be doing), perhaps the data structure you want is a dictionary:

julia> d = Dict("R" => 1.01, "k" => 4.7, "y" => 1.3)
Dict{String,Float64} with 3 entries:
  "k" => 4.7
  "R" => 1.01
  "y" => 1.3

julia> d["k"]
4.7
3 Likes
#5

So here is the code I would like this to be embedded in:

using ForwardDiff

var_init = .9
Names = ["A","Rbar","z","eps_z","beta","rhoz"]
Values = [var_init,var_init,var_init, 0.0, .975, .75]

idx = ["Rbar"]
return_index = 3

function wrapper(x::Real; idx = idx, return_index = return_index, Values = Values, Names = Names)

    Values[Names .== idx] .= x 

    #values
    A       = Values[Names .== "A"][1]
    Rbar    = Values[Names .== "Rbar"][1]
    z       = Values[Names .== "z"][1]

   #parameters
    eps_z   = Values[Names .== "eps_z"][1]
    beta    = Values[Names .== "beta"][1]
    rhoz    = Values[Names .== "rhoz"][1]

   #equations
    F    = Array{Number}(undef, 3)
    F[1] = A -( exp(z))
    F[2] = z -( rhoz*z+eps_z)
    F[3] = Rbar - 1/beta

    return F[return_index]
end

ForwardDiff.derivative(wrapper,var_init)

It doesn’t run as of now, but I hope the idea is clear.

The idea is that the #variable, #parameter and #equation parts are exchangeable. With idx and return_index I can change the other parts. The variables, parameters and equations are parsed from a text file and ultimately end up in arrays of strings. Now I would like this wrapper to be generic enough to handle anything which comes from the text file and pass it on to the solver or take derivatives (this example).

The macro should then replace the manual assignment of the variables and the construction of F.

But maybe I am using the wrong tool here.

PS: thanks for the warm welcome

#6

As you can see in the example below I use ForwardDiff and to my understanding that constraints me in using arrays. Also I don’t know the variables upfront. They are parsed from a text file.

Thanks for the welcome and the help.

#7

I begin to understand my trouble here. At a high level I want to parse a text file with equations, parameters values and take derivatives and calculate roots on these equations, given the parameters. The parsing part is taken care of and I get everything handed from python as arrays of strings but converting that in an automatic way to a function that is usable by ForwardDiff and root is where I am stuck.

#8

Well if you want to write a whole new function with arrays of names and symbols, you can just use eval I think, because a function exists at global scope. What you can’t do is use a macro within a function to make that function behave differently with different local variables given runtime values. But a whole new function is fair game

#9

One way to do this is to turn your variables and equation into the syntax of a normal Julia function, then use eval to turn that syntax into a usable function. For example, given the variables “x” and “y” as strings and the equation “sin(x) + y”, you could write a function to generate the following Julia code:

(x, y) -> sin(x) + y

which creates an anonymous function of x and y which computes sin(x) + y.

Here’s a simple example which seems to work:

julia> function generate_equation(variables, equation)
         return eval(:($(Expr(:tuple, Symbol.(variables)...)) -> $(Meta.parse(equation))))
       end
generate_equation (generic function with 1 method)

For example:

julia> variables = ["x", "y"]
2-element Array{String,1}:
 "x"
 "y"

julia> equation = "sin(x) + y"
"sin(x) + y"

julia> f = generate_equation(variables, equation)
#9 (generic function with 1 method)

julia> f(1, 2)
2.8414709848078967

This has some important caveats:

  • It assumes that equation is valid Julia syntax and that it will only refer to the input variables or other globally available names (like the base sin function).
  • It assumes that you trust the author of equation to not put any malicious or unsafe code into the equation. If that author is you, then it’s probably ok. If that author is some person on the internet, then it’s probably not ok.

You can also use the function generated in this manner with ForwardDiff. Since ForwardDiff.gradient expects a function of a single vector input, you just need to wrap f in a function that takes a vector v and then splits that vector into separate arguments. In our case, that’s pretty easy:

julia> ForwardDiff.gradient(v -> f(v[1], v[2]), [1.0, 2.0])
2-element Array{Float64,1}:
 0.5403023058681398
 1.0
3 Likes
#10

I also just tried out a short example of evaling a function with random variable names from outside:

variables = [:a, :b, :c]
vals = [1, 2, 3]

eval(quote function myfunction(x)
    ($(variables...),) = $vals
    a + b + c + x
end end)

myfunction(3)

Of course you wouldn’t write a, b and c in there if you don’t know them beforehand, but I wanted to check that the variables were created correctly.

#11

Many thanks, this was very helpful.

Now I understand better that I need to be careful in creating functions or variables at run / compile time.

In essence, a function in Julia needs to be able to see the arguments at compile time. Is that why you cannot pass additional (parameter) values to ForwardDiff to set up a diff function with a parameter? I solved this by writing a function which writes a function (the wrapper in the above example) with known values.

To come back to the overall problem and how to solve it: I understand I must know variables, parameters and equations before I can set up the function. So could I solve it by just writing the wrapper function and the wrapper of the wrapper (in order to assign new parameters to the wrapper) and parse - eval it?

Here is an attempt:

Meta.parse("function wrapper_sqrd(idx, return_index, model)

    Values = model.values
    Names = model.names
    var_init = model.var_init

    function wrapper(x::Real; idx = idx, return_index = return_index, Values = Values, Names = Names, Equations = Equations)
        Values[Names .== idx] .= x

        for nms in $Names 
            Symbol(nms) = Values[$Names .== nms][1]
        end

        F = Array{Number}(undef, length(Equations))
        ii = 1
        for eqn in Equations
            F[ii] = eqn
            ii += 1
        end
        return F[return_index]
    end

    ForwardDiff.derivative(wrapper,var_init)

end") |> eval

I didn’t test it, but it would be good to know if this is a right path.

#12

In a word, no :wink:

Meta.parse and eval are tools that should be used extremely rarely because they make your code difficult to understand and implement correctly. You should restrict use to just the smallest possible case: in your instance converting the eqn string into executable code. Other than that, you should probably not be using them at all.

Moreover, the way your example is written seems pretty inefficient–you’re going through all this trouble to compute all of the elements of F but then only returning one of them. It’s a little hard to see through to exactly what you want your example to achieve–if you want just a derivative of one function, why not pass that single function in instead?

BTW, I noticed that you do things like Values[Names .== "A"][1] in a bunch of places. That’s actually a really inefficient way to look up a single element in Julia (or, for that matter, in any language). You’re doing the following:

  • Creating a new array of booleans
  • Filling that array by comparing against "A"
  • Creating another array of a subset of Values
  • Throwing away all but the first entry in that array.

That code allocates two new arrays and then throws them away. A much more efficient approach would be:

julia> values[findfirst(isequal("A"), names)]

Comparing performance:

julia> using BenchmarkTools

julia> @btime $values[findfirst(isequal("A"), $names)]
  10.027 ns (0 allocations: 0 bytes)
1

julia> @btime $values[$names .== "A"][1]
  93.668 ns (3 allocations: 224 bytes)
1

about 10x faster, and just as easy to write.

3 Likes
#13

Rather than define a new language for your users to write equations in, why not have them write Julia functions and pass them to your code? That will be a lot more flexible.

8 Likes
#14

First of all, thanks for the help.

Mine is a poor mans solution and I will adopt your proposal throughout.

Now, as to the bigger issue. Let me try to explain why I want all the elements of F.
The end goal is one function of the sort:

f(model, action, name, equation)

where model contains the parameters, values and equations; action can be derivative, fzero, or nlsolve; name indicates for which parameter(s) or value(s) I want to perform the action (e.g. find the root of an equation and give me back the value of that variable); and equation indicates which equation(s) I want to perform the action on.
My thinking was that it is efficient to create one function which can do it all, and call that function later to “solve” my model. Alternatively, I could set up one function at each step of solving the model (finding roots, taking derivatives,…). This solution seemed inefficient because I would create too many functions which look very similar. Hence, my drive to put it all in one.

To be more concrete, here is what I want to start with and create the function(s) from:

equations = ["A -( exp(z))",
             "z -( rhoz*z+eps_z)",
             "Rbar - 1/beta"]

Names = ["A",
         "Rbar",
         "z",
         "eps_z",
         "beta",
         "rhoz"]

var_init = 0.9

Values = [var_init,
          var_init,
          var_init,
          0,
          0.975,
          0.75]
1 Like
#15

My hands are a little tied unfortunately. I want to parse files from an existing DSL (dynare). The parsing is fine and I get everything in strings handed over from python but now I want to turn that into Julia code.
My examples are based on my attempt to parse this model.

If I were to set it up myself I understand now that I could use macros to do a lot of the metaprogramming necessary for solving these kind of models but I struggle to see how to get time indexed variables. Is there a way to define a variable with an index? As in, make Julia understand that k(+1) means variable k in t+1. That would be an interesting avenue to explore.