# What is the right way to construct and then run a Julia function programmatically?
**URL:** https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845
**Category:** General Usage
**Tags:** macros
**Created:** [June 29, 2019, 6:51pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845 "2019-06-29T18:51:30Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@Nathaniel](https://discourse.julialang.org/u/Nathaniel)
#### Post date: [June 29, 2019, 6:51pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/1 "2019-06-29T18:51:30Z")
I would like to write a Julia program that does the following:
1. Build up a big, complicated Expr object using metaprogramming
2. Compile that expression into a Julia function
3. Call that function repeatedly with different arguments
My main reason for doing things this way is performance, so I really want the compilation step, not just repeated calls to `eval`. It seemed like the following should do this, but it gives an error (below) - what is the correct way to achieve what I want?
Here’s the minimal test code:
```julia
function make_expression()
return :(x+2)
end
function main()
expr = make_expression()
@eval f(x) = $expr
println(f(10))
end
main()
```
and here’s the error:
```julia
$ julia test.jl
ERROR: LoadError: MethodError: no method matching f(::Int64)
The applicable method may be too new: running in world age 25574, while current world is 25575.
Closest candidates are:
f(::Any) at [...]/test.jl:11 (method too new to be called from this world context.)
Stacktrace:
[1] main() at [...]/test.jl:13
[2] top-level scope at none:0
[3] include at ./boot.jl:326 [inlined]
[4] include_relative(::Module, ::String) at ./loading.jl:1038
[5] include(::Module, ::String) at ./sysimg.jl:29
[6] exec_options(::Base.JLOptions) at ./client.jl:267
[7] _start() at ./client.jl:436
in expression starting at [...]/test.jl:17
```
Note that the error goes away if I put the contents of the `main` function in a global scope instead. But I don’t really want to do that.
---
### Author:  [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)
#### Post date: [June 29, 2019, 7:05pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/2 "2019-06-29T19:05:41Z")
Here are some potentially useful links:
> [@World Age Problem Explanation](https://discourse.julialang.org/t/world-age-problem-explanation/9714/4):
>
> The reason for worlds is because of optimizations. If functions can be defined (and redefined) at arbitrary points (and instantly take effect), then that would limit the possible optimizations that can be made. Therefore, new functions are only made visible to the system (become usable) after hitting toplevel. If you try to call a new function before this, then you get a world error. In some cases (like if you are writing a REPL), you do actually want to call your newly defined function without…
> [@"running in world age X, while current world is Y" errors](https://discourse.julialang.org/t/running-in-world-age-x-while-current-world-is-y-errors/5871/2):
>
> Here’s my understanding: In version 0.6, Julia now keeps track of all the callers of a given function (called backedges) so it can go back and invalidate the previous code with any updated methods. This invalidation can only happen at “safe” points within the process — you cannot trash a piece of currently running code and expect everything to go well. You cannot even go back and update a function pointer since the old method be inlined or the new one might change the inferred optimizations. …
---
### Author:  [@Nathaniel](https://discourse.julialang.org/u/Nathaniel)
#### Post date: [June 29, 2019, 7:18pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/3 "2019-06-29T19:18:23Z")
Thank you @kristoffer.carlsson, that’s most helpful indeed. So it seems that the following is a solution:
```julia
function make_expression()
return :(x+2)
end
function main()
expr = make_expression()
@eval f(x) = $expr
println(Base.invokelatest(f,10))
end
main()
```
But it seems also that this is not necessarily the best way to do it, and it might be better to think about whether I can do the same task using macros instead. (It’s not immediately obvious how in my case, but I’ll think about it.)
---
### Author:  [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)
#### Post date: [June 29, 2019, 7:23pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/4 "2019-06-29T19:23:05Z")
> [@Nathaniel](#):
>
> But it seems also that this is not necessarily the best way to do it
From the code you have posted so far the best way to do it would be
```julia
function make_expression()
return :(x+2)
end
expr = make_expression()
@eval f(x) = $expr
function main()
println(f(10))
end
main()
```
Unless you need to JIT a function based on runtime values of your “main” then just do it outside so that the world age can update. If you need to do it dynamically, then perhaps you don’t need full fledged julia functions but if you instead could write a small “interpreter” for the type of functions you need. And if you do need the full julia language, then you are going to need some optimization barrier like `invokelatest´.
`invokelatest` will call your function correctly but you should also know that the function it calls will look like a black box
> [@Nathaniel](#):
>
> I can do the same task using macros instead. (It’s not immediately obvious how in my case, but I’ll think about it.)
A macro is a very different thing from a function. A macro is basically just a tool to allow you to write a piece of syntax to then generate another piece of syntax. In some sense, it is purely a convenience tool and it doesn’t give any extra “powers” since in the end it just gives some syntax that will be evaluated and you could just have written that syntax yourself in the first place.
---
### Author:  [@Nathaniel](https://discourse.julialang.org/u/Nathaniel)
#### Post date: [June 29, 2019, 7:39pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/5 "2019-06-29T19:39:17Z")
> Unless you need to JIT a function based on runtime values of your “main”
I do need to do that. Or at least, it was the plan. (The real `make_expression` function is much more complicated and takes parameters at runtime.)
> A macro is a very different thing from a function. […]
Sure, I understand that. I imagine the idea (referred to in the thread you linked) is to use macros to build up the function `f`, instead of constructing it as an Expr explicitly.
---
### Author:  [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)
#### Post date: [June 29, 2019, 7:58pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/6 "2019-06-29T19:58:31Z")
> [@Nathaniel](#):
>
> is to use macros to build up the function `f` , instead of constructing it as an Expr explicitly.
It seems to me that if you can write a macro to do it then you are not using runtime information of main.
---
### Author:  [@Nathaniel](https://discourse.julialang.org/u/Nathaniel)
#### Post date: [June 29, 2019, 8:23pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/7 "2019-06-29T20:23:11Z")
You’re probably right.
The other thing I got from that thread is that something like the following might actually work fine, and be fast in Julia. I need to think about whether it will work, and profile it, but if it works out it will be much simpler:
```julia
function make_func()
return x -> x+2
end
function main()
f = make_func()
println(f(10))
end
main()
```
My previous implementation is a horrible mess of Python code that generates C++, so I guess I might just have been a bit stuck in the mindset of thinking I need to generate code and then compile it, instead of just constructing a function.
---
### Author:  [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)
#### Post date: [June 29, 2019, 8:29pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/8 "2019-06-29T20:29:16Z")
> [@Nathaniel](#):
>
> so I guess I might just have been a bit stuck in the mindset of thinking I need to generate code and then compile it, instead of just constructing a function.
Perhaps you can give a bit more info about what problem you are actually trying to solve. It feels like the original small example right now is a bit too simple to be representative of what you want to do.
---
### Author:  [@ninjaaron](https://discourse.julialang.org/u/ninjaaron)
#### Post date: [June 30, 2019, 8:22am UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/9 "2019-06-30T08:22:16Z")
That error message about the `world age` and `world context` is crazy.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [June 30, 2019, 12:09pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/10 "2019-06-30T12:09:18Z")
How would you improve it?
---
### Author:  [@ninjaaron](https://discourse.julialang.org/u/ninjaaron)
#### Post date: [June 30, 2019, 3:29pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/11 "2019-06-30T15:29:16Z")
`functions generated at runtime (with `eval`, for example) are not available for use until control flow returns to the module's top-level scope because of the compiler's optimization strategies.`
---
### Author:  [@Nathaniel](https://discourse.julialang.org/u/Nathaniel)
#### Post date: [June 30, 2019, 5:41pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/12 "2019-06-30T17:41:36Z")
Ok, as requested, here’s some more details on what I’m actually trying to do.
Basically I’m implementing a sort of CAD-like system, using implicit functions. This is purely for a hobby project. The user (i.e. me) will write code to specify geometrical objects, along the lines of `intersection(sphere([0,0,0],1),cube([1,1,1]),rounding=0.1)`. This will be used to construct a function that’s positive inside the object and negative outside. Then a mesh will be generated from that function using marching cubes or some other algorithm.
The input to `make_expression` is generated by the user code, and takes the form of a data structure representing the object the user has specified. Its output should be a function of n real numbers, representing spatial dimensions. The function it outputs has to run fast, because it will be called many times in generating the mesh. Actual user code may create many thousands of geometrical objects, and hence generate very complicated functions, and I want that to work as well as possible.
The only slight complication is that it may sometimes be necessary to differentiate the resulting function, either for the purpose of mesh generation or to generate certain features such as offsets or fillets. In my Python/C++ implementation I was doing this symbolically using sympy, but I don’t actually need full symbolic differentiation - it’s enough to be able to evaluate the derivative at a given point - so it might be possible to do it using normal Julia functions and Flux.jl or similar. (Of course I will have to profile that.)
So in my mind there are a few more questions at this point:
1. How good is the compiler at optimising chains of anonymous functions, as opposed to functions generated from expressions in the ‘normal’ way? For example, can I trust it to turn `x->(x->x+2)((x->x+2)(x))` into `x->x+4`? If so, I might not need to generate expressions at all.
2. Is it feasible to use Flux.jl (or some other existing automatic differentiation package) on huge chains of anonymous functions like this?
3. If no to either of the above, then I’m probably better off generating Expr objects as I initially planned. In that case it seems I’m advised to have the main part of my code run in a global scope (!), but I’m having trouble seeing how I can arrange for that to happen _after_ the user runs their code. As far as I know, code in a global scope normally runs as soon as the module is imported, but in this case it needs to happen at a later point, in response to something invoked by the user. Is there some way to achieve that, and what would be the sensible way to structure the project, if that’s the way it has to work?
---
### Author:  [@oxinabox](https://discourse.julialang.org/u/oxinabox)
#### Post date: [June 30, 2019, 6:45pm UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/13 "2019-06-30T18:45:37Z")
> [@Nathaniel](#):
>
> How good is the compiler at optimising chains of anonymous functions, as opposed to functions generated from expressions in the ‘normal’ way? For example, can I trust it to turn `x->(x->x+2)((x->x+2)(x))` into `x->x+4` ? If so, I might not need to generate expressions at all.
Anonymous functions optimize just fine
```julia
julia> function makefun()
f1 = x->x+2
f2 = x->(f1(f1(x)))
return f2
end
makefun (generic function with 1 methods)
julia> aa = makefun();
julia> @code_typed aa(3)
CodeInfo(
│╻╷ #74 1 ─ %1 = (Base.add_int)(x, 2)::Int64
││╻ + │ %2 = (Base.add_int)(%1, 2)::Int64
│ └── return %2
) => Int64
julia> bb(x) = x+2+2;
julia> @code_typed bb(3)
CodeInfo(
│╻╷ +1 1 ─ %1 = (Base.add_int)(x, 2)::Int64
││┃ + │ %2 = (Base.add_int)(%1, 2)::Int64
│ └── return %2
) => Int64
```
Noted that `aa` and `bb` get lowered to exactly the same thing.
* * *
> Is it feasible to use Flux.jl (or some other existing automatic differentiation package) on huge chains of anonymous functions like this?
Yes.
(well flux isn’t AD, but Flux will invoke Tracker.jl (or oneday Zygote.jl) and both have no issue with this.)
---
### Author:  [@rdeits](https://discourse.julialang.org/u/rdeits)
#### Post date: [July 1, 2019, 3:28am UTC](https://discourse.julialang.org/t/what-is-the-right-way-to-construct-and-then-run-a-julia-function-programmatically/25845/14 "2019-07-01T03:28:54Z")
Flux will probably work, but for functions with very few inputs [ForwardDiff.jl](https://github.com/JuliaDiff/ForwardDiff.jl) is likely to be easier to set up and potentially quite efficient.