# Generate functions inside a function from string

**URL:** <https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331>\
**Category:** Performance\
**Tags:** question\
**Created:** [July 12, 2018, 1:15pm UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331 "2018-07-12T13:15:13Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![wuhuagumu](https://avatars.discourse-cdn.com/v4/letter/w/7ab992/32.png) [@wuhuagumu](https://discourse.julialang.org/u/wuhuagumu)\
**Post date:** [July 12, 2018, 1:15pm UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/1 "2018-07-12T13:15:13Z")

</div>

Hi,

I noticed that the function generated by `eval` is always a [global function](https://docs.julialang.org/en/stable/manual/methods/#Redefining-Methods-1), making the code silly if I would like to generate some function defined in string. Are there any other ways to do something like this while avoid using `eval`? Examples are:

```
function foo()
    return eval(parse("x->x"))
end
foo()(1)  
# ERROR: MethodError: no method matching (::##9#10)(::Int64)
# The applicable method may be too new: 
# running in world age 21870, while current world is 21871.

function foo()
    return x->x
end
foo()(1) # works fine

```

By the way I noticed there is a [question](https://discourse.julialang.org/t/creation-of-function-using-eval/4901) mentioned this problem but no useful solution there.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [July 12, 2018, 1:24pm UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/2 "2018-07-12T13:24:23Z")

</div>

Related discussion: [World age error when parsing function from String](https://discourse.julialang.org/t/world-age-error-when-parsing-function-from-string/11277)

---

<div class="post-metadata">

**Author:** ![wuhuagumu](https://avatars.discourse-cdn.com/v4/letter/w/7ab992/32.png) [@wuhuagumu](https://discourse.julialang.org/u/wuhuagumu)\
**Post date:** [July 12, 2018, 1:25pm UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/3 "2018-07-12T13:25:12Z")

</div>

Do you have any workarounds？

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [July 12, 2018, 1:26pm UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/4 "2018-07-12T13:26:22Z")

</div>

No more than the two I described in the linked discussion.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [July 12, 2018, 1:56pm UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/5 "2018-07-12T13:56:46Z")

</div>

Use `eval` to define `foo` not in the body of `foo`:

```julia
julia> @eval foo() = $(Meta.parse("x->x"))
foo (generic function with 1 method)

julia> foo()
#5 (generic function with 1 method)

julia> foo()(1)
1

```

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [July 12, 2018, 5:20pm UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/6 "2018-07-12T17:20:15Z")

</div>

Are you entirely certain you actually _need_ to be defining functions by parsing strings? This can certainly be done, but it’s a design that can lead to fragile code, security vulnerabilities, and/or performance issues. Can you describe the actual problem you’re trying to solve?

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [July 12, 2018, 6:28pm UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/7 "2018-07-12T18:28:22Z")

</div>

FWIW, there’s a hack that allows you to define functions from expressions at runtime, without world age issues:

```julia
julia> funs = []
0-element Array{Any,1}

julia> @generated make_fun(::Val{N}, x) where N = funs[N]
make_fun (generic function with 3 methods)

julia> evil(code) = (push!(funs, code); x->make_fun(Val{length(funs)}(), x))
evil (generic function with 1 method)

julia> evil(:(x+2))(10)
12

```

`invokelatest` is way cleaner though. And unless you’re doing stuff like evolutionary programming, you should consider using macros instead of parsing strings.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [July 12, 2018, 7:11pm UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/8 "2018-07-12T19:11:05Z")

</div>

`invokelatest` and other hacks are only necessary if you’re calling code from something that’s long-running and can’t get recompiled and called afresh later when you (re)define `foo`. For example, the REPL itself. If you define `foo()` using eval and then call `foo()` later it will work fine without any additional tricks just as I demonstrated above. For example:

```julia
function f1(code::String)
    @eval g1() = $(Meta.parse(code))
    g1() # calls the old definition of `g1` if any
end

julia> f1("1 + 2")
ERROR: MethodError: no method matching g1()
The applicable method may be too new: running in world age 27552, while current world is 27553.
Closest candidates are:
  g1() at REPL[1]:2 (method too new to be called from this world context.)
Stacktrace:
 [1] f1(::String) at ./REPL[1]:3
 [2] top-level scope at none:0

julia> f1("'x'")
3

julia> f1("3/5")
'x': ASCII/Unicode U+0078 (category Ll: Letter, lowercase)

```

versus

```julia
function f2(code::String)
    @eval g2() = $(Meta.parse(code))
    Base.invokelatest(g2) # calls the new definition of `g2`
end

julia> f2("1 + 2")
3

julia> f2("'x'")
'x': ASCII/Unicode U+0078 (category Ll: Letter, lowercase)

julia> f2("3/5")
0.6

```

If you’re defining `foo` at the top of a script and then calling it later, you don’t need to do anything special.

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [July 13, 2018, 2:36am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/9 "2018-07-13T02:36:24Z")

</div>

My solution to this problem is implemented in the [SyntaxTree](https://github.com/chakravala/SyntaxTree.jl) package (on the most recent master branch)

```nohighlight
julia> using SyntaxTree

julia> SyntaxTree.genfun(:(x^2+1),[:x])
(::#1) (generic function with 1 method)

julia> ans(2)
5

```

This automatically handles the `Base.invokelatest` action for you, so all you need to do is specify an expression and a list or function arguments.

---

<div class="post-metadata">

**Author:** ![wuhuagumu](https://avatars.discourse-cdn.com/v4/letter/w/7ab992/32.png) [@wuhuagumu](https://discourse.julialang.org/u/wuhuagumu)\
**Post date:** [July 13, 2018, 2:38am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/10 "2018-07-13T02:38:38Z")

</div>

Well, I am working with `SymPy`. I would like to build a symbolic matrix and then numerically calculate it. However I found `SymPy` cannot do the calculation fast enough, e.g.

```
using SymPy

@vars x

expr=integrate(sin(x))
f=eval(parse("x->$(repr(expr))"))
g=lambdify(expr,[x])

@time subs(expr,x=>1.0)
@time f(1.0)
@time g(1.0)

# timing:
# 0.000365 seconds (80 allocations: 2.453 KiB)
# 0.000003 seconds (5 allocations: 176 bytes)
# 0.000016 seconds (9 allocations: 240 bytes)

```

I found `eval` is the fastest. As I have to diagonalize a rather large matrix many times (e.g. ten thousand of 400\*400 matrices), I want to make the performance better.

Besides, I thought building something from string should not be a very uncommon thing…

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [July 13, 2018, 2:42am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/11 "2018-07-13T02:42:11Z")

</div>

Have you tried using

> **[GitHub - chakravala/Reduce.jl: Symbolic parser for Julia language term...](https://github.com/chakravala/Reduce.jl)**
>
> Symbolic parser for Julia language term rewriting using REDUCE algebra - GitHub - chakravala/Reduce.jl: Symbolic parser for Julia language term rewriting using REDUCE algebra

for symbolic calculation?

```plaintext
using SyntaxTree, Reduce

expr = Algebra.int(:(sin(x)),:x)
f = SyntaxTree.genfun(repr(expr),[:x])

```

Don’t know that `repr` is though

---

<div class="post-metadata">

**Author:** ![Chong\_Wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chong_wang/32/20307_2.png) [@Chong\_Wang](https://discourse.julialang.org/u/Chong_Wang)\
**Post date:** [July 13, 2018, 2:48am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/12 "2018-07-13T02:48:30Z")

</div>

Now that you brought it up, I have been looking at Reduce.jl but don’t quite get it.

How’s Reduce.jl compared to Sympy and Maxima, in the sense of speed and features?

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [July 13, 2018, 2:58am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/13 "2018-07-13T02:58:02Z")

</div>

Reduce.jl was initially forked from Maxima.jl and modified for Reduce CAS, but since then it has many more generalized features than Maxima.jl currently has due to the parser generator. The premise behind Reduce is different than that of SymPy. In SymPy, new symbol objects are defined, in Reduce the native Julia symbol and expression type is used and translated into Reduce commands.

I have not done a benchmark to compare the speed with SymPy, but it should be fairly quick (although not as quick as SymEngine yet). There are multiple optimizations and rewrites planned to increase the performance in various aspects of the package, however, it will require some tweaking and fine tuning.

My recommendation is to try it out and let me know your own thoughts about it.

---

<div class="post-metadata">

**Author:** ![Chong\_Wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chong_wang/32/20307_2.png) [@Chong\_Wang](https://discourse.julialang.org/u/Chong_Wang)\
**Post date:** [July 13, 2018, 2:58am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/14 "2018-07-13T02:58:35Z")

</div>

Also, it seems your code only works for me without the `repr`, and the function produced `f` is type unstable?

```julia
julia> using SyntaxTree, Reduce

julia> expr = Algebra.int(:(sin(x)),:x)
:(-(cos(x)))

julia> f = SyntaxTree.genfun(expr,[:x])
(::#1) (generic function with 1 method)

julia> @code_warntype(f(1.0))
Variables:
  #self#::SyntaxTree.##1#2{Symbol}
  x::Float64

Body:
  begin
      SSAValue(0) = $(Expr(:foreigncall, :(:jl_toplevel_eval_in), Any, svec(Any, Any), :(SyntaxTree.SyntaxTree), 0, :((Core.getfield)(#
self#, :gs)::Symbol), 0))
      return (Core._apply_latest)(SSAValue(0), (Core.tuple)(x::Float64)::Tuple{Float64})::Any
  end::Any

```

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [July 13, 2018, 3:03am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/15 "2018-07-13T03:03:49Z")

</div>

> [@Chong\_Wang](#):
>
> Also, it seems your code only works for me without the `repr` , and the function produced `f` is type unstable?

As I said in my post, I don’t have any information about the original author’s `repr` method.

Yes, you’re right about the type instability, might have to tweak the implementation, it’s a concept design.

---

<div class="post-metadata">

**Author:** ![wuhuagumu](https://avatars.discourse-cdn.com/v4/letter/w/7ab992/32.png) [@wuhuagumu](https://discourse.julialang.org/u/wuhuagumu)\
**Post date:** [July 13, 2018, 3:07am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/16 "2018-07-13T03:07:55Z")

</div>

Well, I only used `repr` to generate strings, if you already have an expression, I don’t think you have to use it. In my code, `parsing` has changed the string to expressions.

---

<div class="post-metadata">

**Author:** ![Chong\_Wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chong_wang/32/20307_2.png) [@Chong\_Wang](https://discourse.julialang.org/u/Chong_Wang)\
**Post date:** [July 13, 2018, 3:08am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/17 "2018-07-13T03:08:52Z")

</div>

Thank you.I’ll try the package next time I do symbolic calculation.

---

<div class="post-metadata">

**Author:** ![wuhuagumu](https://avatars.discourse-cdn.com/v4/letter/w/7ab992/32.png) [@wuhuagumu](https://discourse.julialang.org/u/wuhuagumu)\
**Post date:** [July 13, 2018, 3:21am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/18 "2018-07-13T03:21:24Z")

</div>

This is really good, 1 allocation more and almost comparable time.

```
using SymPy
using SyntaxTree

@vars x

expr=integrate(sin(x))
f=eval(parse("x->$(repr(expr))"))
h=SyntaxTree.genfun(parse("$(repr(expr))"),[:x])

@time f(1.0)
@time h(1.0)
# 0.000003 seconds (5 allocations: 176 bytes)
# 0.000005 seconds (6 allocations: 192 bytes)

```

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [July 13, 2018, 3:25am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/19 "2018-07-13T03:25:15Z")

</div>

> [@wuhuagumu](#):
>
> h=SyntaxTree.genfun(parse(“x-\>$(repr(expr))”),[:x])

Note that the input to `SyntaxTree.genfun` doesn’t need to be a function, it should only be the expression of the function that is added, so you might want to use `parse("$(repr(expr))")` instead, since it automatically turns it into the function using the arguments from the list.

---

<div class="post-metadata">

**Author:** ![wuhuagumu](https://avatars.discourse-cdn.com/v4/letter/w/7ab992/32.png) [@wuhuagumu](https://discourse.julialang.org/u/wuhuagumu)\
**Post date:** [July 13, 2018, 3:28am UTC](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331/20 "2018-07-13T03:28:49Z")

</div>

Any consequences for `invokelatest`?

[Next page](https://discourse.julialang.org/t/generate-functions-inside-a-function-from-string/12331.md?page=2)