Help calling a function defined from expressions

I was using this basic pattern to define a function but it now fails when called a certain way on v0.6-dev

function test(vars)
    body = :x
    eval(Expr(:function,
              Expr(:call, gensym(), map(Symbol,vars)...),
              body))
end

fn = test(["x"]); fn(1)   # works
test(["x"])(1)               # fails 

The error is

ERROR: MethodError: no method matching ##283(::Int64)
The applicable method may be too new: running in world age 20537, while current world is 20538.
Closest candidates are:
  ##283(::Any) at :0 (method too new to be called from this world context.)

Is there something I should be doing differently now?

1 Like

The only way would be to call eval. Is there some reason you need to do this, rather than defining a higher-order function in the usual way by x -> ...?

It comes up in SymPy in a “lambdify” function that walks an expression tree and creates a function from a symbolic expression, so I’m not sure I have a simple alternative, if I want to keep this functionality. I’m sure there is another workaround when I use this, but I don’t understand what makes it not work now where it had before.

1 Like

This is similar to the issue discussed in callback functions see "old world" in 0.6 · Issue #19774 · JuliaLang/julia · GitHub, and has to do with the resolution to #265.

In Julia 0.6, when you change the definition of a function foo, any functions that call foo are recompiled the next time they are called. Technically, this involves “timestamping” each method with a “world age” that says how old it is, which is used to determine whether it needs to be recompiled. Moreover, when you are executing a function, it only calls other functions from the “world” as it existed when the function was compiled.

This affects you if you call eval during execution of a function foo. Any new methods that are defined by the eval are in a newer world, and hence are not the functions that are called within the same call to foo. (The next call to foo will recompile it and see the newer world.) To see why that is, consider:

bar() = 1
function foo()
    eval(current_module(), :(bar() = "hello"))
    return bar()
end

What will foo() return? The first time you call it, it will return 1, because it will use the old version of bar() that was defined when foo() was first compiled. The second time you call it, it will return "hello", because it will be automatically recompiled (since it depends on bar and bar has changed). In fact, the function bar() will be inlined when foo() is compiled, so there is no way for foo() to get the “updated” version of bar() without recompiling foo(), which can’t happen while foo() is running — it has to wait until the next call.

Now, suppose you don’t like this behavior: you want foo() to return "hello" the first time you call it, i.e. you want it to always call the latest version of bar(). To achieve that, not only must the compiler not inline the function bar(), it can’t infer the return type of bar() either, because you might redefine bar() to return a different type (as we did above) — the function call would have to be completely dynamic. If the compiler did that for every function you call in any function, it would completely kill performance. By default, therefore, it has to call functions as they were defined when foo() was compiled. However, you can invoke the latest version “manually” by calling eval:

bar() = 1
function foo()
    eval(current_module(), :(bar() = "hello"))
    return eval(current_module(), :(bar()))
end

at the price of performance (no inlining or type inference). (Calling eval is awkward enough, in the rare cases where this is needed, that I’d prefer to have an invokelatest intrinsic function for this: add invokelatest to circumvent world-age problems by stevengj · Pull Request #19784 · JuliaLang/julia · GitHub)

A subtle design choice arises for Function objects that are not known at compile time, such as a new function that you construct via eval (as in your example) or a Function extracted from a pointer in a callback routine (as in the issue I linked). In this case, the compiler won’t be able to inline it or (probably) do type-inference on the result, so it might as well invoke the latest-world version of the function, no? On the other hand, this makes the semantics dependent on type inference (if it can infer the specific type of function, it invokes the old-world version, but if it can only infer Function, then it invokes the new-world version), and it is hard to define precisely when inference will succeed.

My preference would still be to call the latest-world version of a function when its type is inferred as Function, since I think this is almost always what you want (especially for anonymous functions that aren’t even defined in the old world), and world-age errors are awfully confusing, but reasonable people can disagree about this.

16 Likes

Thanks so much for the generous and thorough explanation!

@vtjnash says that this should be improved before 0.6 is released, though I don’t know exactly what semantics he is planning: fix REPL callbacks to use eval so newest world function is executed by KristofferC · Pull Request #19924 · JuliaLang/julia · GitHub

1 Like

Sorry, that I come back to this after a year. I came across the same situation like @j_verzani repeatedly and was hoping for an improvement.

I extremely support this preference.

Was there a progress or a concluding discussion about this promise in the meanwhile?
I came along the same trouble in several cases, when I created a new function by eval and could not call it directly, and had to use Base.invokelatest.
That works fine, but it does not look clean - and it counters my intuition.

my boiled down example:

Version 0.7.0-DEV.3216 (2017-12-30 09:42 UTC)
 
julia> print1(x::String) = print2(format(x))
print1 (generic function with 1 method)

julia> print2(fmt::Function) = fmt() # does not work - use Base.invokelatest(fmt)
print2 (generic function with 1 method)

julia> format(x::String) = eval( :(() -> $x))
format (generic function with 1 method)

julia> print1("abc")
ERROR: MethodError: no method matching (::getfield(, Symbol("##5#6")))()
The applicable method may be too new: running in world age 25100, while current world is 25101.
Closest candidates are:
  #5() at REPL[7]:1 (method too new to be called from this world context.)
Stacktrace:
 [1] print2(::getfield(, Symbol("##5#6"))) at ./REPL[6]:1
 [2] print1(::String) at ./REPL[5]:1
 [3] top-level scope

julia> print2(format("abc"))
"abc"

julia> print1("abc")
ERROR: MethodError: no method matching (::getfield(, Symbol("##11#12")))()
The applicable method may be too new: running in world age 25103, while current world is 25104.
Closest candidates are:
  #11() at REPL[7]:1 (method too new to be called from this world context.)
Stacktrace:
 [1] print2(::getfield(, Symbol("##11#12"))) at ./REPL[6]:1
 [2] print1(::String) at ./REPL[5]:1
 [3] top-level scope

julia> 

2 Likes

I’ve struggled a lot with this as well. But from the sounds of it this is here to stay.