I often find myself in a situation where I’m calling a Julia function many times in a for loop, and I would like to run the function for, say, 10 seconds, and if it hasn’t finished executing in that time, it should return something trivial (e.g. NaN) and skip to the next iteration.
For example, when solving large systems of non-linear equations or ODEs, over a broad range of parameters, sometimes you hit on a specific set of parameters which cause the solvers to diverge or take a small eternity to converge. When trying to get a quick-and-dirty visualization of how the system behaves, it is extremely annoying when one call (out of maybe the 20 required to generate a plot) takes 15 minutes to execute, when the rest take 1 second.
Is there a way in Julia of forcing a blackbox function to timeout if it hasn’t executed within a fixed period of time?
After digging around in various forums and trying various approaches, I’m very happy to share what appears to be a relatively general-purpose solution. The solution was provided by hhaensel here:
They define a macro as follows,
macro timeout(seconds, expr, fail)
quote
tsk = @task $expr
schedule(tsk)
Timer($seconds) do timer
istaskdone(tsk) || Base.throwto(tsk, InterruptException())
end
try
fetch(tsk)
catch _
$fail
end
end
end
They give the trivial application, which assigns x = 1 if the expression within begin/end takes less than 1 second, else it assigns x = “failed”.
x = @timeout 1 begin
sleep(1.1)
println("done")
1
end "failed"
Playing around with this macro, it appears to work quite well when calling more complex libraries. For example, here are applications to NLsolve.jl and DifferentialEquations.jl:
using NLsolve
function f!(F, x)
sleep(0.1)
F[1] = (x[1]+3)*(x[2]^3-7)+18
F[2] = sin(x[2]*exp(x[1])-1)
end
function j!(J, x)
J[1, 1] = x[2]^3-7
J[1, 2] = 3*x[2]^2*(x[1]+3)
u = exp(x[1])*cos(x[2]*exp(x[1])-1)
J[2, 1] = x[2]*u
J[2, 2] = u
end
MaxTime = 10
res1 = @timeout MaxTime begin
nlsolve(f!, j!, [ 0.1; 1.2]).zero
end NaN
println("Given MaxTime = 10 seconds, we get res1 = ", res1)
MaxTime = 0.1
res2 = @timeout MaxTime begin
nlsolve(f!, j!, [ 0.1; 1.2]).zero
end NaN
println("Given MaxTime = 0.1 seconds, we get res2 = ", res2)
On my computer, this returns the correct solution when MaxTime = 10s, but returns NaN when MaxTime = 0.1s. Similarly for DifferentialEquations.jl we have,
using DifferentialEquations
function f(u, p, t)
sleep(0.001)
return 1.01 * u
end
u0 = 1 / 2
tspan = (0.0, 1.0)
prob = ODEProblem(f, u0, tspan)
@time sol = solve(prob, Tsit5(), reltol = 1e-8, abstol = 1e-8)
MaxTime = 100
sol = @timeout MaxTime begin
sol = solve(prob, Tsit5(), reltol = 1e-8, abstol = 1e-8)
end NaN
println("Given 100 seconds, sol converges, so sol(0.1) = ", sol(0.1))
MaxTime = 0.01
sol = @timeout MaxTime begin
sol = solve(prob, Tsit5(), reltol = 1e-8, abstol = 1e-8)
end NaN
println("Given 0.01 seconds, sol doesn't converge, and sol = ", sol)
There are a few challenges. For example, BlackBoxOptim automatically recovers and returns a partial solution in case of an InterruptException, which is what the macro throws. You need to turn this off explicitly to make the macro work as intended:
using BlackBoxOptim
function rosenbrock2d(x)
sleep(0.0001)
return (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2
end
MaxTime = 1
x = @timeout MaxTime begin
sleep(0.9)
res = bboptimize(rosenbrock2d; SearchRange = (-5.0, 5.0), NumDimensions = 2,
TraceMode = :silent, RecoverResults=false)
res
end NaN
println("Given MaxTime = 1 second, we get x = ", x)
MaxTime = 30
x = @timeout MaxTime begin
sleep(0.9)
res = bboptimize(rosenbrock2d; SearchRange = (-5.0, 5.0), NumDimensions = 2,
TraceMode = :silent, RecoverResults=false)
res
end NaN
println("Given MaxTime = 30 seconds, we converge, with get best_fitness(x) = ", best_fitness(x))
I don’t know enough about the @task macro to determine if/when this might fail to work as intended, but for now it certainly seems a useful macro.
I also want to mention another solution from later in the same thread. This defines a timeout function, which wraps an existing function and gives a different return value upon failure,
function timeout(f, arg, seconds, fail)
tsk = @task f(arg)
schedule(tsk)
Timer(seconds) do timer
istaskdone(tsk) || Base.throwto(tsk, InterruptException())
end
try
fetch(tsk)
catch _;
fail
end
end
This is more flexible when you want a function to have a time limit. For example, we can define a ‘time capped’ version of a function which returns a different value if the functio ntakes too long:
function rosenbrock2d(x)
sleep(rand()*0.1)
return (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2
end
function rosenbrock2dtimelimit(x)
return timeout(rosenbrock2d, x, 0.05, 1e99)
end
#We get roughly a 50/50 split between real values and timed out values.
for i = 1:10
println(rosenbrock2dtimelimit([1,2]))
end
timeout() is a useful useful wrapper to objective functions in BlackBoxOptim or NLsolve.
I have not been able to interrupt it with any timeouts. Is it because the above expression calls an external library that refuses to yield? I need to be able to interrupt this kind of computations, what do I do?
Minor tweak - I think the expressions and seconds should be escaped so it works also within modules. Updated code below:
"""
@timeout(seconds, expr_to_run, expr_when_fails)
Simple macro to run an expression with a timeout of `seconds`. If the `expr_to_run` fails to finish in `seconds` seconds, `expr_when_fails` is returned.
# Example
```julia
x = @timeout 1 begin
sleep(1.1)
println("done")
1
end "failed"
```
"""
macro timeout(seconds, expr_to_run, expr_when_fails)
quote
tsk = @task $(esc(expr_to_run))
schedule(tsk)
Timer($(esc(seconds))) do timer
istaskdone(tsk) || Base.throwto(tsk, InterruptException())
end
try
fetch(tsk)
catch _
$(esc(expr_when_fails))
end
end
end