How to interrupt fun(a,b) in a for loop after 10 seconds?

General Usage
21

The problem of terminating an async task has been discussed over here:

The conclusion seems to have been

task = @async some_long_running_function()
sleep(5) # give it some time to run
schedule(task, InterruptException(), error=true) # send an interrupt exception
22

gives

ERROR: MethodError: objects of type Bool are not callable

Modifying it to:


function limitowany_czas(url, tagg, limit)
       my_task = @async get_links_by_tagg(url, tagg)
       my_istaskdone() = istaskdone(my_task)
       my_result = if timedwait(my_istaskdone, limit) == :ok
           fetch(my_task)
       else
           # some `yieldto` code if you do not want the computation to finnish in the background
           (false, false, false)
       end
       x, y, z = my_result
end

Thanks for bringing timedwait up into this discussion.

2 Likes
23

Ah, good catch ( I did say I’d not executed the code myself :wink: ).

1 Like
24

Thanks, Nice, it works in simple situations, I’ll do some heavy tests!

1 Like
25 
function limitowany_czas(url, tagg, limit)

This does not always happen, but when there is a heavy load of many JULIA sessions in separate windows, this error appears in one of the windows and interrupts all sessions

6 onet.pl
Utworzono katalog: dane_txt_0/2024-04-15\17-41\onet.pl
Limitowany_czas
100.497638 seconds (503.98 k allocations: 44.503 MiB 0.02z gc time, 4.17z compilation time)
 0.009291 seconds (20.98 k allocations: 594.773 KiB5
103.256181 seconds (585.66 k allocations: 51.940 MiB, 0.02z gc time, 4.57z compilation time)

7 opoka.org.pl
Utworzono katalog: dane_txt_0/2024-04-15\17-41\opoka.org.pl
Limitowany_czas
263.548309 seconds (683.90 k allocations: 132.710 MiB 0.13z gc time, 1.74z compilation time)
 0.669989 seconds (44.08 k allocations: 1.856 MiB, 7.65z compilation time)
265.182190 seconds (779.61 k allocations: 138.934 MiB, 0.13z gc time, 2.01Z compilation time)
....
8 parenting.pl
Utworzono katalog: dane_txt_0/2024-04-15\17-41\parenting.pl
Limitowany czas
ERROR: LoadError: OutOfMemoryError()
Stacktrace:
    timedwait(testcb::var"#my_istaskdone#3"flask), timeout::Int64; pollint::Float64)
  @ Base .\asyncevent.j1:344
 [2] timedwait
  @ .\as/ncevent.j1:336 Einlined]
 [3] limitowany_czas(ur1::String, tagg::String, limit::Int64) 
        D:\pawel\rynek\kontener\Kitomierz\funkcje\limitowany_czas.py:4
 EC macro expansion
  @ .\timing.j1:279 Einlined]
 [5] top-level scope
  @ D:\pawel\rynek\kontener\Kitomierz\kitomierz_zbieram_strony_0.py:14E
in expression starting at D:\pawel\rynek\kontener\Kitomierz\kitomierz_zbieram_strony_0.py:132

my fun:

function limitowany_czas(url, tagg, limit)
       my_task = @async get_links_by_tagg(url, tagg)
       my_istaskdone() = istaskdone(my_task)
       my_result = if timedwait(my_istaskdone, limit) == :ok
           fetch(my_task)
       else
           # some `yieldto` code if you do not want the computation to finnish in the background
           (false, false, false)
       end
       # x, y, z = my_result
end

Some ideas?
Thx Paul

26

You can always buy more RAM.

Other than that, I do not yet believe that you are successfully interrupting the task if it takes to long, you are simply returning something else, and the computation keeps going in the background. Do that enough times (and in multiple julia sessions?? Why though?) and you are surely going to run out of system resources…

1 Like
27

You have already figured out how to return either the successful result, or a “this timed out” default after 10 seconds.

So now the remaining question is: After the 10 seconds, how can you kill the now orphaned task so that it doesn’t consume system resources?

Julia only supports cooperative multithreading.

You cannot interrupt the task. This is not supported by the julia runtime. Interrupting (preempting) a thread is supported by hardware, but the julia runtime cannot recover from arbitrary points.

So instead, your task has to regularly poll something to figure out whether it is supposed to give up.

If the code is executing something you wrote, yield is a thing, If you are in a tight loop, you can check an atomic boolean. You can also poll the current time and check whether enough time has passed.

If the code is doing IO / waiting for something, then schedule(task, InterruptException(), error=true) does the job – once the task reaches a yield point, either from yield() or waiting on a lock or IO, it will receive the exception (and if it is currently waiting on something, then it gets immediately woken up with the exception).

If the code is executing a tight loop that is outside of your control, without any yield points – possibly in some C library – then you are out of luck.

The appropriate approach is then to create a new process, and kill it on timeout. If the child process spawns further processes, you might need a cgroup on linux to kill it for good, no idea about windows.

This is not a particularly unique limitation of the julia runtime. Cf eg https://docs.oracle.com/javase/8/docs/technotes/guides/concurrency/threadPrimitiveDeprecation.html. pthreads et al don’t safely support that either.

3 Likes
28

Like the Java link you posted talked about, this may cause unpredictable concurrency bugs that are nearly impossible to diagnose, and thus is strongly discouraged. It also may break in the future when we implement detection of this sort of concurrency mistake.

1 Like