Interruptible routine: the routine does not receive SIGINT if there is no gap in the loop

Bossax · December 8, 2021, 4:56am

Hi all,

I am learning to create an Interruptible routine for my camera sensor. My goal is to run an image acquisition routine on a remote process so that the local process can receive command from the user to perform other tasks. I start with this simple example. The routine creates a file, opens it and writes something into it at a certain period as shown below (Please note that the code in the while loop can be anything, I picked writing to a file as an example)

@everywhere function worker_func(fname::String, path::String)
    Base.exit_on_sigint(false)
    cd(path)
    touch(fname)
    f = open(fname,"w")
    counter = 1

    try
      while true
       if  counter%1e8 == 0
          # @info "."
          # println(".")
          #sleep(0.1)
          write(f,"markerd\n")
        end
        counter += 1
      end
    catch e
      if e isa InterruptException
         # cleanup
         println("function terminated by user")
         write(f,"....... file is closed here ........")
         close(f)
      else
        rethrow(e)
      end
    end

end

As seen in the code block, I comment out the stdout print functions (@info and println) and a sleep function to see its behavior first. I simply call the function in the local process - worker_func("output.txt", pwd()). Here are the results

When the @info, println, and sleep are commented out, the output file has nothing in it. The keyboard interrupt signal is not received. The only way to stop the routine is send multiple SIGINT until the process throws WARNING: Force throwing a SIGINT and quits.
When I uncomment either @info,println, or sleep, keyboard interrupt is easily received by the routine and it stops nicely. The output file also has texts in it with the expected manner.

Another experiment is I run this routine on a remote process. I call addprocs(3) and then run the routine

remote_do(worker_func,2,fname,path)

I leave it running for a couple of tens of seconds before interrupting it by

interrupt(2)

And this is printed out

Worker 2 terminated.

However, the output file is empty.

Can anyone explain these behaviors? and suggest the best practice to interrupt a remote process?

Best,

Sitthichat

gustaphe · December 8, 2021, 6:54am

I’m not sure what the problem is, but I see a couple of code smells.

counter%1e8 isa Float64, and should be compared with ≈
Is your loop just supposed to introduce a timing? If so, sleep is the idiomatic way to do that. I’m not sure but I think the compiler could potentially just remove most of your loop.

Bossax · December 8, 2021, 7:15am

Hi,

I first tried this in a Windows machine. Now I have tested it on a Linux machine. On a Linux machine, running this on a remote process seems to work fine (with either of @info , println , or sleep uncommented). The empty file resulted in the Window machine could be due to the program failing at the first few lines (changing directory and creating the file); I am sure these commands work fine in Linux.

Now, the point is, the routine still does not receive SIGINT either from keyboard input or interrupt() function if there is no gap in the while loop; eg. the routine stops to print out a message or sleeps for a brief amount of time.

@gustaphe The while loop encapsulates any routine that runs continuously (in my case it would be an image acquisition). Let me put it this way

@everywhere function routine()
    try
        while true
           # do something until it gets interrupted
        end
    catch e
        if e isa InterruptException
            # finalize stuff in the while loop above
            return nothing
        else
            rethrow(e)
            return nothing
        end
    end

end

This would be the template of the interruptible routine that I use.

However, the routine seems not able to be interrupted if there is no gap in the while loop

while true
           # do something until it gets interrupted
           # some function that makes the loop pause for a brief moment
        end

For example, the while loop can just increments a variable

while true
            counter += 1
end

But doing this, the rountine cannot be interrupted

However, if I just add a sleep function in the while loop

while true
            sleep(0.0001)
            counter += 1
end

The routine terminates as expected.

I could just work around it by adding a sleep function into my routine. However, the image acquisition routine has to achieve a quite high frame rate so adding sleep would ruin the goal of the routine.

I renamed the topic so that it reflects the real issue better. I am sorry for being unclear in the first place.

Cheers,

gustaphe · December 8, 2021, 12:18pm

Do you know for sure this happens in the real case? The problem (I think) is that the compiler can optimize away the entire loop, if it knows nothing will change:

julia> function f()
       i = 0
       while true
       i = i+1
       if i==100
       return i
       end
       end
       end
f (generic function with 1 method)

julia> @code_native(f())
	.text
	.file	"f"
	.globl	julia_f_335                     # -- Begin function julia_f_335
	.p2align	4, 0x90
	.type	julia_f_335,@function
julia_f_335:                            # @julia_f_335
; ┌ @ REPL[6]:1 within `f`
	.cfi_startproc
# %bb.0:                                # %top
; │ @ REPL[6]:6 within `f`
	movl	$100, %eax
	retq
.Lfunc_end0:
	.size	julia_f_335, .Lfunc_end0-julia_f_335
	.cfi_endproc
; └
                                        # -- End function
	.section	".note.GNU-stack","",@progbits
julia> g() = 100
g (generic function with 2 methods)

julia> @code_native(g())
	.text
	.file	"g"
	.globl	julia_g_344                     # -- Begin function julia_g_344
	.p2align	4, 0x90
	.type	julia_g_344,@function
julia_g_344:                            # @julia_g_344
; ┌ @ REPL[11]:1 within `g`
	.cfi_startproc
# %bb.0:                                # %top
	movl	$100, %eax
	retq
.Lfunc_end0:
	.size	julia_g_344, .Lfunc_end0-julia_g_344
	.cfi_endproc
; └
                                        # -- End function
	.section	".note.GNU-stack","",@progbits

I’m no expert, but this looks like these two functions are identical, so no loop is running, so no loop can be interrupted. Adding a non-pure routine to the loop forces the compiler to not optimize it away.

gbaraldi · December 8, 2021, 1:24pm

Another thing. If your real loop isn’t compiled away. adding yield() might help it be interruptible by other tasks.

lawless-m · December 8, 2021, 1:30pm

Bear in mind, Julia 1.7.0 is having problems with Signal handling

https://github.com/JuliaLang/julia/issues/43103