Handling SIGTERM in Julia

josuagrw · December 13, 2019, 2:06pm

Hi, from researching a bit I learned that handling unix signals has been tricky in Julia for a while. Our queuing systems send signals to programs to indicate they should come to an end; and I’d like to do stuff like storing all data in response to that. Did I miss something or is that still impossible in Julia code?

(I could probably achieve the same results by creating and deleting files, and checking their existence periodically - if anyone has code for that please let me know.)

stevengj · December 13, 2019, 4:05pm

Just register an atexit function. That will be called on a SIGTERM.

tkf · December 13, 2019, 8:45pm

Can you change the signal to be sent from the queuing system to SIGINT? Then there is a way to use try-catch for this. From How do I catch CTRL-C in a script?:

Running a Julia script using julia file.jl does not throw InterruptException when you try to terminate it with CTRL-C (SIGINT). To run a certain code before terminating a Julia script, which may or may not be caused by CTRL-C, use atexit. Alternatively, you can use julia -e 'include(popfirst!(ARGS))' file.jl to execute a script while being able to catch InterruptException in the try block.

josuagrw · December 13, 2019, 9:02pm

I agree that could also be a solution; I’m sure it would be possible to write a bash-wrapper that converts catches SIGTERM and sends SIGINT instead. The atexit() hook seems to be a much more elegant solution though.

josuagrw · December 13, 2019, 9:04pm

Thank you, that’s perfect.

mbesancon · June 21, 2021, 6:20pm

Reviving this thread because the links to the documentation seem dead now. We are looking for a good design for gracefully exiting an iterative procedure in FrankWolfe.jl

We would like not to make normal use cases pay the price for interruptions handling they don’t use, so try-catching the whole inner block is to be avoided. Would there be a recommended way to do it, not too hacky if possible.

fredrikekre · June 21, 2021, 6:33pm

Update doc links:

mbesancon · June 21, 2021, 7:57pm

Thanks! This confirms neither of these is suited to our use case. Pinging @cscherrer who found a solution when this is not a hot loop being interrupted

cscherrer · June 21, 2021, 8:35pm

This mostly works, but it’s easy to find cases where it doesn’t:

julia> macro cleanbreak(ex)
           quote
               try
                   $(esc(ex))
               catch e
                   if e isa InterruptException
                       @warn "Computation interrupted"
                   else
                       rethrow()
                   end    
               end
           end 
       end
@cleanbreak (macro with 1 method)

Here’s an example:

julia> x = 0
0

julia> @cleanbreak while true
       x += 1
       end
^C┌ Warning: Computation interrupted
└ @ Main REPL[1]:7

julia> x
18298742

And here’s an example from before I had made it into a macro. Much more realistic example, and you could use the macro to simplify it a bit:

github.com

cscherrer/BayesianLinearRegression.jl/blob/52a6fda9c76b5daeeecc2bc0e64e3e7e76d51d91/src/BayesianLinearRegression.jl#L224


      
              end
          
          
    updateWeights!(m)     
              
              m.iterationCount += 1
              return (m, iteration + 1)
          end
          
          
export fit!
          
          
function fit!(m::BayesianLinReg; kwargs...)
              m.done = false
              callback = get(kwargs, :callback, stopAtIteration(10))
          
          
    if m.updatePrior
                  m.priorPrecision = 1.0
              end
          
          
    if m.updateNoise
                  m.noisePrecision = 1.0
              end

captchanjack1 · June 19, 2022, 11:47pm

Hey all!

Question: Does anyone know if there is a Julia native solution for handling SIGTERM?

It seems callbacks registered with atexit function doesn’t actually run on SIGTERM, only on SIGINT

This is quite a problem for me because I have to find other solutions (e.g. rely on Kubernetes preStop hooks to handle SIGTERM gracefully), it would be nice to have a native solution

See these examples below:

SIGINT; you can see is handled correctly

julia> f() = (@info "Sleeping before exit"; sleep(3); @info "Exiting!")
f (generic function with 1 method)

julia> atexit(f)

julia> exit(0)
[ Info: Sleeping before exit
[ Info: Exiting!

SIGTERM; signal sent with kill -15 <pid>, throws an error “schedule: Task not runnable”, clearly the registered callback was terminated early

julia> getpid()
46496

julia> f() = (@info "Sleeping before exit"; sleep(3); @info "Exiting!")
f (generic function with 1 method)

julia> atexit(f)

julia> 
signal (15): Terminated: 15
in expression starting at none:0
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
_pthread_cond_wait at /usr/lib/system/libsystem_pthread.dylib (unknown line)
uv_cond_wait at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_task_get_next at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
poptask at ./task.jl:760
wait at ./task.jl:769
task_done_hook at ./task.jl:494
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_finish_task at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_threadfun at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
_pthread_start at /usr/lib/system/libsystem_pthread.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
_pthread_cond_wait at /usr/lib/system/libsystem_pthread.dylib (unknown line)
uv_cond_wait at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_task_get_next at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
poptask at ./task.jl:760
wait at ./task.jl:769
task_done_hook at ./task.jl:494
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_finish_task at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_threadfun at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
_pthread_start at /usr/lib/system/libsystem_pthread.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
_pthread_cond_wait at /usr/lib/system/libsystem_pthread.dylib (unknown line)
uv_cond_wait at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_task_get_next at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
poptask at ./task.jl:760
wait at ./task.jl:769
task_done_hook at ./task.jl:494
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_finish_task at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_threadfun at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
_pthread_start at /usr/lib/system/libsystem_pthread.dylib (unknown line)
unknown function (ip: 0x0)
kevent at /usr/lib/system/libsystem_kernel.dylib (unknown line)
uv__io_poll at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
uv_run at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_task_get_next at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
poptask at ./task.jl:760
wait at ./task.jl:769
wait at ./condition.jl:106
wait_readnb at ./stream.jl:413
eof at ./stream.jl:106
jfptr_eof_37611 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eof at ./io.jl:414
jfptr_eof_37616 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
match_input at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:1430
match_input at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:1430 [inlined]
match_input at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:1430
match_input at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:1430
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
prompt! at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:2526
run_interface at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:2436
jfptr_run_interface_45721 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
run_frontend at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:1126
#44 at ./task.jl:411
jfptr_YY.44_46662 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
start_task at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
unknown function (ip: 0x0)
Allocations: 5135309 (Pool: 5133189; Big: 2120); GC: 6
[ Info: Sleeping before exit
schedule: Task not runnable
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] schedule(t::Task, arg::Any; error::Bool)
    @ Base ./task.jl:644
  [3] schedule
    @ ./task.jl:638 [inlined]
  [4] uv_writecb_task(req::Ptr{Nothing}, status::Int32)
    @ Base ./stream.jl:1110
  [5] poptask(W::Base.InvasiveLinkedListSynchronized{Task})
    @ Base ./task.jl:760
  [6] wait()
    @ Base ./task.jl:769
  [7] uv_write(s::Base.TTY, p::Ptr{UInt8}, n::UInt64)
    @ Base ./stream.jl:992
  [8] unsafe_write(s::Base.TTY, p::Ptr{UInt8}, n::UInt64)
    @ Base ./stream.jl:1064
  [9] unsafe_write
    @ ./io.jl:646 [inlined]
 [10] write(s::Base.TTY, a::Vector{UInt8})
    @ Base ./io.jl:669
 [11] handle_message(logger::Logging.ConsoleLogger, level::Base.CoreLogging.LogLevel, message::Any, _module::Any, group::Any, id::Any, filepath::Any, line::Any; kwargs::Any)
    @ Logging /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Logging/src/ConsoleLogger.jl:167
 [12] handle_message(logger::Logging.ConsoleLogger, level::Base.CoreLogging.LogLevel, message::Any, _module::Any, group::Any, id::Any, filepath::Any, line::Any)
    @ Logging /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Logging/src/ConsoleLogger.jl:102
 [13] macro expansion
    @ ./logging.jl:310 [inlined]
 [14] f()
    @ Main ./REPL[2]:1
 [15] _atexit()
    @ Base ./initdefs.jl:343

Topic		Replies	Views
Catch ctrl-c in script General Usage question , interruptexception	2	152	September 27, 2024
How to prevent Ctrl-C from terminating a program during readline() General Usage question	3	1850	July 9, 2020
Catching signals, or other ways of branching within a loop? General Usage	1	457	August 11, 2017
Error in atexit() when stopping with SIGTERM signal General Usage error , deployment , atexit	0	786	May 3, 2021
Graceful interruption and ensure finishing current loop General Usage question , interruptexception	0	313	December 31, 2022

Handling SIGTERM in Julia

Related topics