Hi, from researching a bit I learned that handling unix signals has been tricky in Julia for a while. Our queuing systems send signals to programs to indicate they should come to an end; and I’d like to do stuff like storing all data in response to that. Did I miss something or is that still impossible in Julia code?
(I could probably achieve the same results by creating and deleting files, and checking their existence periodically - if anyone has code for that please let me know.)
Can you change the signal to be sent from the queuing system to SIGINT? Then there is a way to use try-catch for this. From How do I catch CTRL-C in a script?:
Running a Julia script using
julia file.jldoes not throwInterruptExceptionwhen you try to terminate it with CTRL-C (SIGINT). To run a certain code before terminating a Julia script, which may or may not be caused by CTRL-C, useatexit. Alternatively, you can usejulia -e 'include(popfirst!(ARGS))' file.jlto execute a script while being able to catchInterruptExceptionin thetryblock.
I agree that could also be a solution; I’m sure it would be possible to write a bash-wrapper that converts catches SIGTERM and sends SIGINT instead. The atexit() hook seems to be a much more elegant solution though.
Thank you, that’s perfect.
Reviving this thread because the links to the documentation seem dead now. We are looking for a good design for gracefully exiting an iterative procedure in FrankWolfe.jl
We would like not to make normal use cases pay the price for interruptions handling they don’t use, so try-catching the whole inner block is to be avoided. Would there be a recommended way to do it, not too hacky if possible.
Thanks! This confirms neither of these is suited to our use case. Pinging @cscherrer who found a solution when this is not a hot loop being interrupted
This mostly works, but it’s easy to find cases where it doesn’t:
julia> macro cleanbreak(ex)
quote
try
$(esc(ex))
catch e
if e isa InterruptException
@warn "Computation interrupted"
else
rethrow()
end
end
end
end
@cleanbreak (macro with 1 method)
Here’s an example:
julia> x = 0
0
julia> @cleanbreak while true
x += 1
end
^C┌ Warning: Computation interrupted
└ @ Main REPL[1]:7
julia> x
18298742
And here’s an example from before I had made it into a macro. Much more realistic example, and you could use the macro to simplify it a bit:
Hey all!
Question: Does anyone know if there is a Julia native solution for handling SIGTERM?
It seems callbacks registered with atexit function doesn’t actually run on SIGTERM, only on SIGINT
This is quite a problem for me because I have to find other solutions (e.g. rely on Kubernetes preStop hooks to handle SIGTERM gracefully), it would be nice to have a native solution
See these examples below:
SIGINT; you can see is handled correctly
julia> f() = (@info "Sleeping before exit"; sleep(3); @info "Exiting!")
f (generic function with 1 method)
julia> atexit(f)
julia> exit(0)
[ Info: Sleeping before exit
[ Info: Exiting!
SIGTERM; signal sent with kill -15 <pid>, throws an error “schedule: Task not runnable”, clearly the registered callback was terminated early
julia> getpid()
46496
julia> f() = (@info "Sleeping before exit"; sleep(3); @info "Exiting!")
f (generic function with 1 method)
julia> atexit(f)
julia>
signal (15): Terminated: 15
in expression starting at none:0
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
_pthread_cond_wait at /usr/lib/system/libsystem_pthread.dylib (unknown line)
uv_cond_wait at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_task_get_next at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
poptask at ./task.jl:760
wait at ./task.jl:769
task_done_hook at ./task.jl:494
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_finish_task at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_threadfun at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
_pthread_start at /usr/lib/system/libsystem_pthread.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
_pthread_cond_wait at /usr/lib/system/libsystem_pthread.dylib (unknown line)
uv_cond_wait at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_task_get_next at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
poptask at ./task.jl:760
wait at ./task.jl:769
task_done_hook at ./task.jl:494
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_finish_task at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_threadfun at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
_pthread_start at /usr/lib/system/libsystem_pthread.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
_pthread_cond_wait at /usr/lib/system/libsystem_pthread.dylib (unknown line)
uv_cond_wait at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_task_get_next at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
poptask at ./task.jl:760
wait at ./task.jl:769
task_done_hook at ./task.jl:494
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_finish_task at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_threadfun at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
_pthread_start at /usr/lib/system/libsystem_pthread.dylib (unknown line)
unknown function (ip: 0x0)
kevent at /usr/lib/system/libsystem_kernel.dylib (unknown line)
uv__io_poll at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
uv_run at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_task_get_next at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
poptask at ./task.jl:760
wait at ./task.jl:769
wait at ./condition.jl:106
wait_readnb at ./stream.jl:413
eof at ./stream.jl:106
jfptr_eof_37611 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eof at ./io.jl:414
jfptr_eof_37616 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
match_input at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:1430
match_input at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:1430 [inlined]
match_input at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:1430
match_input at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:1430
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
prompt! at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:2526
run_interface at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:2436
jfptr_run_interface_45721 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
run_frontend at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:1126
#44 at ./task.jl:411
jfptr_YY.44_46662 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
start_task at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
unknown function (ip: 0x0)
Allocations: 5135309 (Pool: 5133189; Big: 2120); GC: 6
[ Info: Sleeping before exit
schedule: Task not runnable
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] schedule(t::Task, arg::Any; error::Bool)
@ Base ./task.jl:644
[3] schedule
@ ./task.jl:638 [inlined]
[4] uv_writecb_task(req::Ptr{Nothing}, status::Int32)
@ Base ./stream.jl:1110
[5] poptask(W::Base.InvasiveLinkedListSynchronized{Task})
@ Base ./task.jl:760
[6] wait()
@ Base ./task.jl:769
[7] uv_write(s::Base.TTY, p::Ptr{UInt8}, n::UInt64)
@ Base ./stream.jl:992
[8] unsafe_write(s::Base.TTY, p::Ptr{UInt8}, n::UInt64)
@ Base ./stream.jl:1064
[9] unsafe_write
@ ./io.jl:646 [inlined]
[10] write(s::Base.TTY, a::Vector{UInt8})
@ Base ./io.jl:669
[11] handle_message(logger::Logging.ConsoleLogger, level::Base.CoreLogging.LogLevel, message::Any, _module::Any, group::Any, id::Any, filepath::Any, line::Any; kwargs::Any)
@ Logging /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Logging/src/ConsoleLogger.jl:167
[12] handle_message(logger::Logging.ConsoleLogger, level::Base.CoreLogging.LogLevel, message::Any, _module::Any, group::Any, id::Any, filepath::Any, line::Any)
@ Logging /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Logging/src/ConsoleLogger.jl:102
[13] macro expansion
@ ./logging.jl:310 [inlined]
[14] f()
@ Main ./REPL[2]:1
[15] _atexit()
@ Base ./initdefs.jl:343