Docker Task Error (SIGINT)

General Usage
1

I’m trying to run some code in a Docker container on AWS. It works locally, but fails with a cryptic error once in the Cloud. I can’t find a MWE because the stack trace really doesn’t give me much information to go off of:

message
  Activating project at `/usr/local`
[ Info: Running `main` partition function
[1] signal (15): Terminated
in expression starting at /usr/local/consolidator_script.jl:35
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
task_done_hook at ./task.jl:675
jfptr_task_done_hook_75257.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_finish_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:320
jl_threadfun at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:191
start_thread at /lib64/libc.so.6 (unknown line)
clone at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
__futex_abstimed_wait_common at /lib64/libc.so.6 (unknown line)
pthread_cond_wait at /lib64/libc.so.6 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:509
poptask at ./task.jl:985
wait at ./task.jl:994
#wait#645 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
_trywait at ./asyncevent.jl:138
wait at ./asyncevent.jl:155 [inlined]
profile_printing_listener at ./Base.jl:566
#1054 at ./Base.jl:602
jfptr_YY.1054_84908.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
start_task at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/task.c:1238
unknown function (ip: (nil))
epoll_wait at /lib64/libc.so.6 (unknown line)
uv__io_poll at /workspace/srcdir/libuv/src/unix/epoll.c:236
uv_run at /workspace/srcdir/libuv/src/unix/core.c:400
ijl_task_get_next at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/partr.c:470
poptask at ./task.jl:985
wait at ./task.jl:994
#wait#645 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
wait at ./process.jl:661
success at ./process.jl:523
jfptr_success_77427.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
compilecache at ./loading.jl:2370
compilecache at ./loading.jl:2334 [inlined]
#967 at ./loading.jl:1968
#mkpidlock#7 at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:92
unknown function (ip: 0x7fce3072d136)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
mkpidlock at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:89
#mkpidlock#6 at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:87 [inlined]
mkpidlock at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:87
unknown function (ip: 0x7fce3072c7f0)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
do_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/builtins.c:768
#trymkpidlock#11 at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:110
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
do_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/builtins.c:768
trymkpidlock at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:108
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/builtins.c:812
#invokelatest#2 at ./essentials.jl:889 [inlined]
invokelatest at ./essentials.jl:884 [inlined]
#maybe_cachefile_lock#982 at ./loading.jl:2978
maybe_cachefile_lock at ./loading.jl:2975 [inlined]
_require at ./loading.jl:1964
__require_prelocked at ./loading.jl:1806
jfptr___require_prelocked_80749.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_f__call_in_world at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/builtins.c:831
#invoke_in_world#3 at ./essentials.jl:921 [inlined]
invoke_in_world at ./essentials.jl:918 [inlined]
_require_prelocked at ./loading.jl:1797
macro expansion at ./loading.jl:1784 [inlined]
macro expansion at ./lock.jl:267 [inlined]
__require at ./loading.jl:1747
jfptr___require_80714.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
jl_f__call_in_world at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/builtins.c:831
#invoke_in_world#3 at ./essentials.jl:921 [inlined]
invoke_in_world at ./essentials.jl:918 [inlined]
require at ./loading.jl:1740
jfptr_require_80711.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
call_require at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/toplevel.c:481 [inlined]
eval_import_path at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/toplevel.c:518
jl_toplevel_eval_flex at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/toplevel.c:752
eval_body at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/interpreter.c:582
jl_interpret_toplevel_thunk at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/interpreter.c:774
jl_toplevel_eval_flex at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:383 [inlined]
include_string at ./loading.jl:2070
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
_include at ./loading.jl:2130
include at ./Base.jl:489
jfptr_include_46308.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
exec_options at ./client.jl:318
_start at ./client.jl:552
jfptr__start_82727.1 at /usr/local/julia-1.10.0-beta2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:2889 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/gf.c:3071
jl_apply at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/julia.h:1966 [inlined]
true_main at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at /cache/build/default-amdci5-3/julialang/julia-release-1-dot-10/cli/loader_exe.c:58
__libc_start_call_main at /lib64/libc.so.6 (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
unknown function (ip: (nil))
Allocations: 528515 (Pool: 527517; Big: 998); GC: 1

After some Google searching, I found a similar error in MongoDB that seemed to be Linux shutting it off because it was too slow to respond. This error seems to be occurring during precompilation, so maybe that’s the culprit?

2

FWIW, here is the code:

using Pkg
Pkg.activate(".")
using Logging
include("src/ScriptUtils.jl")

"""Loop over the sqs queue and consolidate them"""
function consolidate_partitions(target_bucket, queue_url, dead_letter_queue_url; kwargs...)
    @info "Starting to consolidate partitions"
    while true
        partition_path = waiting_receive_message(queue_url)
        if partition_path === nothing
            @info "No more messages in queue"
            break
        end
        try
            TableConsolidator.consolidate_partition(partition_path["ReceiveMessageResult"]["Message"]["Body"], target_bucket; kwargs...)
        catch e
            @error "Failed to consolidate partition" partition_path e stacktrace()
            Sqs.send_message(dead_letter_queue_url, partition_path)
        end
        Sqs.delete_message(queue_url, partition_path["ReceiveMessageResult"]["Message"]["ReceiptHandle"])
    end
end


"""Get the list of partitions in the given table and put them to the SQS queue"""
function populate_queue(database_name, table_name)
    queue_url = ENV["PartitionListQueueUrl"]
    for partition_path in list_partitions(database_name, table_name)
        Sqs.send_message(partition_path, queue_url)
    end
end


# Kick off the correct task
if ENV["task_name"] == "populate_queue"
    @info "Running `populate_queue`"
    populate_queue(ENV["database_name"], ENV["table_name"])

elseif ENV["task_name"] == "consolidate_partitions"
    @info "Running `main` partition function"

    using TableConsolidator
    using TypedTables
    @info "Successfully loaded TableConsolidator.jl"
    queue_url = ENV["PartitionListQueueUrl"]
    deadletter_queue_url = ENV["DeadLetterPartitionListQueueUrl"]

    kw_defaults = [
        ("batch_size",              Int, "500"),
        ("scan_precision",          Int, "4097"),
        ("row_group_chunk_size",    Int, "500"),
        ("chunks_per_file",         Int, "30"),
        ("big_file_cutoff",         Int, "100"),
        ("track_dataset_buffers",   Bool, "false"),
    ]

    kwargs = Dict(
        Symbol(kw[1]) => parse(kw[2], get(ENV, kw[1], kw[3]))
        for kw in kw_defaults
    )

    @info "Running `consolidate_partitions` with kwargs" kwargs
    consolidate_partitions(ENV["target_bucket"], queue_url, deadletter_queue_url;
        bin_column = ENV["bin_column"],
        column_dtype = TableConsolidator.julia_type(ENV["column_dtype"]),
        kwargs...
        )
end

TableConsolidator is the package I wrote for this task. Happy to share more code, but I’m not even getting into the package, so I don’t think that’s the problem.

Line 35 is if ENV["task_name"] == "populate_queue"

3

I only have some general advice.

In similar situations, I would try to write the most important details to the log when starting up, and check for any significant differences:

  • Available memory,
  • Number of cores,
  • Memory amount per thread,
  • CPU architecture (Arm64 vs. x86, AVX2, AVX-512),
  • Julia version number, installed package versions,
  • Docker version, Host system linux kernel version,
  • Any native optimizations for the CPU.

For example, sometimes I forget to update the Docker base image either locally or in the cloud, resulting in different program versions. But I’m more careful about this now.

Or the Alpine Linux image, which updates quickly twice a year, is sensitive if the host kernel is too old.

It’s also worth checking the host system’s Linux kernel log. There might be messages that can help identify the problem.

And what is your Docker base image?

  • julia:latest
  • ubuntu:22.04
  • public.ecr.aws/amazonlinux/amazonlinux:latest
  • …
4

I really appreciate your help!

Frustratingly, I redeployed with some additional logging statements, and my next test run ran without a hitch. Makes me concerned that it’s a sometimes bug :grimacing:

I’ll check the differences tomorrow, but in the meantime, I know that my Julia versions match and that my thread to memory ratio (8 to 32) is quite generous.

I’m interested to hear you say differences in CPU. I am precompiling during the build phase which, at the moment, is happening locally, so there is definitely a difference in CPU type/capabilities (both x86, but possibly different on AVX).

If that is different, does that mean I need to not precompile? It’s a little sad that docker isn’t actually immune to “works on my machine”!

5

Another thought:

You could share a few details, like which specific AWS service you’re talking about. This might help others facing similar problems. (EC2, ECS, Fairgate, … )

For example, what I only partially understand is that the container lifecycle in Amazon Elastic Container Service (Amazon ECS) is quite interesting.
And handling expected Spot terminations isn’t straightforward.

6

I’m kicking off tasks defined in Fargate from a step function, so they aren’t managed by a service or anything.

It just occured to me that my failed run had multiple workers, but I only looked at the first error that the step function presented, but since this is a graceful exit, its possible that another worker failed for a more intelligible reason and then the step function ordered a graceful shutdown of all the other workers, yielding this weird one.

Can’t get on to investigate this late. But thanks for the lead!

1 Like