Hi, I have some code that works fine when I run it on a single node of our slurm cluster using sbatch, but when I try to run it on multiple nodes I get errors similar to the following:
srun: error: cnode22: tasks 7,12,15,19: Exited with exit code 1
srun: Terminating job step 252267.0
srun: error: cnode22: tasks 6,13,16-17,21,27: Exited with exit code 1
srun: error: cnode22: tasks 0-5,8-11,14,18,20,22-26: Exited with exit code 1
ERROR: LoadError: TaskFailedException:
UndefVarError: warn not defined
Stacktrace:
[1] launch(::SlurmManager, ::Dict{Symbol,Any}, ::Array{WorkerConfig,1}, ::Base.GenericCondition{Base.AlwaysLockedST}) at /home/dcs/csrxgb/.julia/packages/ClusterManagers/7pPEP/src/slurm.jl:52
[2] (::Distributed.var"#39#42"{SlurmManager,Dict{Symbol,Any},Array{WorkerConfig,1},Base.GenericCondition{Base.AlwaysLockedST}})() at ./task.jl:358
Stacktrace:
[1] wait at ./task.jl:267 [inlined]
[2] addprocs_locked(::SlurmManager; kwargs::Base.Iterators.Pairs{Symbol,String,Tuple{Symbol},NamedTuple{(:o,),Tuple{String}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:494
[3] addprocs(::SlurmManager; kwargs::Base.Iterators.Pairs{Symbol,String,Tuple{Symbol},NamedTuple{(:o,),Tuple{String}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:441
[4] top-level scope at /gpfs/home/dcs/csrxgb/julia_stuff/src/logistic_regression/run.jl:3
[5] include(::Module, ::String) at ./Base.jl:377
[6] exec_options(::Base.JLOptions) at ./client.jl:288
[7] _start() at ./client.jl:484
in expression starting at /gpfs/home/dcs/csrxgb/julia_stuff/src/logistic_regression/run.jl:3
[csrxgb@orac:login1 julia_stuff]$ cat slurm.cnode15.252269.err
srun: error: cnode15: tasks 3,12,14,26: Exited with exit code 1
srun: Terminating job step 252269.0
srun: error: cnode22: tasks 31-34,36,39-40,42,44-47,50-52: Exited with exit code 1
srun: error: cnode71: tasks 143,147,154: Exited with exit code 1
srun: error: cnode22: tasks 28,30,37-38,41,49,53-54: Exited with exit code 143
srun: error: cnode22: tasks 29,35,43,48,55: Exited with exit code 1
srun: error: cnode71: tasks 140,144,146,151,155-156,159,167: Exited with exit code 1
srun: error: cnode71: tasks 141-142,145,148-150,152-153,157-158,160-166: Exited with exit code 143
srun: error: cnode15: tasks 0,5,13,25: Exited with exit code 143
srun: error: cnode15: tasks 7-8,17,20,22: Exited with exit code 1
srun: error: cnode15: tasks 1-2,4,6,9-11,15-16,18-19,21,23,27: Exited with exit code 143
srun: error: cnode15: task 24: Exited with exit code 1
srun: error: cnode75: tasks 168-195: Exited with exit code 143
srun: error: cnode76: tasks 196-198,204-205,207-208,218,221: Exited with exit code 1
srun: error: cnode76: tasks 199-203,206,210-212,217,219,222-223: Exited with exit code 1
srun: error: cnode76: tasks 209,213-216,220: Exited with exit code 143
srun: error: cnode31: tasks 84-111: Exited with exit code 143
srun: error: cnode25: tasks 56-83: Exited with exit code 143
srun: error: cnode33: tasks 112-139: Exited with exit code 143
ERROR: LoadError: TaskFailedException:
UndefVarError: warn not defined
Stacktrace:
[1] launch(::SlurmManager, ::Dict{Symbol,Any}, ::Array{WorkerConfig,1}, ::Base.GenericCondition{Base.AlwaysLockedST}) at /home/dcs/csrxgb/.julia/packages/ClusterManagers/7pPEP/src/slurm.jl:52
[2] (::Distributed.var"#39#42"{SlurmManager,Dict{Symbol,Any},Array{WorkerConfig,1},Base.GenericCondition{Base.AlwaysLockedST}})() at ./task.jl:358
Stacktrace:
[1] wait at ./task.jl:267 [inlined]
[2] addprocs_locked(::SlurmManager; kwargs::Base.Iterators.Pairs{Symbol,String,Tuple{Symbol},NamedTuple{(:o,),Tuple{String}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:494
[3] addprocs(::SlurmManager; kwargs::Base.Iterators.Pairs{Symbol,String,Tuple{Symbol},NamedTuple{(:o,),Tuple{String}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:441
[4] top-level scope at /gpfs/home/dcs/csrxgb/julia_stuff/src/logistic_regression/run.jl:3
[5] include(::Module, ::String) at ./Base.jl:377
[6] exec_options(::Base.JLOptions) at ./client.jl:288
[7] _start() at ./client.jl:484
I am using addprocs_slurm from the ClusterManagers package and then a distributed pmap to run some computations. I can’t really see anything useful from the error above to try and debug this. I additionally get the following output from some of the workers:
julia_worker:9403#10.1.24.15
julia_worker:9405#10.1.24.15
julia_worker:9939#10.1.24.22
julia_worker:9953#10.1.24.22
julia_worker:9959#10.1.24.22
julia_worker:9952#10.1.24.22
julia_worker:9947#10.1.24.22
julia_worker:9394#10.1.24.15
julia_worker:9754#10.1.24.71
julia_worker:9940#10.1.24.22
julia_worker:9948#10.1.24.22
julia_worker:9958#10.1.24.22
julia_worker:9942#10.1.24.22
julia_worker:9417#10.1.24.15
julia_worker:9944#10.1.24.22
julia_worker:9765#10.1.24.71
julia_worker:9960#10.1.24.22
julia_worker:9955#10.1.24.22
julia_worker:9954#10.1.24.22
julia_worker:9950#10.1.24.22
julia_worker:9758#10.1.24.71
julia_worker:9941#10.1.24.22
julia_worker:9956#10.1.24.22
julia_worker:9755#10.1.24.71
julia_worker:9778#10.1.24.71
julia_worker:9767#10.1.24.71
julia_worker:9757#10.1.24.71
julia_worker:9963#10.1.24.22
julia_worker:9943#10.1.24.22
julia_worker:9751#10.1.24.71
julia_worker:9762#10.1.24.71
julia_worker:9961#10.1.24.22
julia_worker:9770#10.1.24.71
julia_worker:9408#10.1.24.15
julia_worker:9951#10.1.24.22
julia_worker:9937#10.1.24.22
julia_worker:9413#10.1.24.15
julia_worker:9766#10.1.24.71
julia_worker:9936#10.1.24.22
julia_worker:9774#10.1.24.71
julia_worker:9945#10.1.24.22
julia_worker:9399#10.1.24.15
julia_worker:9411#10.1.24.15
julia_worker:9938#10.1.24.22
julia_worker:9949#10.1.24.22
julia_worker:9398#10.1.24.15
julia_worker:9775#10.1.24.71
julia_worker:9946#10.1.24.22
julia_worker:9962#10.1.24.22
julia_worker:9760#10.1.24.71
julia_worker:9957#10.1.24.22
julia_worker:9756#10.1.24.71
julia_worker:9773#10.1.24.71
julia_worker:9769#10.1.24.71
julia_worker:9415#10.1.24.15
julia_worker:9400#10.1.24.15
julia_worker:9414#10.1.24.15
julia_worker:9396#10.1.24.15
julia_worker:9406#10.1.24.15
julia_worker:9412#10.1.24.15
julia_worker:9753#10.1.24.71
julia_worker:9768#10.1.24.71
julia_worker:9395#10.1.24.15
julia_worker:9772#10.1.24.71
julia_worker:9761#10.1.24.71
julia_worker:9404#10.1.24.15
julia_worker:9392#10.1.24.15
julia_worker:9391#10.1.24.15
julia_worker:9401#10.1.24.15
julia_worker:9416#10.1.24.15
julia_worker:9418#10.1.24.15
julia_worker:9409#10.1.24.15
julia_worker:9777#10.1.24.71
julia_worker:9764#10.1.24.71
julia_worker:9393#10.1.24.15
julia_worker:9759#10.1.24.71
julia_worker:9402#10.1.24.15
julia_worker:9410#10.1.24.15
julia_worker:9407#10.1.24.15
julia_worker:9397#10.1.24.15
julia_worker:9771#10.1.24.71
julia_worker:9776#10.1.24.71
julia_worker:9752#10.1.24.71
julia_worker:9763#10.1.24.71
julia_worker:9091#10.1.24.76
julia_worker:9082#10.1.24.76
julia_worker:9104#10.1.24.76
julia_worker:9093#10.1.24.76
julia_worker:9084#10.1.24.76
julia_worker:9090#10.1.24.76
julia_worker:9083#10.1.24.76
julia_worker:9107#10.1.24.76
julia_worker:9094#10.1.24.76
julia_worker:9098#10.1.24.76
julia_worker:9109#10.1.24.76
julia_worker:9087#10.1.24.76
julia_worker:9089#10.1.24.76
julia_worker:9096#10.1.24.76
julia_worker:9086#10.1.24.76
julia_worker:9105#10.1.24.76
julia_worker:9103#10.1.24.76
julia_worker:9097#10.1.24.76
julia_worker:9085#10.1.24.76
julia_worker:9088#10.1.24.76
julia_worker:9092#10.1.24.76
julia_worker:9108#10.1.24.76
julia_worker:9101#10.1.24.76
julia_worker:9100#10.1.24.76
julia_worker:9095#10.1.24.76
julia_worker:9099#10.1.24.76
julia_worker:9102#10.1.24.76
julia_worker:9106#10.1.24.76
julia_worker:9335#10.1.24.75
julia_worker:9345#10.1.24.75
julia_worker:9339#10.1.24.75
julia_worker:9347#10.1.24.75
julia_worker:9344#10.1.24.75
julia_worker:9334#10.1.24.75
julia_worker:9346#10.1.24.75
julia_worker:9340#10.1.24.75
julia_worker:9355#10.1.24.75
julia_worker:9351#10.1.24.75
julia_worker:9338#10.1.24.75
julia_worker:9354#10.1.24.75
julia_worker:9343#10.1.24.75
julia_worker:9350#10.1.24.75
julia_worker:9331#10.1.24.75
julia_worker:9336#10.1.24.75
julia_worker:9333#10.1.24.75
julia_worker:9348#10.1.24.75
julia_worker:9352#10.1.24.75
julia_worker:9337#10.1.24.75
julia_worker:9332#10.1.24.75
julia_worker:9341#10.1.24.75
julia_worker:9328#10.1.24.75
julia_worker:9329#10.1.24.75
julia_worker:9330#10.1.24.75
julia_worker:9349#10.1.24.75
julia_worker:9353#10.1.24.75
julia_worker:9342#10.1.24.75
julia_worker:9584#10.1.24.31
julia_worker:9565#10.1.24.31
julia_worker:9582#10.1.24.31
julia_worker:9175#10.1.24.33
julia_worker:9577#10.1.24.31
julia_worker:9563#10.1.24.31
julia_worker:9578#10.1.24.31
julia_worker:9315#10.1.24.25
julia_worker:9587#10.1.24.31
julia_worker:9318#10.1.24.25
julia_worker:9566#10.1.24.31
julia_worker:9181#10.1.24.33
julia_worker:9314#10.1.24.25
julia_worker:9581#10.1.24.31
julia_worker:9579#10.1.24.31
julia_worker:9172#10.1.24.33
julia_worker:9316#10.1.24.25
julia_worker:9178#10.1.24.33
julia_worker:9173#10.1.24.33
julia_worker:9580#10.1.24.31
julia_worker:9325#10.1.24.25
julia_worker:9313#10.1.24.25
julia_worker:9572#10.1.24.31
julia_worker:9328#10.1.24.25
julia_worker:9562#10.1.24.31
julia_worker:9183#10.1.24.33
julia_worker:9333#10.1.24.25
julia_worker:9170#10.1.24.33
julia_worker:9569#10.1.24.31
julia_worker:9162#10.1.24.33
julia_worker:9334#10.1.24.25
julia_worker:9171#10.1.24.33
julia_worker:9169#10.1.24.33
julia_worker:9179#10.1.24.33
julia_worker:9164#10.1.24.33
julia_worker:9574#10.1.24.31
julia_worker:9185#10.1.24.33
julia_worker:9165#10.1.24.33
julia_worker:9317#10.1.24.25
julia_worker:9573#10.1.24.31
julia_worker:9177#10.1.24.33
julia_worker:9322#10.1.24.25
julia_worker:9166#10.1.24.33
julia_worker:9326#10.1.24.25
julia_worker:9332#10.1.24.25
julia_worker:9575#10.1.24.31
julia_worker:9335#10.1.24.25
julia_worker:9560#10.1.24.31
julia_worker:9163#10.1.24.33
julia_worker:9330#10.1.24.25
julia_worker:9571#10.1.24.31
julia_worker:9156#10.1.24.33
julia_worker:9336#10.1.24.25
julia_worker:9160#10.1.24.33
julia_worker:9321#10.1.24.25
julia_worker:9329#10.1.24.25
julia_worker:9167#10.1.24.33
julia_worker:9331#10.1.24.25
julia_worker:9576#10.1.24.31
julia_worker:9182#10.1.24.33
julia_worker:9338#10.1.24.25
julia_worker:9583#10.1.24.31
julia_worker:9588#10.1.24.31
julia_worker:9312#10.1.24.25
julia_worker:9184#10.1.24.33
julia_worker:9176#10.1.24.33
julia_worker:9568#10.1.24.31
julia_worker:9323#10.1.24.25
julia_worker:9585#10.1.24.31
julia_worker:9320#10.1.24.25
julia_worker:9337#10.1.24.25
julia_worker:9319#10.1.24.25
julia_worker:9567#10.1.24.31
julia_worker:9564#10.1.24.31
julia_worker:9180#10.1.24.33
julia_worker:9168#10.1.24.33
julia_worker:9161#10.1.24.33
julia_worker:9158#10.1.24.33
julia_worker:9327#10.1.24.25
julia_worker:9159#10.1.24.33
julia_worker:9324#10.1.24.25
julia_worker:9561#10.1.24.31
julia_worker:9311#10.1.24.25
julia_worker:9570#10.1.24.31
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
slurmstepd: error: *** STEP 252269.0 ON cnode15 CANCELLED AT 2020-05-12T16:10:07 ***
signal (15): Terminated
in expression starting at none:0
#56 at ./task.jl:358
unknown function (ip: 0x7ff08d486c7c)
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f54a5055c7c)
_trywait at ./asyncevent.jl:110
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f80b442fc7c)
_trywait at ./asyncevent.jl:110
_trywait at ./asyncevent.jl:110
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7fdcafd67c7c)
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f0851da9c7c)
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7fb0da308c7c)
_trywait at ./asyncevent.jl:110
_trywait at ./asyncevent.jl:110
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f4b7271ec7c)
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f3248d42c7c)
wait at ./task.jl:709 [inlined]
wait at ./condition.jl:106
_trywait at ./asyncevent.jl:110
_trywait at ./asyncevent.jl:110
_trywait at ./asyncevent.jl:110
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f19ad739c7c)
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f60fb148c7c)
_trywait at ./asyncevent.jl:110
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7fa4cd263c7c)
_trywait at ./asyncevent.jl:110
_trywait at ./asyncevent.jl:110
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f870022dc7c)
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f7dfa31fc7c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f817f30dc7c)
_trywait at ./asyncevent.jl:110
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f20ad20cc7c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_trywait at ./asyncevent.jl:110
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7fd3c3422c7c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_trywait at ./asyncevent.jl:110
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f69a4cf9c7c)
_trywait at ./asyncevent.jl:110
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f1681349c7c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_trywait at ./asyncevent.jl:110
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7fa433312c7c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7fca81337c7c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
wait at ./task.jl:709 [inlined]
wait at ./condition.jl:106
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_trywait at ./asyncevent.jl:110
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7fb32c100c7c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_trywait at ./asyncevent.jl:110
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7fbedf523c7c)
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_trywait at ./asyncevent.jl:110
poptaskref at ./task.jl:702
wait at ./asyncevent.jl:128 [inlined]
sleep at ./asyncevent.jl:213 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:707 [inlined]
#56 at ./task.jl:358
unknown function (ip: 0x7f2595596c7c)
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
poptaskref at ./task.jl:702
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_trywait at ./asyncevent.jl:110
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873532 (Pool: 873280; Big: 252); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873531 (Pool: 873279; Big: 252); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873530 (Pool: 873279; Big: 251); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873530 (Pool: 873279; Big: 251); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873530 (Pool: 873279; Big: 251); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873532 (Pool: 873279; Big: 253); GC: 1
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873530 (Pool: 873279; Big: 251); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873531 (Pool: 873279; Big: 252); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873532 (Pool: 873280; Big: 252); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873530 (Pool: 873279; Big: 251); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873533 (Pool: 873280; Big: 253); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873531 (Pool: 873280; Big: 251); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873531 (Pool: 873279; Big: 252); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873530 (Pool: 873279; Big: 251); GC: 1
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873530 (Pool: 873279; Big: 251); GC: 1
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 873533 (Pool: 873280; Big: 253); GC: 1
Had to remove parts of the above to post this