I have some code that evaluates trajectories of a stochastic master equation in parallel, using a @distributed for with reduction for evaluating average quantities. The code is available here.
Some jobs fail with the following error, which I can’t understand. Can anyone provide some insight or possible ideas on how to diagnose it?
ERROR: LoadError: TaskFailedException:
IOError: stream is closed or unusable
Stacktrace:
[1] uv_write_async(::Sockets.TCPSocket, ::Ptr{UInt8}, ::UInt64) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
[2] uv_write(::Sockets.TCPSocket, ::Ptr{UInt8}, ::UInt64) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
[3] unsafe_write(::Sockets.TCPSocket, ::Ptr{UInt8}, ::UInt64) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
[4] unsafe_write at ./io.jl:593 [inlined]
[5] macro expansion at ./gcutils.jl:91 [inlined]
[6] write at ./io.jl:616 [inlined]
[7] serialize_array_data at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:246 [inlined]
[8] serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Array{Complex{Float64},2}) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:263
[9] serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Array{Array{Complex{Float64},2},1}) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:268
[10] serialize_any(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:629
[11] serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:608
[12] serialize_any(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:629
[13] serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:608
[14] serialize_any(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:629
[15] serialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:608 [inlined]
[16] serialize_msg(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Distributed.CallMsg{:call_fetch}) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:90
[17] #invokelatest#1 at ./essentials.jl:709 [inlined]
[18] invokelatest at ./essentials.jl:708 [inlined]
[19] send_msg_(::Distributed.Worker, ::Distributed.MsgHeader, ::Distributed.CallMsg{:call_fetch}, ::Bool) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:185
[20] send_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:134 [inlined]
[21] #remotecall_fetch#145(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_fetch), ::Function, ::Distributed.Worker, ::Function, ::Vararg{Any,N} where N) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:385
[22] remotecall_fetch(::Function, ::Distributed.Worker, ::Function, ::Vararg{Any,N} where N) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:382
[23] #remotecall_fetch#148 at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:417 [inlined]
[24] remotecall_fetch at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:417 [inlined]
[25] (::Distributed.var"#159#160"{typeof(+),ContinuousMeasurementFI.var"#15#19"{Float64,Float64,TimerOutputs.TimerOutput,Int64,Float64,ContinuousMeasurementFI.var"#dW#16"{Float64},RemoteChannel{Channel{Bool}}},UnitRange{Int64},Array{UnitRange{Int64},1},Int64,Int64})() at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/macros.jl:270
Stacktrace:
[1] wait at ./task.jl:251 [inlined]
[2] fetch at ./task.jl:266 [inlined]
[3] iterate at ./generator.jl:47 [inlined]
[4] collect_to!(::Array{Array{Float64,2},1}, ::Base.Generator{Array{Task,1},typeof(fetch)}, ::Int64, ::Int64) at ./array.jl:667
[5] collect_to_with_first!(::Array{Array{Float64,2},1}, ::Array{Float64,2}, ::Base.Generator{Array{Task,1},typeof(fetch)}, ::Int64) at ./array.jl:646
[6] collect(::Base.Generator{Array{Task,1},typeof(fetch)}) at ./array.jl:627
[7] preduce(::Function, ::Function, ::UnitRange{Int64}) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/macros.jl:274
[8] macro expansion at /home/matros/.julia/packages/ProgressMeter/g1lse/src/ProgressMeter.jl:595 [inlined]
[9] macro expansion at /home/matros/.julia/packages/ContinuousMeasurementFI/tmNbJ/src/Eff_QFI_HD_Dicke.jl:135 [inlined]
[10] macro expansion at /home/matros/.julia/packages/TimerOutputs/7Id5J/src/TimerOutput.jl:214 [inlined]
[11] #Eff_QFI_HD_Dicke#12(::Float64, ::Float64, ::Float64, ::Float64, ::Int64, ::TimerOutputs.TimerOutput, ::typeof(Eff_QFI_HD_Dicke), ::Int64, ::Int64, ::Float64, ::Float64) at /home/matros/.julia/packages/ContinuousMeasurementFI/tmNbJ/src/Eff_QFI_HD_Dicke.jl:134
[12] (::ContinuousMeasurementFI.var"#kw##Eff_QFI_HD_Dicke")(::NamedTuple{(:κ, :κcoll, :ω, :η, :outpoints),Tuple{Float64,Float64,Float64,Float64,Int64}}, ::typeof(Eff_QFI_HD_Dicke), ::Int64, ::Int64, ::Float64, ::Float64) at ./none:0
[13] top-level scope at /home/matros/continuous_measurement_dicke/sims_20200217/simulateTama.jl:28
[14] include_relative(::Module, ::String) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
[15] include(::Module, ::String) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
[16] exec_options(::Base.JLOptions) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
[17] _start() at /home/matros/julia-1.3.0/lib/julia/sys.so:?
in expression starting at /home/matros/continuous_measurement_dicke/sims_20200217/simulateTama.jl:28
ARGS = ["50", "5000", "1.0", "0.00005", "1.0", "1.0", "0.01", "1.0", "200"]
ContinuousMeasurementFI.SuperOperator{Complex{Float64},Int64}
ContinuousMeasurementFI.SuperOperator{Complex{Float64},Int64}
From worker 73:
From worker 73: signal (11): Segmentation fault
From worker 73: in expression starting at none:0
From worker 73: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
From worker 73: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
From worker 73: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 73: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 76:
From worker 76: signal (11): Segmentation fault
From worker 76: in expression starting at none:0
From worker 73: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 76: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
From worker 76: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
From worker 76: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 73: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 76: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 73: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 73: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 76: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 73: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 76: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 73: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 76: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 73: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 73: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
From worker 73: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 76: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 73: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
From worker 73: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
From worker 73: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
From worker 76: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 73: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 76: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 73: process_tcp_streams at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 73: #101 at ./task.jl:333
From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 73: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
From worker 73: start_task at /buildworker/worker/package_linux64/build/src/task.c:659
From worker 73: unknown function (ip: 0xffffffffffffffff)
From worker 73: Allocations: 16906524 (Pool: 16903554; Big: 2970); GC: 18
From worker 76: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 76: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
From worker 76: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 76: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
From worker 76: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
From worker 76: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
From worker 76: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 76: process_tcp_streams at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 76: #101 at ./task.jl:333
From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 76: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
From worker 76: start_task at /buildworker/worker/package_linux64/build/src/task.c:659
From worker 76: unknown function (ip: 0xffffffffffffffff)
From worker 76: Allocations: 16906533 (Pool: 16903560; Big: 2973); GC: 17
From worker 79:
From worker 79: signal (11): Segmentation fault
From worker 79: in expression starting at none:0
From worker 81:
From worker 81: signal (11): Segmentation fault
From worker 81: in expression starting at none:0
From worker 57:
From worker 57: signal (11): Segmentation fault
From worker 57: in expression starting at none:0
From worker 57: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
From worker 57: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
From worker 81: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
From worker 81: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
From worker 81: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 57: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 79: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
From worker 79: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
From worker 79: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
From worker 79: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 78:
From worker 78: signal (11): Segmentation fault
From worker 78: in expression starting at none:0
From worker 57: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 81: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 79: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 78: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
From worker 78: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
From worker 78: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
From worker 78: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 57: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 81: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 79: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 78: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 57: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 81: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 67:
From worker 67: signal (11): Segmentation fault
From worker 67: in expression starting at none:0
From worker 79: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 79: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 78: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 57: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 67: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
From worker 67: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
From worker 67: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
From worker 67: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
From worker 67: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 67: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 81: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 79: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 78: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 78: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 57: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 81: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 67: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 57: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 78: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 79: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 79: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 55:
From worker 55: signal (11): Segmentation fault
From worker 55: in expression starting at none:0
From worker 81: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 57: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 78: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 78: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 67: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 55: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
From worker 55: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
From worker 55: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
From worker 55: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
From worker 55: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 55: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 79: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 81: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 57: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 78: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 57: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
From worker 57: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 67: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 67: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 67: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 55: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 57: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
From worker 57: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
From worker 57: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
From worker 79: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 79: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 78: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 78: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 81: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 81: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
From worker 81: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 55: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 67: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 57: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 81: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
From worker 81: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
From worker 81: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
From worker 79: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 79: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
From worker 79: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
From worker 79: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 78: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 78: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
From worker 78: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
From worker 78: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 55: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 55: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 55: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 57: process_tcp_streams at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 67: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 57: #101 at ./task.jl:333
From worker 79: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
From worker 79: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
From worker 67: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 67: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 79: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
From worker 78: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
From worker 78: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
From worker 78: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
From worker 57: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
From worker 57: start_task at /buildworker/worker/package_linux64/build/src/task.c:659
From worker 57: unknown function (ip: 0xffffffffffffffff)
From worker 57: Allocations: 16904401 (Pool: 16901426; Big: 2975); GC: 18
From worker 81: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 55: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 67: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 78: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 79: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 81: process_tcp_streams at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 81: #101 at ./task.jl:333
From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 55: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
From worker 55: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
From worker 55: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
From worker 81: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
From worker 81: start_task at /buildworker/worker/package_linux64/build/src/task.c:659
From worker 81: unknown function (ip: 0xffffffffffffffff)
From worker 81: Allocations: 16906535 (Pool: 16903564; Big: 2971); GC: 18
I’m running these simulations on a multinode cluster. Julia is started by passing the node list with the --machine-file parameters.