# "ERROR: LoadError: TaskFailedException: IOError: stream is closed or unusable" with @distributed on multinode cluster

**URL:** https://discourse.julialang.org/t/error-loaderror-taskfailedexception-ioerror-stream-is-closed-or-unusable-with-distributed-on-multinode-cluster/34768
**Category:** Julia at Scale
**Tags:** cluster
**Created:** [February 17, 2020, 6:39pm UTC](https://discourse.julialang.org/t/error-loaderror-taskfailedexception-ioerror-stream-is-closed-or-unusable-with-distributed-on-multinode-cluster/34768 "2020-02-17T18:39:00Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![teored90](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/teored90/32/5142_2.png) [@teored90](https://discourse.julialang.org/u/teored90)
#### Post date: [February 17, 2020, 6:39pm UTC](https://discourse.julialang.org/t/error-loaderror-taskfailedexception-ioerror-stream-is-closed-or-unusable-with-distributed-on-multinode-cluster/34768/1 "2020-02-17T18:39:00Z")

</div>

I have some code that evaluates trajectories of a stochastic master equation in parallel, using a `@distributed for` with reduction for evaluating average quantities. The code is available [here](https://github.com/matteoacrossi/ContinuousMeasurementFI/tree/dicke/src).

Some jobs fail with the following error, which I can’t understand. Can anyone provide some insight or possible ideas on how to diagnose it?

```julia
ERROR: LoadError: TaskFailedException:
IOError: stream is closed or unusable
Stacktrace:
 [1] uv_write_async(::Sockets.TCPSocket, ::Ptr{UInt8}, ::UInt64) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
 [2] uv_write(::Sockets.TCPSocket, ::Ptr{UInt8}, ::UInt64) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
 [3] unsafe_write(::Sockets.TCPSocket, ::Ptr{UInt8}, ::UInt64) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
 [4] unsafe_write at ./io.jl:593 [inlined]
 [5] macro expansion at ./gcutils.jl:91 [inlined]
 [6] write at ./io.jl:616 [inlined]
 [7] serialize_array_data at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:246 [inlined]
 [8] serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Array{Complex{Float64},2}) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:263
 [9] serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Array{Array{Complex{Float64},2},1}) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:268
 [10] serialize_any(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:629
 [11] serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:608
 [12] serialize_any(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:629
 [13] serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:608
 [14] serialize_any(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:629
 [15] serialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:608 [inlined]
 [16] serialize_msg(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Distributed.CallMsg{:call_fetch}) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:90
 [17] #invokelatest#1 at ./essentials.jl:709 [inlined]
 [18] invokelatest at ./essentials.jl:708 [inlined]
 [19] send_msg_(::Distributed.Worker, ::Distributed.MsgHeader, ::Distributed.CallMsg{:call_fetch}, ::Bool) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:185
 [20] send_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:134 [inlined]
 [21] #remotecall_fetch#145(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_fetch), ::Function, ::Distributed.Worker, ::Function, ::Vararg{Any,N} where N) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:385
 [22] remotecall_fetch(::Function, ::Distributed.Worker, ::Function, ::Vararg{Any,N} where N) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:382
 [23] #remotecall_fetch#148 at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:417 [inlined]
 [24] remotecall_fetch at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:417 [inlined]
 [25] (::Distributed.var"#159#160"{typeof(+),ContinuousMeasurementFI.var"#15#19"{Float64,Float64,TimerOutputs.TimerOutput,Int64,Float64,ContinuousMeasurementFI.var"#dW#16"{Float64},RemoteChannel{Channel{Bool}}},UnitRange{Int64},Array{UnitRange{Int64},1},Int64,Int64})() at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/macros.jl:270
Stacktrace:
 [1] wait at ./task.jl:251 [inlined]
 [2] fetch at ./task.jl:266 [inlined]
 [3] iterate at ./generator.jl:47 [inlined]
 [4] collect_to!(::Array{Array{Float64,2},1}, ::Base.Generator{Array{Task,1},typeof(fetch)}, ::Int64, ::Int64) at ./array.jl:667
 [5] collect_to_with_first!(::Array{Array{Float64,2},1}, ::Array{Float64,2}, ::Base.Generator{Array{Task,1},typeof(fetch)}, ::Int64) at ./array.jl:646
 [6] collect(::Base.Generator{Array{Task,1},typeof(fetch)}) at ./array.jl:627
 [7] preduce(::Function, ::Function, ::UnitRange{Int64}) at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/macros.jl:274
 [8] macro expansion at /home/matros/.julia/packages/ProgressMeter/g1lse/src/ProgressMeter.jl:595 [inlined]
 [9] macro expansion at /home/matros/.julia/packages/ContinuousMeasurementFI/tmNbJ/src/Eff_QFI_HD_Dicke.jl:135 [inlined]
 [10] macro expansion at /home/matros/.julia/packages/TimerOutputs/7Id5J/src/TimerOutput.jl:214 [inlined]
 [11] #Eff_QFI_HD_Dicke#12(::Float64, ::Float64, ::Float64, ::Float64, ::Int64, ::TimerOutputs.TimerOutput, ::typeof(Eff_QFI_HD_Dicke), ::Int64, ::Int64, ::Float64, ::Float64) at /home/matros/.julia/packages/ContinuousMeasurementFI/tmNbJ/src/Eff_QFI_HD_Dicke.jl:134
 [12] (::ContinuousMeasurementFI.var"#kw##Eff_QFI_HD_Dicke")(::NamedTuple{(:κ, :κcoll, :ω, :η, :outpoints),Tuple{Float64,Float64,Float64,Float64,Int64}}, ::typeof(Eff_QFI_HD_Dicke), ::Int64, ::Int64, ::Float64, ::Float64) at ./none:0
 [13] top-level scope at /home/matros/continuous_measurement_dicke/sims_20200217/simulateTama.jl:28
 [14] include_relative(::Module, ::String) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
 [15] include(::Module, ::String) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
 [16] exec_options(::Base.JLOptions) at /home/matros/julia-1.3.0/lib/julia/sys.so:?
 [17] _start() at /home/matros/julia-1.3.0/lib/julia/sys.so:?
in expression starting at /home/matros/continuous_measurement_dicke/sims_20200217/simulateTama.jl:28
ARGS = ["50", "5000", "1.0", "0.00005", "1.0", "1.0", "0.01", "1.0", "200"]
ContinuousMeasurementFI.SuperOperator{Complex{Float64},Int64}
ContinuousMeasurementFI.SuperOperator{Complex{Float64},Int64}
      From worker 73:
      From worker 73: signal (11): Segmentation fault
      From worker 73: in expression starting at none:0
      From worker 73: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
      From worker 73: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
      From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
      From worker 73: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
      From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 73: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 76:
      From worker 76: signal (11): Segmentation fault
      From worker 76: in expression starting at none:0
      From worker 73: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 76: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
      From worker 76: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
      From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
      From worker 76: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
      From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 73: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 76: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 73: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 73: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 76: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 73: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 76: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 73: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 76: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 73: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 73: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
      From worker 73: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
      From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 76: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 73: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
      From worker 73: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
      From worker 73: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
      From worker 76: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 73: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 76: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 73: process_tcp_streams at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 73: #101 at ./task.jl:333
      From worker 73: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 73: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 73: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
      From worker 73: start_task at /buildworker/worker/package_linux64/build/src/task.c:659
      From worker 73: unknown function (ip: 0xffffffffffffffff)
      From worker 73: Allocations: 16906524 (Pool: 16903554; Big: 2970); GC: 18
      From worker 76: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 76: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
      From worker 76: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
      From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 76: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
      From worker 76: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
      From worker 76: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
      From worker 76: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 76: process_tcp_streams at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 76: #101 at ./task.jl:333
      From worker 76: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 76: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 76: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
      From worker 76: start_task at /buildworker/worker/package_linux64/build/src/task.c:659
      From worker 76: unknown function (ip: 0xffffffffffffffff)
      From worker 76: Allocations: 16906533 (Pool: 16903560; Big: 2973); GC: 17
      From worker 79:
      From worker 79: signal (11): Segmentation fault
      From worker 79: in expression starting at none:0
      From worker 81:
      From worker 81: signal (11): Segmentation fault
      From worker 81: in expression starting at none:0
      From worker 57:
      From worker 57: signal (11): Segmentation fault
      From worker 57: in expression starting at none:0
      From worker 57: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
      From worker 57: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
      From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
      From worker 81: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
      From worker 81: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
      From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
      From worker 81: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
      From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 57: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
      From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 79: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
      From worker 79: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
      From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
      From worker 79: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
      From worker 79: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 78:
      From worker 78: signal (11): Segmentation fault
      From worker 78: in expression starting at none:0
      From worker 57: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 81: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 79: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 78: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
      From worker 78: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
      From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
      From worker 78: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
      From worker 78: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 57: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 81: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 79: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 78: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 57: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 81: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 67:
      From worker 67: signal (11): Segmentation fault
      From worker 67: in expression starting at none:0
      From worker 79: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 79: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 78: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 57: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 67: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
      From worker 67: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
      From worker 67: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
      From worker 67: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
      From worker 67: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 67: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 81: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 79: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 78: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 78: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 57: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 81: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 67: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 57: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 78: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 79: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 79: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 55:
      From worker 55: signal (11): Segmentation fault
      From worker 55: in expression starting at none:0
      From worker 81: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 57: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 78: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 78: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 67: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 55: sig_match_fast at /buildworker/worker/package_linux64/build/src/gf.c:2160 [inlined]
      From worker 55: jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2243 [inlined]
      From worker 55: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2301
      From worker 55: deserialize_fillarray! at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1094
      From worker 55: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 55: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 79: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 81: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 57: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 78: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 57: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
      From worker 57: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
      From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 67: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 67: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 67: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 55: deserialize_array at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 57: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
      From worker 57: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
      From worker 57: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
      From worker 79: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 79: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 78: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 78: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 81: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 81: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
      From worker 81: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
      From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 55: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 67: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 57: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 81: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
      From worker 81: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
      From worker 81: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
      From worker 79: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 79: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
      From worker 79: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
      From worker 79: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 79: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 78: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 78: deserialize at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
      From worker 78: deserialize_msg at /home/matros/julia-1.3.0/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
      From worker 78: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 78: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 55: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 55: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 55: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 57: process_tcp_streams at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 67: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 57: #101 at ./task.jl:333
      From worker 79: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
      From worker 79: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
      From worker 67: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 67: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 57: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 57: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 79: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
      From worker 78: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
      From worker 78: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:627
      From worker 78: jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:665
      From worker 57: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
      From worker 57: start_task at /buildworker/worker/package_linux64/build/src/task.c:659
      From worker 57: unknown function (ip: 0xffffffffffffffff)
      From worker 57: Allocations: 16904401 (Pool: 16901426; Big: 2975); GC: 18
      From worker 81: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 55: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 67: handle_deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 78: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 79: message_handler_loop at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 81: process_tcp_streams at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 81: #101 at ./task.jl:333
      From worker 81: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
      From worker 81: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 55: deserialize at /home/matros/julia-1.3.0/lib/julia/sys.so (unknown line)
      From worker 55: _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
      From worker 55: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
      From worker 81: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
      From worker 81: start_task at /buildworker/worker/package_linux64/build/src/task.c:659
      From worker 81: unknown function (ip: 0xffffffffffffffff)
      From worker 81: Allocations: 16906535 (Pool: 16903564; Big: 2971); GC: 18

```

I’m running these simulations on a multinode cluster. Julia is started by passing the node list with the `--machine-file` parameters.

---

<div class="post-metadata">

### Author: ![vlandau](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vlandau/32/8986_2.png) [@vlandau](https://discourse.julialang.org/u/vlandau)
#### Post date: [March 19, 2020, 4:07pm UTC](https://discourse.julialang.org/t/error-loaderror-taskfailedexception-ioerror-stream-is-closed-or-unusable-with-distributed-on-multinode-cluster/34768/2 "2020-03-19T16:07:35Z")

</div>

Did this ever get resolved? I’ve got a [similar issue](https://github.com/Circuitscape/Omniscape.jl/issues/32) occurring with a Julia package I wrote.

---

<div class="post-metadata">

### Author: ![teored90](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/teored90/32/5142_2.png) [@teored90](https://discourse.julialang.org/u/teored90)
#### Post date: [March 19, 2020, 4:15pm UTC](https://discourse.julialang.org/t/error-loaderror-taskfailedexception-ioerror-stream-is-closed-or-unusable-with-distributed-on-multinode-cluster/34768/3 "2020-03-19T16:15:33Z")

</div>

Nope. I just modified the code to minimize the communication between workers, and I don’t get this error anymore, although I don’t know if the two things are related. I was passing big matrices around, now they are initialized independently by each worker.

---

<div class="post-metadata">

### Author: ![vlandau](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vlandau/32/8986_2.png) [@vlandau](https://discourse.julialang.org/u/vlandau)
#### Post date: [March 19, 2020, 4:22pm UTC](https://discourse.julialang.org/t/error-loaderror-taskfailedexception-ioerror-stream-is-closed-or-unusable-with-distributed-on-multinode-cluster/34768/4 "2020-03-19T16:22:57Z")

</div>

Ah, yep I’m passing big matrices around as well. I’ve switched to multi-threading in the latest unreleased version, so maybe that will help? We’ll see. Thanks for the quick reply!