Worker is terminated with "connection reset by peer" after activating environment and using packages

New to Julia
,
1

I’m working on a Windows laptop, running Julia in VSCode. I run the following code to set up distributed processors and packages:

using Distributed 
addprocs(3)
@everywhere path = "C:/Users/User/Documents/bounds"

@everywhere using Pkg 
@everywhere Pkg.activate(path)
using BSON 
@everywhere using JuMP 
@everywhere using Ipopt

I did quite a few runs with no problems, but the last few times I have tried to run I get an error with the notification that one or more of the workers have terminated due to “unhandled task error/connection reset by peer.” I read somewhere that this could be caused by memory overflow in one of the processors, but I get the error even if I don’t run any code after the above which simply activates packages.

The error printout is shown below:

PS C:\Users\User\Documents\bounds> julia .\src\mp_test.jl
  Activating project at `C:\Users\User\Documents\bounds`
      From worker 4:      Activating project at `C:\Users\User\Documents\bounds`
      From worker 2:      Activating project at `C:\Users\User\Documents\bounds`
      From worker 3:      Activating project at `C:\Users\User\Documents\bounds`
Worker 3 terminated.
Worker 4 terminated.Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base .\stream.jl:410
  [2] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base .\stream.jl:947
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base .\stream.jl:953
  [4] unsafe_read
    @ .\io.jl:759 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base .\io.jl:758
  [6] read!
    @ .\io.jl:760 [inlined]
  [7] deserialize_hdr_raw
    @ C:\Users\User\AppData\Local\Programs\Julia-1.8.2\share\julia\stdlib\v1.8\Distributed\src\messages.jl:167 [inlined] 
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed C:\Users\User\AppData\Local\Programs\Julia-1.8.2\share\julia\stdlib\v1.8\Distributed\src\process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed C:\Users\User\AppData\Local\Programs\Julia-1.8.2\share\julia\stdlib\v1.8\Distributed\src\process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed .\task.jl:484
┌ Error: Error during package callback
│   exception =
│    1-element ExceptionStack:
│    ProcessExitedException(2)
│ 
│    ...and 2 more exceptions.
│ 
│    Stacktrace:
│      [1] sync_end(c::Channel{Any})
│        @ Base .\task.jl:436
│      [2] macro expansion
│        @ .\task.jl:455 [inlined]
│      [3] _require_callback(mod::Base.PkgId)
│        @ Distributed C:\Users\User\AppData\Local\Programs\Julia-1.8.2\share\julia\stdlib\v1.8\Distributed\src\Distributed.jl:77
│      [4] #invokelatest#2
│        @ .\essentials.jl:729 [inlined]
│      [5] invokelatest
│        @ .\essentials.jl:726 [inlined]
│      [6] run_package_callbacks(modkey::Base.PkgId)
│        @ Base .\loading.jl:869
│      [7] _require_prelocked(uuidkey::Base.PkgId)
│        @ Base .\loading.jl:1206
│      [8] macro expansion
│        @ .\loading.jl:1180 [inlined]
│      [9] macro expansion
│        @ .\lock.jl:223 [inlined]
│     [10] require(into::Module, mod::Symbol)
│        @ Base .\loading.jl:1144
│     [11] top-level scope
│        @ C:\Users\User\AppData\Local\Programs\Julia-1.8.2\share\julia\stdlib\v1.8\Distributed\src\macros.jl:200        
│     [12] include(mod::Module, _path::String)
│        @ Base .\Base.jl:419
│     [13] exec_options(opts::Base.JLOptions)
│        @ Base .\client.jl:303
│     [14] _start()
│        @ Base .\client.jl:522
└ @ Base loading.jl:874

Worker 2 terminated.Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base .\stream.jl:410
  [2] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base .\stream.jl:947
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base .\stream.jl:953
  [4] unsafe_read
    @ .\io.jl:759 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base .\io.jl:758
  [6] read!
    @ .\io.jl:760 [inlined]
  [7] deserialize_hdr_raw
    @ C:\Users\User\AppData\Local\Programs\Julia-1.8.2\share\julia\stdlib\v1.8\Distributed\src\messages.jl:167 [inlined] 
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed C:\Users\User\AppData\Local\Programs\Julia-1.8.2\share\julia\stdlib\v1.8\Distributed\src\process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed C:\Users\User\AppData\Local\Programs\Julia-1.8.2\share\julia\stdlib\v1.8\Distributed\src\process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed .\task.jl:484

Does anyone know what might be causing this error?

2

Unfortunately, errors like these are hard to track down and resolve without a consistent minimal working example (MWE). If you are able to reproduce this on another system with the same piece of code or the same setup, then that might make it easier to investigate.

Do you happen to know which line of mp_test.jl throws this error, or about how long it takes before it occurs?

3

After restarting my computer, I have not seen the error again. I’m not sure why it happened in the first place, but it seems like it was a one-off thing and it’s working just fine now.

4

Side comment: Since Julia 1.9, workers should inherit the Pkg environment (PR that fixed the issue), so you don’t need to use Pkg.activate on each worker anymore (as long as you are working in the desired environment already on the master, i.e. the active REPL process).

BTW, I saw that you only joined recently, so welcome to the Julia discourse :slight_smile:

2 Likes
5

That’s very good to know. Thank you!