# What does this error mean?

**URL:** <https://discourse.julialang.org/t/what-does-this-error-mean/49796>\
**Category:** Julia at Scale\
**Tags:** question\
**Created:** [November 8, 2020, 5:03pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796 "2020-11-08T17:03:00Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![threadedblue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/threadedblue/32/19247_2.png) [@threadedblue](https://discourse.julialang.org/u/threadedblue)\
**Post date:** [November 8, 2020, 5:03pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796/1 "2020-11-08T17:03:00Z")

</div>

I am attempting to use the Distributed package from: macOS to: Ubuntu.  
Julia 1.5.2 on both machines. The remote machine has ssh on port 2222 and not the usual 22. I attempted at one point to provide a bind address of uno:2222, but took it out. I am not clear about its use. I have also tried using an sshflag of -p 2222 like I do when calling ssh from the cl. Once again, I am not clear as to whether I need it.

Here is my most recent attempt and the error in question is below. I am not sure what it is telling me.

**julia\> using Distributed**

**julia\>workervec = [“haz@uno:2222”, 3]**

**2-element Array{Any,1}:**

**“haz@uno:2222”**

**3**

**julia\> addprocs(workervec; dir=“/home/haz”, exename=“/usr/bin/julia”)**

**exception launching on machine 3 : MethodError(Distributed.launch\_on\_machine, (SSHManager(machines=Dict{Any,Any}(3 =\> 1,“haz@uno:2222” =\> 1)), 3, 1, Dict{Symbol,Any}(:lazy =\> true,:tunnel =\> false,:topology =\> :all\_to\_all,:multiplex =\> false,:sshflags =\> `,:max_parallel => 10,:exeflags => `,:enable\_threaded\_blas =\> false,:exename =\> “/usr/bin/julia”,:dir =\> “/home/haz”), WorkerConfig[], Base.GenericCondition{Base.AlwaysLockedST}(Base.InvasiveLinkedList{Task}(Task (runnable) @0x0000000116868010, Task (runnable) @0x0000000116868010), Base.AlwaysLockedST(1))), 0x0000000000006ca4)**\*\*

---

<div class="post-metadata">

**Author:** ![yha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yha/32/3502_2.png) [@yha](https://discourse.julialang.org/u/yha)\
**Post date:** [November 8, 2020, 5:25pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796/2 "2020-11-08T17:25:33Z")

</div>

> [@threadedblue](#):
>
> julia\>workervec = [“haz@uno:2222”, 3]

I think you want `[(“haz@uno:2222”, 3)]` (`addprocs` expects a vector of names or `(name,count)` tuples).

---

<div class="post-metadata">

**Author:** ![threadedblue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/threadedblue/32/19247_2.png) [@threadedblue](https://discourse.julialang.org/u/threadedblue)\
**Post date:** [November 8, 2020, 8:09pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796/3 "2020-11-08T20:09:49Z")

</div>

I tried the tuple. It looks like this error is on the macOS side. I still don’t know how to understand the error.

Here’s what happened:

Julia\> workervec = [(“haz@uno:2222”, 3)]

**1-element Array{Tuple{String,Int64},1}:**

**(“haz@uno:2222”, 3)**

Julia\> addprocs(workervec; dir=“/home/haz”, exename=“/usr/bin/julia”)

**ERROR: TaskFailedException:**

**IOError: connect: connection timed out (ETIMEDOUT)**

**Stacktrace:**

**[1] worker\_from\_id(::Distributed.ProcessGroup, ::Int64) at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:1074**

**[2] worker\_from\_id at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:1071 [inlined]**

**[3] #remote\_do#154 at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:486 [inlined]**

**[4] remote\_do at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:486 [inlined]**

**[5] kill(::Distributed.SSHManager, ::Int64, ::WorkerConfig) at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/managers.jl:603**

**[6] create\_worker(::Distributed.SSHManager, ::WorkerConfig) at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:585**

**[7] setup\_launched\_worker(::Distributed.SSHManager, ::WorkerConfig, ::Array{Int64,1}) at /Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:526**

**[8] (::Distributed.var"#41#44"{Distributed.SSHManager,Array{Int64,1},WorkerConfig})() at ./task.jl:356**

**Stacktrace:**

**[1]** **sync\_end(**  **::Channel{Any}** **)**  **at**  **./task.jl:314**

**[2]**  **macro expansion**  **at**  **./task.jl:333** **[inlined]**

**[3]** **addprocs\_locked(** **::Distributed.SSHManager; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{6,Symbol},NamedTuple{(:tunnel, :multiplex, :sshflags, :max\_parallel, :dir, :exename),Tuple{Bool,Bool,Cmd,Int64,String,String}}}** **)**  **at**  **/Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:480**

**[4]** **addprocs(** **::Distributed.SSHManager; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{6,Symbol},NamedTuple{(:tunnel, :multiplex, :sshflags, :max\_parallel, :dir, :exename),Tuple{Bool,Bool,Cmd,Int64,String,String}}}** **)**  **at**  **/Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:444**

**[5]**  **#addprocs#241**  **at**  **/Users/julia/buildbot/worker/package\_macos64/build/usr/share/julia/stdlib/v1.5/Distributed/src/managers.jl:120** **[inlined]**

**[6] top-level scope at** **REPL[24]:1**

---

<div class="post-metadata">

**Author:** ![yha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yha/32/3502_2.png) [@yha](https://discourse.julialang.org/u/yha)\
**Post date:** [November 9, 2020, 11:22am UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796/4 "2020-11-09T11:22:53Z")

</div>

Is the reason for the 2222 port that you’re trying to connect into WSL on a Windows machine?

---

<div class="post-metadata">

**Author:** ![threadedblue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/threadedblue/32/19247_2.png) [@threadedblue](https://discourse.julialang.org/u/threadedblue)\
**Post date:** [November 9, 2020, 1:04pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796/5 "2020-11-09T13:04:58Z")

</div>

The reason for 2222 is because I am trying to connect from macOS host to a VBox guest running Ubuntu. I have an rsa public key setup and can ssh into the guest passwordless.

addprocs(3), for example, works on both host and guest without error.

My problem is I can’t figure out how to trouble shoot because the error message (above), I can’t interpret.

---

<div class="post-metadata">

**Author:** ![threadedblue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/threadedblue/32/19247_2.png) [@threadedblue](https://discourse.julialang.org/u/threadedblue)\
**Post date:** [November 9, 2020, 1:45pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796/6 "2020-11-09T13:45:32Z")

</div>

One thing I have noticed is the path: **/Users/Julia/buildbot** is referred to repeatedly in the error. I can find nothing on my host that matches. Do we have a problem with the macOS version of Julia?

---

<div class="post-metadata">

**Author:** ![yha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yha/32/3502_2.png) [@yha](https://discourse.julialang.org/u/yha)\
**Post date:** [November 9, 2020, 1:55pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796/7 "2020-11-09T13:55:26Z")

</div>

I don’t think the paths in the error stacktrace are related to the problem, since I often see `buildbot` paths for errors inside the standard library (running on Windows).  
I’ve had similar problems trying to run remote workers on WSL, which seem to be due to it having a different ip address than the host machine (merely forwarding the ssh port was not enough). Maybe this is the case here?

---

<div class="post-metadata">

**Author:** ![tisztamo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tisztamo/32/16200_2.png) [@tisztamo](https://discourse.julialang.org/u/tisztamo)\
**Post date:** [November 9, 2020, 6:25pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796/8 "2020-11-09T18:25:33Z")

</div>

Although the error may not be related to this, and also a quick look at the docs was not enough to find it stated directly, but I am almost sure that cross-OS distribution is not supported. I think this because it depends on [Serialization · The Julia Language](https://docs.julialang.org/en/v1/stdlib/Serialization/#Serialization.serialize)

---

<div class="post-metadata">

**Author:** ![threadedblue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/threadedblue/32/19247_2.png) [@threadedblue](https://discourse.julialang.org/u/threadedblue)\
**Post date:** [November 10, 2020, 3:23pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796/9 "2020-11-10T15:23:55Z")

</div>

By now, I suspect tistzamo might well be right. Distribution only works between like OSes. I shall test this out and let you all know. If true, it means I’ll have to launch any kind of distributed process by another means. I am thinking of having it as a micro service. A Julia http server would be nice. Failing that it’ll be more polyglot.

---

<div class="post-metadata">

**Author:** ![threadedblue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/threadedblue/32/19247_2.png) [@threadedblue](https://discourse.julialang.org/u/threadedblue)\
**Post date:** [November 11, 2020, 11:25pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796/10 "2020-11-11T23:25:05Z")

</div>

I said I would test using two like OSes. I did. I tested from Ubuntu to Ubuntu. Both machines have Julia 1.5.2. They are called uno and haz00 respectively. I can sign in from uno to haz00 using ssh with rsa public key–no problem.

When I run the following–note the -v flag–all seems to go well. The debug output indicates a connection. After that, as you can see below, some kind of error occurs and the remote is inaccessible.

I am still not sure what is going wrong.

**julia\>** **addprocs(workervec; sshflags = `-v`, dir=“/home/gcr”, exename=`julia`)**

**OpenSSH\_7.9p1 Ubuntu-10, OpenSSL 1.1.1b 26 Feb 2019**

**debug1: Reading configuration data /etc/ssh/ssh\_config**

\*\*debug1: /etc/ssh/ssh\_config line 19: Applying options for \*\*\*

**debug1: Connecting to haz00 [10.185.3.197] port 22.**

**debug1: Connection established.**

**debug1: identity file /home/haz/.ssh/id\_rsa type 0**

**debug1: identity file /home/haz/.ssh/id\_rsa-cert type -1**

**debug1: identity file /home/haz/.ssh/id\_dsa type -1**

**debug1: identity file /home/haz/.ssh/id\_dsa-cert type -1**

**debug1: identity file /home/haz/.ssh/id\_ecdsa type -1**

**debug1: identity file /home/haz/.ssh/id\_ecdsa-cert type -1**

**debug1: identity file /home/haz/.ssh/id\_ed25519 type -1**

**debug1: identity file /home/haz/.ssh/id\_ed25519-cert type -1**

**debug1: identity file /home/haz/.ssh/id\_xmss type -1**

**debug1: identity file /home/haz/.ssh/id\_xmss-cert type -1**

**debug1: Local version string SSH-2.0-OpenSSH\_7.9p1 Ubuntu-10**

**debug1: Remote protocol version 2.0, remote software version OpenSSH\_7.6p1 Ubuntu-4ubuntu0.3**

_debug1: match: OpenSSH\_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH\_7.0,OpenSSH\_7.1_,OpenSSH\_7.2\*,OpenSSH\_7.3\*,OpenSSH\_7.4\*,OpenSSH\_7.5\*,OpenSSH\_7.6\*,OpenSSH\_7.7\* compat 0x04000002\*\*

**debug1: Authenticating to haz00:22 as ‘gcr’**

**debug1: SSH2\_MSG\_KEXINIT sent**

**debug1: SSH2\_MSG\_KEXINIT received**

**debug1: kex: algorithm: curve25519-sha256**

**debug1: kex: host key algorithm: ecdsa-sha2-nistp256**

**debug1: kex: server-\>client cipher: [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com) MAC: compression: none**

**debug1: kex: client-\>server cipher: [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com) MAC: compression: none**

**debug1: expecting SSH2\_MSG\_KEX\_ECDH\_REPLY**

**debug1: Server host key: ecdsa-sha2-nistp256 SHA256:S98DQb8FCMuzIi1cjMBn8lvX/J5bRBVVj7QM9AMWL7I**

**debug1: Host ‘haz00’ is known and matches the ECDSA host key.**

**debug1: Found key in /home/haz/.ssh/known\_hosts:2**

**debug1: rekey after 134217728 blocks**

**debug1: SSH2\_MSG\_NEWKEYS sent**

**debug1: expecting SSH2\_MSG\_NEWKEYS**

**debug1: SSH2\_MSG\_NEWKEYS received**

**debug1: rekey after 134217728 blocks**

**debug1: Will attempt key: /home/haz/.ssh/id\_rsa RSA SHA256:GoGcdDxfYU6S1GLEHNPjaaaus0Fbhg84Rupe4kg2h5c**

**debug1: Will attempt key: /home/haz/.ssh/id\_dsa**

**debug1: Will attempt key: /home/haz/.ssh/id\_ecdsa**

**debug1: Will attempt key: /home/haz/.ssh/id\_ed25519**

**debug1: Will attempt key: /home/haz/.ssh/id\_xmss**

**debug1: SSH2\_MSG\_EXT\_INFO received**

**debug1: kex\_input\_ext\_info: server-sig-algs=\<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521\>**

**debug1: SSH2\_MSG\_SERVICE\_ACCEPT received**

**debug1: Authentications that can continue: publickey**

**debug1: Next authentication method: publickey**

**debug1: Offering public key: /home/haz/.ssh/id\_rsa RSA SHA256:GoGcdDxfYU6S1GLEHNPjaaaus0Fbhg84Rupe4kg2h5c**

**debug1: Server accepts key: /home/haz/.ssh/id\_rsa RSA SHA256:GoGcdDxfYU6S1GLEHNPjaaaus0Fbhg84Rupe4kg2h5c**

**debug1: Authentication succeeded (publickey).**

**Authenticated to haz00 ([10.185.3.197]:22).**

**debug1: channel 0: new [client-session]**

**debug1: Requesting [no-more-sessions@openssh.com](mailto:no-more-sessions@openssh.com)**

**debug1: Entering interactive session.**

**debug1: pledge: network**

**debug1: client\_input\_global\_request: rtype [hostkeys-00@openssh.com](mailto:hostkeys-00@openssh.com) want\_reply 0**

**debug1: Sending environment.**

**debug1: Sending env LANG = en\_US.UTF-8**

**debug1: Sending command: sh -l -c 'cd – /home/gcr**

**julia --worker’**

**ERROR: TaskFailedException:**

**IOError: connect: connection refused (ECONNREFUSED)**

**Stacktrace:**

**[1] worker\_from\_id(::Distributed.ProcessGroup, ::Int64) at /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:1074**

**[2] worker\_from\_id at /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:1071 [inlined]**

**[3] #remote\_do#154 at /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:486 [inlined]**

**[4] remote\_do at /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:486 [inlined]**

**[5] kill(::Distributed.SSHManager, ::Int64, ::WorkerConfig) at /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/managers.jl:603**

**[6] create\_worker(::Distributed.SSHManager, ::WorkerConfig) at /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:585**

**[7] setup\_launched\_worker(::Distributed.SSHManager, ::WorkerConfig, ::Array{Int64,1}) at /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:526**

**[8] (::Distributed.var"#41#44"{Distributed.SSHManager,Array{Int64,1},WorkerConfig})() at ./task.jl:356**

**Stacktrace:**

**[1]** **sync\_end(**  **::Channel{Any}** **)**  **at**  **./task.jl:314**

**[2]**  **macro expansion**  **at**  **./task.jl:333** **[inlined]**

**[3]** **addprocs\_locked(** **::Distributed.SSHManager; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{6,Symbol},NamedTuple{(:tunnel, :multiplex, :sshflags, :max\_parallel, :dir, :exename),Tuple{Bool,Bool,Cmd,Int64,String,Cmd}}}** **)**  **at**  **/buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:480**

**[4]** **addprocs(** **::Distributed.SSHManager; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{6,Symbol},NamedTuple{(:tunnel, :multiplex, :sshflags, :max\_parallel, :dir, :exename),Tuple{Bool,Bool,Cmd,Int64,String,Cmd}}}** **)**  **at**  **/buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:444**

**[5]**  **#addprocs#241**  **at**  **/buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/managers.jl:120** **[inlined]**

**[6] top-level scope at** **REPL[31]:1**

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 21, 2021, 3:52pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796/11 "2021-06-21T15:52:33Z")

</div>

I’ve used cross-OS distribution many times before (macOS → Windows, macOS → Linux, Windows → Linux). Last time I checked was in a workshop I gave in March, see [here](https://nbviewer.jupyter.org/github/crstnbr/JuliaNRW21/blob/master/Day3/1_parallel_computing.ipynb) (macOS → Linux).

Having said that, I’m also currently trying to start workers on a ubuntu VM from macOS and it doesn’t work for me. Even more surprising: My old examples don’t work anymore (again, they used to work fine in March). Don’t know what’s going on.