# Problems with addprocs connection to second machine

**URL:** https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060
**Category:** General Usage
**Tags:** question
**Created:** [September 25, 2017, 9:58am UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060 "2017-09-25T09:58:42Z")
**Posts on this page:** 12
**Page:** 1

<div class="post-metadata">

### Author: ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)
#### Post date: [September 25, 2017, 9:58am UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/1 "2017-09-25T09:58:42Z")

</div>

I am having problems connecting to julia on a second machine via ssh. it is odd, because a plain `run` call allows me to connect fine. the `addprocs` call always times out. here is my example:

```julia
julia> versioninfo()
Julia Version 0.5.0
Commit 3c9d753 (2016-09-19 18:14 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, haswell)

julia> run(`ssh rentsScPo "./getshell.sh && julia -e 'println(pwd());println(versioninfo());println(gethostname());exit()'"`)
**************************************************

              WELCOME TO THE 

          SciencesPo Rent server

This system runs:
Ubuntu 16.04.1 LTS

/bin/bash
/home/floswald
Julia Version 0.5.1
Commit 6445c82 (2017-03-05 13:25 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, broadwell)
nothing
scpo-rents

julia> addprocs(["rentsScPo"],dir="/home/floswald",exename="julia")
**************************************************

              WELCOME TO THE 

          SciencesPo Rent server

This system runs:
Ubuntu 16.04.1 LTS

Master process (id 1) could not connect within 60.0 seconds.
exiting.

exiting.
ERROR: connect: connection timed out (ETIMEDOUT)
 in yieldto(::Task, ::ANY) at ./event.jl:136
 in yieldto(::Task, ::ANY) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 in wait() at ./event.jl:169
 in wait(::Condition) at ./event.jl:27
 in wait(::Condition) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 in stream_wait(::TCPSocket, ::Condition, ::Vararg{Condition,N}) at ./stream.jl:44
 in wait_connected(::TCPSocket) at ./stream.jl:265
 in connect at ./stream.jl:960 [inlined]
 in connect_to_worker(::SubString{String}, ::Int16) at ./managers.jl:483
 in connect(::Base.SSHManager, ::Int64, ::WorkerConfig) at ./managers.jl:425
 in create_worker(::Base.SSHManager, ::WorkerConfig) at ./multi.jl:1786
 in setup_launched_worker(::Base.SSHManager, ::WorkerConfig, ::Array{Int64,1}) at ./multi.jl:1733
 in (::Base.##649#653{Base.SSHManager,Array{Int64,1}})() at ./task.jl:360
 in sync_end() at ./task.jl:311
 in macro expansion at ./task.jl:327 [inlined]
 in #addprocs_locked#645(::Array{Any,1}, ::Function, ::Base.SSHManager) at ./multi.jl:1688
 in #addprocs_locked#645(::Array{Any,1}, ::Function, ::Base.SSHManager) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 in (::Base.#kw##addprocs_locked)(::Array{Any,1}, ::Base.#addprocs_locked, ::Base.SSHManager) at ./<missing>:0
 in #addprocs#644(::Array{Any,1}, ::Function, ::Base.SSHManager) at ./multi.jl:1658
 in #addprocs#644(::Array{Any,1}, ::Function, ::Base.SSHManager) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 in (::Base.#kw##addprocs)(::Array{Any,1}, ::Base.#addprocs, ::Base.SSHManager) at ./<missing>:0
 in #addprocs#744(::Bool, ::Cmd, ::Int64, ::Array{Any,1}, ::Function, ::Array{String,1}) at ./managers.jl:112
 in (::Base.#kw##addprocs)(::Array{Any,1}, ::Base.#addprocs, ::Array{String,1}) at ./<missing>:0

```

the same thing happens if i specify the full path to the executable:

```julia
addprocs(["rentsScPo"],dir="/home/floswald",exename="/home/floswald/apps/julia-0.5/bin/julia")

```

---

<div class="post-metadata">

### Author: ![pszufe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pszufe/32/23452_2.png) [@pszufe](https://discourse.julialang.org/u/pszufe)
#### Post date: [September 25, 2017, 1:32pm UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/2 "2017-09-25T13:32:04Z")

</div>

Dear @floswald,  
In order for Julia parallel clusters to work, two following conditions need to be met:

1. The cluster nodes must be able to connect to each other via SSH protocol (TCP/IP port 22)
2. A passwordless SSH needs to be configured between cluster nodes.

The first point is purely a network configuration issue.

In order to setup passwordless SSH on master node type (assuming Ubuntu Linux).

```bash
KEY_NAME=mykey
KEY_FILE=~/.ssh/$KEY_NAME
ssh-keygen -P "" -t rsa -f $KEY_FILE
printf "\nUser ubuntu\nPubKeyAuthentication yes\nStrictHostKeyChecking no\nIdentityFile $KEY_FILE\n" >> ~/.ssh/config		

```

On slave nodes:

```bash
KEY_NAME=mykey
KEY_FILE=~/.ssh/$KEY_NAME
# this key should be copied from master
printf "User ubuntu\nPubKeyAuthentication yes\nIdentityFile ${KEY_FILE}\nStrictHostKeyChecking no" > ~/.ssh/config

```

Additionally, you need to copy the last line (containing information about the above public key) in the `~/.ssh/authorized_keys` file from master to slave.

Hope that helps.  
Przemek.

P.S.  
Since many people have exactly the same problem, I have developed a tool for quick deploying Julia parallel clusters (currently on Amazon Web Services cloud, other cloud vendors planned) - it comes with a detailed tutorial on how to setup Julia parallel on AWS. Have a look at [https://github.com/pszufe/KissCluster](https://github.com/pszufe/KissCluster).

---

<div class="post-metadata">

### Author: ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)
#### Post date: [September 25, 2017, 1:37pm UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/3 "2017-09-25T13:37:12Z")

</div>

Hi!  
I don’t think you looked at my example long enough. Both of those conditions are clearly met, which is evident from my first call actually working. Any idea why that works, but addprocs not?

---

<div class="post-metadata">

### Author: ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)
#### Post date: [September 25, 2017, 2:58pm UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/4 "2017-09-25T14:58:24Z")

</div>

i like your toolbox though!

---

<div class="post-metadata">

### Author: ![pszufe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pszufe/32/23452_2.png) [@pszufe](https://discourse.julialang.org/u/pszufe)
#### Post date: [September 25, 2017, 4:36pm UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/5 "2017-09-25T16:36:39Z")

</div>

Hi floswald,

You get the error:  
`ERROR: connect: connection timed out (ETIMEDOUT)`  
This seems to be the first point from my list (network connectivity problem)

You try to connect to `rentsScPo` machine and cannot. Maybe try the bash command (assuming it is ubuntu) and post the results:

```julia
ssh ubuntu@rentsScPo

```

If you get a similar result to Julia it means that either:

1. The server name `rentsScPo` does not get properly resolved to an IP address
2. The server name `rentsScPo` is properly resolved but the target host can not be found
3. A firewall on `rentsScPo` is not open on `22` port for SSH protocol.

Hope that helps.

---

<div class="post-metadata">

### Author: ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)
#### Post date: [September 25, 2017, 5:23pm UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/6 "2017-09-25T17:23:15Z")

</div>

i told you ssh works. i can even start a julia process on the remote. please look back at my example!

> [@floswald](#):
>
> julia\> run(`ssh rentsScPo "./getshell.sh && julia -e 'println(pwd());println(versioninfo());println(gethostname());exit()'"`)

---

<div class="post-metadata">

### Author: ![avik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avik/32/17_2.png) [@avik](https://discourse.julialang.org/u/avik)
#### Post date: [September 25, 2017, 6:43pm UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/7 "2017-09-25T18:43:05Z")

</div>

Do you have a firewall on the master? The master connects to the worker via ssh to start the julia process, but then the worker julia process connects to the master process via a TCP socket.

---

<div class="post-metadata">

### Author: ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)
#### Post date: [September 25, 2017, 8:15pm UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/8 "2017-09-25T20:15:46Z")

</div>

ha. good point! I’ve got no idea. I’ll check tomorrow. what do I need to verify here? being able to SSH from worker to master is not the right thing? these are just 2 workstations on my company network.

thanks!

---

<div class="post-metadata">

### Author: ![pszufe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pszufe/32/23452_2.png) [@pszufe](https://discourse.julialang.org/u/pszufe)
#### Post date: [September 25, 2017, 8:28pm UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/9 "2017-09-25T20:28:03Z")

</div>

And this connection mentioned by avik must be configured two-ways (both master and slave must have identical passwordless configurations…

Still I would not neglect some ugly issue with name resolving… maybe try with IP addresses instead of names.

I have never used _SciencesPo Rent server_ mentioned in your logs, but some cloud providers (e.g. AWS) offer both “public” and “private” IP addresses for instances. In that case you want to use the private IP to configure your cluster.

---

<div class="post-metadata">

### Author: ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)
#### Post date: [September 25, 2017, 8:33pm UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/10 "2017-09-25T20:33:55Z")

</div>

of course, that’s it! i didn’t think about the connection back from the worker. thanks. will try tomorrow!  
SciencesPo Rent server is just the fancy name I gave the second workstation under my desk. 🙂

cheers guys

---

<div class="post-metadata">

### Author: ![pszufe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pszufe/32/23452_2.png) [@pszufe](https://discourse.julialang.org/u/pszufe)
#### Post date: [September 30, 2017, 7:14pm UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/11 "2017-09-30T19:14:24Z")

</div>

@floswald just out of curiosity - did it work in the end?

---

<div class="post-metadata">

### Author: ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)
#### Post date: [September 30, 2017, 7:36pm UTC](https://discourse.julialang.org/t/problems-with-addprocs-connection-to-second-machine/6060/12 "2017-09-30T19:36:27Z")

</div>

Arg I hit a wall. One computer is on a wired connection and the other on wifi. I can ssh from wifi to wired but other way round says route not found. Seems to be a network issue at my company. But I’m sure your advice would have worked!
