# Addprocs with SSH and a ProxyJump times out

**URL:** <https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647>\
**Category:** Julia at Scale\
**Tags:** question\
**Created:** [July 22, 2019, 8:11pm UTC](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647 "2019-07-22T20:11:43Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ararslan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ararslan/32/3825_2.png) [@ararslan](https://discourse.julialang.org/u/ararslan)\
**Post date:** [July 22, 2019, 8:11pm UTC](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647/1 "2019-07-22T20:11:43Z")

</div>

In my SSH config file, I have

```nohighlight
Host a
    User me
    HostName a.example.com

Host b
    User me
    HostName b.example.com
    ProxyJump a

```

I want to add `b` as a worker process from my local machine using `addprocs`.

I’m able to add `a` as a worker just fine with `addprocs(["a"])`, but `addprocs(["b"])` hangs for a while then times out:

```julia
ERROR: IOError: connect: connection timed out (ETIMEDOUT)
try_yieldto(::typeof(Base.ensure_rescheduled), ::Base.RefValue{Task}) at ./event.jl:196
wait() at ./event.jl:255
wait(::Condition) at ./event.jl:46
stream_wait(::Sockets.TCPSocket, ::Condition) at ./stream.jl:47
wait_connected(::Sockets.TCPSocket) at ./stream.jl:330
connect at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Sockets/src/Sockets.jl:456 [inlined]
connect_to_worker(::SubString{String}, ::UInt16) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/managers.jl:499
connect(::Distributed.SSHManager, ::Int64, ::WorkerConfig) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/managers.jl:437
create_worker(::Distributed.SSHManager, ::WorkerConfig) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/cluster.jl:501
setup_launched_worker(::Distributed.SSHManager, ::WorkerConfig, ::Array{Int64,1}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/cluster.jl:447
(::getfield(Distributed, Symbol("##47#50")){Distributed.SSHManager,WorkerConfig})() at ./task.jl:259
Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:226
 [2] macro expansion at ./task.jl:245 [inlined]
 [3] #addprocs_locked#44(::Base.Iterators.Pairs{Symbol,Any,NTuple{5,Symbol},NamedTuple{(:tunnel, :sshflags, :max_parallel, :dir, :exename),Tuple{Bool,Cmd,Int64,String,String}}}, ::Function, ::Distributed.SSHManager) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/cluster.jl:401
 [4] #addprocs_locked at ./none:0 [inlined]
 [5] #addprocs#43(::Base.Iterators.Pairs{Symbol,Any,NTuple{5,Symbol},NamedTuple{(:tunnel, :sshflags, :max_parallel, :dir, :exename),Tuple{Bool,Cmd,Int64,String,String}}}, ::Function, ::Distributed.SSHManager) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/cluster.jl:365
 [6] #addprocs at ./none:0 [inlined]
 [7] #addprocs#249 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/managers.jl:118 [inlined]
 [8] (::getfield(Distributed, Symbol("#kw##addprocs")))(::NamedTuple{(:dir, :exename),Tuple{String,String}}, ::typeof(addprocs), ::Array{String,1}) at ./none:0
 [9] top-level scope at none:0

```

Some things to note:

- My RSA key is password protected. I’ve run `ssh-add` on my local machine though.
- The path to the Julia executable differs between my local machine, `a`, and `b`. (I’m using the `exename` keyword argument in `addprocs` and setting it to the path on `a`.)
- I’ve tried with and without `tunnel=true`.

Is there something fundamental I’m missing here? My knowledge of SSH in general, as well as Distributed and `SSHManager`, is lacking.

---

<div class="post-metadata">

**Author:** ![anon43133060](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@anon43133060](https://discourse.julialang.org/u/anon43133060)\
**Post date:** [July 22, 2019, 11:30pm UTC](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647/2 "2019-07-22T23:30:50Z")

</div>

I had the same problem, also with multiplexing ssh connections.

I think I’ve added .hushlogin to the home directory and seemed to fix the problem. Can you `ssh b` manually? can you `ssh -T b true`?

Also `function ssh_tunnel(user, host, bind_addr, port, sshflags)` in `Distributed` has hardcoded sleep value of 60 seconds - was preventing everything else from launching. reducing it to 5 seemed to help speed up `addprocs` and doesn’t seem to affect anything.

---

<div class="post-metadata">

**Author:** ![ararslan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ararslan/32/3825_2.png) [@ararslan](https://discourse.julialang.org/u/ararslan)\
**Post date:** [July 25, 2019, 2:37pm UTC](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647/3 "2019-07-25T14:37:00Z")

</div>

> [@anon43133060](#):
>
> Can you `ssh b` manually?

Yep, no problem at all.

> [@anon43133060](#):
>
> can you `ssh -T b true` ?

I don’t know what that does, but the command executes successfully.

> [@anon43133060](#):
>
> was preventing everything else from launching

So you’re saying you were able to get around this problem by reducing the waiting time for connections?

---

<div class="post-metadata">

**Author:** ![anon43133060](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@anon43133060](https://discourse.julialang.org/u/anon43133060)\
**Post date:** [July 25, 2019, 3:12pm UTC](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647/4 "2019-07-25T15:12:53Z")

</div>

Yes, it is working for me. I’ve added ssh multiplexing:

```julia
Host Worker*
   ControlPath ~/.ssh/controlmasters/%C
   ControlMaster auto
   ControlPersist 10m
   ProxyJump Firewall

```

And make a test ssh connection to each worker, so that the controlmaster connection already exists right before Julia starts. Then there’s no delay in establishing a connection in Julia with `addprocs(workers, tunnel=true, topology=:master_worker, exename="worker.sh")`

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 21, 2021, 10:12am UTC](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647/5 "2021-06-21T10:12:21Z")

</div>

@ararslan Did you get this to work? Unfortunately the suggested approach (control master) doesn’t work for me. A worker process gets started on the node (see htop in the gif below) but on the master I get

```julia
julia> addprocs(["l96"]; params...)
ERROR: TaskFailedException

    nested task error: unable to create SSH tunnel after 100 tries. No free port?

```

![julia_ssh_tunnel](https://global.discourse-cdn.com/julialang/original/3X/5/f/5f2cabcf144632f264c844f3110f73db46152927.gif)

Any ideas what I could do to fix this? I didn’t expect that it would be so hard to start a worker through a ssh tunnel that works just fine from the terminal. ☹

My parameters:

```julia
julia> using Distributed

julia> params = (exename=`nice -19 /home/bauer/bin/julia-1.6.1/bin/julia --project=/home/bauer/JuliaNRWSS21`, dir="/home/bauer", tunnel=true, topology=:master_worker)
(exename = `nice -19 /home/bauer/bin/julia-1.6.1/bin/julia --project=/home/bauer/JuliaNRWSS21`, dir = "/home/bauer", tunnel = true, topology = :master_worker)

julia> addprocs(["l96"]; params...)

```

---

<div class="post-metadata">

**Author:** ![DaveMacMahon](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@DaveMacMahon](https://discourse.julialang.org/u/DaveMacMahon)\
**Post date:** [June 21, 2021, 12:50pm UTC](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647/6 "2021-06-21T12:50:49Z")

</div>

You might need to use the “tunnel” option. I think normally the ash connection is used to make initial contact and start things up, but after that the worker expects to be able to connect directly to others. If you are using ProxyJump I think there is a tunneling option in `addprocs` that forces all traffic through the aah tunnel

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 21, 2021, 12:55pm UTC](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647/7 "2021-06-21T12:55:21Z")

</div>

Not sure what you mean. I’m using `tunnel = true` and `topology = :master_worker` (only the master is connected to the workers). Can you elaborate?

---

<div class="post-metadata">

**Author:** ![DaveMacMahon](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@DaveMacMahon](https://discourse.julialang.org/u/DaveMacMahon)\
**Post date:** [June 22, 2021, 6:21am UTC](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647/8 "2021-06-22T06:21:00Z")

</div>

Ah, sorry, I was replying from my phone and hadn’t seen the other details. The tunnel and topology options were what was suggesting, but it sounds like you’ve tried those already,