# Can't add distributed workers per ssh: connection timed out

**URL:** <https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328>\
**Category:** General Usage\
**Tags:** distributed, ssh\
**Created:** [June 21, 2021, 8:21pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328 "2021-06-21T20:21:21Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 21, 2021, 8:21pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328/1 "2021-06-21T20:21:21Z")

</div>

First, let me say that I’m super confused since adding workers per ssh has worked fine for me for many years. Even on the same machine everything was fine a few months back.

I’m trying to start a Julia worker on a thinkpad running ubuntu from my MacBook Pro running macOS (Julia 1.6.1 on both but I tested 1.5 as well). Although the worker process seems to get spawned correctly (see the `htop` screenshot from the target machine below), on my MacBook (the master) I get

```julia
julia> addprocs([("cbthinkpad", 1)]; exename=`/home/crstnbr/.local/bin/julia`, dir="/home/crstnbr", topology=:master_worker)
ERROR: TaskFailedException

    nested task error: IOError: connect: connection timed out (ETIMEDOUT)
    Stacktrace:
     [1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:1082
     [2] worker_from_id
       @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:1079 [inlined]
     [3] #remote_do#154
       @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:486 [inlined]
     [4] remote_do
       @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:486 [inlined]
     [5] kill(manager::Distributed.SSHManager, pid::Int64, config::WorkerConfig)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/managers.jl:680
     [6] create_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:593
     [7] setup_launched_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:534
     [8] (::Distributed.var"#41#44"{Distributed.SSHManager, Vector{Int64}, WorkerConfig})()
       @ Distributed ./task.jl:411

    caused by: IOError: connect: connection timed out (ETIMEDOUT)
    Stacktrace:
     [1] wait_connected(x::Sockets.TCPSocket)
       @ Sockets /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Sockets/src/Sockets.jl:532
     [2] connect
       @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Sockets/src/Sockets.jl:567 [inlined]
     [3] connect_to_worker(host::SubString{String}, port::Int64)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/managers.jl:639
     [4] connect(manager::Distributed.SSHManager, pid::Int64, config::WorkerConfig)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/managers.jl:566
     [5] create_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:589
     [6] setup_launched_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:534
     [7] (::Distributed.var"#41#44"{Distributed.SSHManager, Vector{Int64}, WorkerConfig})()
       @ Distributed ./task.jl:411
Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:369
 [2] macro expansion
   @ ./task.jl:388 [inlined]
 [3] addprocs_locked(manager::Distributed.SSHManager; kwargs::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:exename, :dir, :topology), Tuple{Cmd, String, Symbol}}})
   @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:480
 [4] addprocs(manager::Distributed.SSHManager; kwargs::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:exename, :dir, :topology), Tuple{Cmd, String, Symbol}}})
   @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:444
 [5] #addprocs#237
   @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/managers.jl:143 [inlined]
 [6] top-level scope
   @ REPL[2]:100:

```

 ![Screenshot 2021-06-21 at 22.01.02](https://global.discourse-cdn.com/julialang/original/3X/0/6/068d97691604356997073fde5b3341a7d3729a19.png)

Initially, I thought it might be an issue with the thinkpad. But after having tried to spawn workers on two other machines (another linux desktop and a linux VM) for both of which I saw time outs as well I’m now thinking it must be something about the master, i.e. my MacBook. I just don’t have a clue, in particular given that this was working just fine in March… I found [Properly connect Julia processes in multiple machines with Sockets](https://discourse.julialang.org/t/properly-connect-julia-processes-in-multiple-machines-with-sockets/38487), so maybe Julia is listening on the wrong interface? I tried disabling everything except my WiFi but still see the same error. In [Addprocs with SSH and a ProxyJump times out](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647) a time out is seen when using a proxy jump. As described [here](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647/5), trying to spawn workers on a different linux machine which I can only access via a proxy using the “control master” approach mentioned in this thread I could trade the time out error for the following error:

```julia
ERROR: TaskFailedException

    nested task error: unable to create SSH tunnel after 100 tries. No free port?

```

Finally, [What does this error mean?](https://discourse.julialang.org/t/what-does-this-error-mean/49796) also sees a time out when trying to start a distributed worker. There, someone suggested that it might be a cross-OS issue. But given that this has always worked in the past (and I tried 1.5 where I know for sure that it worked in March) this seems unlikely to me.

Anyone has an idea what’s going on here?!

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 21, 2021, 8:28pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328/2 "2021-06-21T20:28:58Z")

</div>

As suggested by @ppalmes over on Slack, I tried passing `sshflags=`-vvv` ` to get a bit more debug information. Can anyone with more ssh knowledge decipher this?

> **Debug information**
>
> ```julia
> julia> addprocs([("cbthinkpad", 1)]; exename=`/home/crstnbr/.local/bin/julia`, dir="/home/crstnbr", topology=:master_worker, sshflags=`-vvv`)
> OpenSSH_8.1p1, LibreSSL 2.7.3
> debug1: Reading configuration data /Users/crstnbr/.ssh/config
> debug1: /Users/crstnbr/.ssh/config line 2: Applying options for *
> debug1: /Users/crstnbr/.ssh/config line 9: Applying options for cbthinkpad
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug1: /etc/ssh/ssh_config line 47: Applying options for *
> debug1: Connecting to cbthinkpad port 22.
> debug1: Connection established.
> debug1: identity file /Users/crstnbr/.ssh/id_rsa type 0
> debug1: identity file /Users/crstnbr/.ssh/id_rsa-cert type -1
> debug1: identity file /Users/crstnbr/.ssh/id_dsa type -1
> debug1: identity file /Users/crstnbr/.ssh/id_dsa-cert type -1
> debug1: identity file /Users/crstnbr/.ssh/id_ecdsa type -1
> debug1: identity file /Users/crstnbr/.ssh/id_ecdsa-cert type -1
> debug1: identity file /Users/crstnbr/.ssh/id_ed25519 type -1
> debug1: identity file /Users/crstnbr/.ssh/id_ed25519-cert type -1
> debug1: identity file /Users/crstnbr/.ssh/id_xmss type -1
> debug1: identity file /Users/crstnbr/.ssh/id_xmss-cert type -1
> debug1: Local version string SSH-2.0-OpenSSH_8.1
> debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
> debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000
> debug3: fd 6 is O_NONBLOCK
> debug1: Authenticating to cbthinkpad:22 as 'crstnbr'
> debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
> debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:39
> debug3: load_hostkeys: loaded 1 keys from cbthinkpad
> debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
> debug3: send packet: type 20
> debug1: SSH2_MSG_KEXINIT sent
> debug3: receive packet: type 20
> debug1: SSH2_MSG_KEXINIT received
> debug2: local client KEXINIT proposal
> debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
> debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
> debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
> debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
> debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
> debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
> debug2: compression ctos: none,zlib@openssh.com,zlib
> debug2: compression stoc: none,zlib@openssh.com,zlib
> debug2: languages ctos:
> debug2: languages stoc:
> debug2: first_kex_follows 0
> debug2: reserved 0
> debug2: peer server KEXINIT proposal
> debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
> debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
> debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
> debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
> debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
> debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
> debug2: compression ctos: none,zlib@openssh.com
> debug2: compression stoc: none,zlib@openssh.com
> debug2: languages ctos:
> debug2: languages stoc:
> debug2: first_kex_follows 0
> debug2: reserved 0
> debug1: kex: algorithm: curve25519-sha256
> debug1: kex: host key algorithm: ecdsa-sha2-nistp256
> debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
> debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
> debug3: send packet: type 30
> debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
> debug3: receive packet: type 31
> debug1: Server host key: <HIDDEN>
> debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
> debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:39
> debug3: load_hostkeys: loaded 1 keys from cbthinkpad
> debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
> debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:48
> debug3: load_hostkeys: loaded 1 keys from <HIDDEN>
> debug1: Host 'cbthinkpad' is known and matches the ECDSA host key.
> debug1: Found key in /Users/crstnbr/.ssh/known_hosts:39
> debug3: send packet: type 21
> debug2: set_newkeys: mode 1
> debug1: rekey out after 134217728 blocks
> debug1: SSH2_MSG_NEWKEYS sent
> debug1: expecting SSH2_MSG_NEWKEYS
> debug3: receive packet: type 21
> debug1: SSH2_MSG_NEWKEYS received
> debug2: set_newkeys: mode 0
> debug1: rekey in after 134217728 blocks
> debug1: Will attempt key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
> debug1: Will attempt key: crstnbr@cbpro.local RSA <HIDDEN>
> debug1: Will attempt key: crstnbr@cbpro.fritz.box <HIDDEN>
> debug1: Will attempt key: /Users/crstnbr/.ssh/id_dsa
> debug1: Will attempt key: /Users/crstnbr/.ssh/id_ecdsa
> debug1: Will attempt key: /Users/crstnbr/.ssh/id_ed25519
> debug1: Will attempt key: /Users/crstnbr/.ssh/id_xmss
> debug2: pubkey_prepare: done
> debug3: send packet: type 5
> debug3: receive packet: type 7
> debug1: SSH2_MSG_EXT_INFO received
> debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
> debug3: receive packet: type 6
> debug2: service_accept: ssh-userauth
> debug1: SSH2_MSG_SERVICE_ACCEPT received
> debug3: send packet: type 50
> debug3: receive packet: type 51
> debug1: Authentications that can continue: publickey,password
> debug3: start over, passed a different list publickey,password
> debug3: preferred publickey,keyboard-interactive,password
> debug3: authmethod_lookup publickey
> debug3: remaining preferred: keyboard-interactive,password
> debug3: authmethod_is_enabled publickey
> debug1: Next authentication method: publickey
> debug1: Offering public key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
> debug3: send packet: type 50
> debug2: we sent a publickey packet, wait for reply
> debug3: receive packet: type 60
> debug1: Server accepts key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
> debug3: sign_and_send_pubkey: <HIDDEN>
> debug3: sign_and_send_pubkey: signing using rsa-sha2-512
> debug3: send packet: type 50
> debug3: receive packet: type 52
> debug1: Authentication succeeded (publickey).
> Authenticated to cbthinkpad ([<HIDDEN>]:22).
> debug2: fd 7 setting O_NONBLOCK
> debug2: fd 8 setting O_NONBLOCK
> debug1: channel 0: new [client-session]
> debug3: ssh_session2_open: channel_new: 0
> debug2: channel 0: send open
> debug3: send packet: type 90
> debug1: Requesting no-more-sessions@openssh.com
> debug3: send packet: type 80
> debug1: Entering interactive session.
> debug1: pledge: network
> debug3: receive packet: type 80
> debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
> debug3: receive packet: type 4
> debug1: Remote: /home/crstnbr/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
> debug3: receive packet: type 4
> debug1: Remote: /home/crstnbr/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
> debug3: receive packet: type 91
> debug2: channel_input_open_confirmation: channel 0: callback start
> debug2: fd 6 setting TCP_NODELAY
> debug3: ssh_packet_set_tos: set IPV6_TCLASS 0x20
> debug2: client_session2_setup: id 0
> debug1: Sending environment.
> debug3: Ignored env TERM_SESSION_ID
> debug3: Ignored env SSH_AUTH_SOCK
> debug1: Sending env LC_TERMINAL_VERSION = 3.4.8
> debug2: channel 0: request env confirm 0
> debug3: send packet: type 98
> debug3: Ignored env COLORFGBG
> debug3: Ignored env ITERM_PROFILE
> debug3: Ignored env XPC_FLAGS
> debug3: Ignored env PWD
> debug3: Ignored env SHELL
> debug3: Ignored env __CFBundleIdentifier
> debug3: Ignored env SECURITYSESSIONID
> debug1: Sending env LC_CTYPE = UTF-8
> debug2: channel 0: request env confirm 0
> debug3: send packet: type 98
> debug3: Ignored env TERM_PROGRAM_VERSION
> debug3: Ignored env TERM_PROGRAM
> debug3: Ignored env PATH
> debug3: Ignored env DISPLAY
> debug1: Sending env LC_TERMINAL = iTerm2
> debug2: channel 0: request env confirm 0
> debug3: send packet: type 98
> debug3: Ignored env COLORTERM
> debug3: Ignored env COMMAND_MODE
> debug3: Ignored env TERM
> debug3: Ignored env HOME
> debug3: Ignored env TMPDIR
> debug3: Ignored env USER
> debug3: Ignored env XPC_SERVICE_NAME
> debug3: Ignored env LOGNAME
> debug3: Ignored env LaunchInstanceID
> debug3: Ignored env __CF_USER_TEXT_ENCODING
> debug3: Ignored env ITERM_SESSION_ID
> debug3: Ignored env SHLVL
> debug3: Ignored env OLDPWD
> debug3: Ignored env LS_COLORS
> debug3: Ignored env ZSH
> debug3: Ignored env PAGER
> debug3: Ignored env LESS
> debug3: Ignored env LSCOLORS
> debug3: Ignored env CONDA_EXE
> debug3: Ignored env _CE_M
> debug3: Ignored env _CE_CONDA
> debug3: Ignored env CONDA_PYTHON_EXE
> debug3: Ignored env CONDA_SHLVL
> debug3: Ignored env CONDA_PREFIX
> debug3: Ignored env CONDA_DEFAULT_ENV
> debug3: Ignored env CONDA_PROMPT_MODIFIER
> debug1: Sending env LANG = en_US
> debug2: channel 0: request env confirm 0
> debug3: send packet: type 98
> debug3: Ignored env TZ
> debug3: Ignored env EDITOR
> debug3: Ignored env GIT_EDITOR
> debug3: Ignored env JULIA_EDITOR
> debug3: Ignored env JULIA_NUM_THREADS
> debug3: Ignored env JULIA_PKG_SERVER
> debug3: Ignored env JULIA_MPI_PATH
> debug3: Ignored env JILL_UPSTREAM
> debug3: Ignored env GOPATH
> debug3: Ignored env GOROOT
> debug3: Ignored env PYTHONSTARTUP
> debug3: Ignored env MICRO_TRUECOLOR
> debug3: Ignored env LESS_TERMCAP_mb
> debug3: Ignored env LESS_TERMCAP_md
> debug3: Ignored env LESS_TERMCAP_me
> debug3: Ignored env LESS_TERMCAP_se
> debug3: Ignored env LESS_TERMCAP_so
> debug3: Ignored env LESS_TERMCAP_ue
> debug3: Ignored env LESS_TERMCAP_us
> debug3: Ignored env _
> debug3: Ignored env OPENBLAS_MAIN_FREE
> debug3: Ignored env OPENBLAS_NUM_THREADS
> debug1: Sending command: sh -l -c 'cd -- /home/crstnbr
> '\\''/home/crstnbr/.local/bin/julia'\\'' --worker'
> debug2: channel 0: request exec confirm 1
> debug3: send packet: type 98
> debug2: channel_input_open_confirmation: channel 0: callback done
> debug2: channel 0: open confirm rwindow 0 rmax 32768
> debug2: channel 0: rcvd adjust 2097152
> debug3: receive packet: type 99
> debug2: channel_input_status_confirm: type 99 id 0
> debug2: exec request accepted on channel 0
> debug3: receive packet: type 98
> debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
> debug3: receive packet: type 98
> debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
> debug2: channel 0: rcvd eow
> debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 7 efd 9 [write])
> debug2: channel 0: input open -> closed
> debug3: receive packet: type 96
> debug2: channel 0: rcvd eof
> debug2: channel 0: output open -> drain
> debug2: channel 0: obuf empty
> debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 8 efd 9 [write])
> debug2: channel 0: output drain -> closed
> debug3: receive packet: type 97
> debug2: channel 0: rcvd close
> debug3: channel 0: will not send data after close
> debug2: channel 0: almost dead
> debug2: channel 0: gc: notify user
> debug2: channel 0: gc: user detached
> debug2: channel 0: send close
> debug3: send packet: type 97
> debug2: channel 0: is dead
> debug2: channel 0: garbage collecting
> debug1: channel 0: free: client-session, nchannels 1
> debug3: channel 0: status: The following connections are open:
> #0 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/9 sock -1 cc -1)
> 
> debug3: send packet: type 1
> debug1: fd 0 clearing O_NONBLOCK
> debug3: fd 1 is not O_NONBLOCK
> Transferred: sent 3016, received 3168 bytes, in 62.0 seconds
> Bytes per second: sent 48.6, received 51.1
> debug1: Exit status 1
> 
> ```

---

<div class="post-metadata">

**Author:** ![ppalmes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ppalmes/32/6838_2.png) [@ppalmes](https://discourse.julialang.org/u/ppalmes)\
**Post date:** [June 21, 2021, 8:42pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328/3 "2021-06-21T20:42:51Z")

</div>

try ssh -vvv toyourhost and compare the logs with that by julia. it seems that it successfully created the socket if you look at the logs. can you ssh to your localhost without password?

---

<div class="post-metadata">

**Author:** ![ppalmes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ppalmes/32/6838_2.png) [@ppalmes](https://discourse.julialang.org/u/ppalmes)\
**Post date:** [June 21, 2021, 8:48pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328/4 "2021-06-21T20:48:24Z")

</div>

i can see from the logs that authentication succeeded but the command to start julia seemed to fail. check if the path of julia binary is correct.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 21, 2021, 8:57pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328/5 "2021-06-21T20:57:34Z")

</div>

> [@ppalmes](#):
>
> check if the path of julia binary is correct

The path is correct. I can start Julia with `/home/crstnbr/.local/bin/julia` on the remote machine.

> [@ppalmes](#):
>
> can you ssh to your localhost without password?

You mean the reverse direction remote → local? No, that’s not set up (and never was in the past).

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 21, 2021, 9:02pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328/6 "2021-06-21T21:02:23Z")

</div>

> [@ppalmes](#):
>
> try ssh -vvv toyourhost and compare the logs with that by julia.

This is what I get from `ssh -vvv cbthinkpad`:

```julia
➜ ~ ssh -vvv cbthinkpad
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/crstnbr/.ssh/config
debug1: /Users/crstnbr/.ssh/config line 2: Applying options for *
debug1: /Users/crstnbr/.ssh/config line 9: Applying options for cbthinkpad
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to cbthinkpad port 22.
debug1: Connection established.
debug1: identity file /Users/crstnbr/.ssh/id_rsa type 0
debug1: identity file /Users/crstnbr/.ssh/id_rsa-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_dsa type -1
debug1: identity file /Users/crstnbr/.ssh/id_dsa-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_ecdsa type -1
debug1: identity file /Users/crstnbr/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_ed25519 type -1
debug1: identity file /Users/crstnbr/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_xmss type -1
debug1: identity file /Users/crstnbr/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000
debug3: fd 6 is O_NONBLOCK
debug1: Authenticating to cbthinkpad:22 as 'crstnbr'
debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:39
debug3: load_hostkeys: loaded 1 keys from cbthinkpad
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 <HIDDEN>
debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:39
debug3: load_hostkeys: loaded 1 keys from cbthinkpad
debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:48
debug3: load_hostkeys: loaded 1 keys from <HIDDEN>
debug1: Host 'cbthinkpad' is known and matches the ECDSA host key.
debug1: Found key in /Users/crstnbr/.ssh/known_hosts:39
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
debug1: Will attempt key: crstnbr@cbpro.local <HIDDEN>
debug1: Will attempt key: crstnbr@cbpro.fritz.box <HIDDEN>
debug1: Will attempt key: /Users/crstnbr/.ssh/id_dsa
debug1: Will attempt key: /Users/crstnbr/.ssh/id_ecdsa
debug1: Will attempt key: /Users/crstnbr/.ssh/id_ed25519
debug1: Will attempt key: /Users/crstnbr/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
debug3: sign_and_send_pubkey: <HIDDEN>
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to cbthinkpad ([<HIDDEN>]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 4
debug1: Remote: /home/crstnbr/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 4
debug1: Remote: /home/crstnbr/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 0: request auth-agent-req@openssh.com confirm 0
debug3: send packet: type 98
debug2: fd 6 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IPV6_TCLASS 0x48
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env SSH_AUTH_SOCK
debug1: Sending env LC_TERMINAL_VERSION = 3.4.8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env COLORFGBG
debug3: Ignored env ITERM_PROFILE
debug3: Ignored env XPC_FLAGS
debug3: Ignored env PWD
debug3: Ignored env SHELL
debug3: Ignored env __CFBundleIdentifier
debug3: Ignored env SECURITYSESSIONID
debug1: Sending env LC_CTYPE = UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env PATH
debug3: Ignored env DISPLAY
debug1: Sending env LC_TERMINAL = iTerm2
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env COLORTERM
debug3: Ignored env COMMAND_MODE
debug3: Ignored env TERM
debug3: Ignored env HOME
debug3: Ignored env TMPDIR
debug3: Ignored env USER
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env LOGNAME
debug3: Ignored env LaunchInstanceID
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env ITERM_SESSION_ID
debug3: Ignored env SHLVL
debug3: Ignored env OLDPWD
debug3: Ignored env LS_COLORS
debug3: Ignored env ZSH
debug3: Ignored env PAGER
debug3: Ignored env LESS
debug3: Ignored env LSCOLORS
debug3: Ignored env CONDA_EXE
debug3: Ignored env _CE_M
debug3: Ignored env _CE_CONDA
debug3: Ignored env CONDA_PYTHON_EXE
debug3: Ignored env CONDA_SHLVL
debug3: Ignored env CONDA_PREFIX
debug3: Ignored env CONDA_DEFAULT_ENV
debug3: Ignored env CONDA_PROMPT_MODIFIER
debug1: Sending env LANG = en_US
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env TZ
debug3: Ignored env EDITOR
debug3: Ignored env GIT_EDITOR
debug3: Ignored env JULIA_EDITOR
debug3: Ignored env JULIA_NUM_THREADS
debug3: Ignored env JULIA_PKG_SERVER
debug3: Ignored env JULIA_MPI_PATH
debug3: Ignored env JILL_UPSTREAM
debug3: Ignored env GOPATH
debug3: Ignored env GOROOT
debug3: Ignored env PYTHONSTARTUP
debug3: Ignored env MICRO_TRUECOLOR
debug3: Ignored env LESS_TERMCAP_mb
debug3: Ignored env LESS_TERMCAP_md
debug3: Ignored env LESS_TERMCAP_me
debug3: Ignored env LESS_TERMCAP_se
debug3: Ignored env LESS_TERMCAP_so
debug3: Ignored env LESS_TERMCAP_ue
debug3: Ignored env LESS_TERMCAP_us
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0

```

---

<div class="post-metadata">

**Author:** ![mgkuhn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mgkuhn/32/6276_2.png) [@mgkuhn](https://discourse.julialang.org/u/mgkuhn)\
**Post date:** [June 21, 2021, 9:09pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328/7 "2021-06-21T21:09:12Z")

</div>

What shell are you logging into? bash? zsh? tcsh? (Asking because tcsh is not a POSIX shell and known to not work.)

---

<div class="post-metadata">

**Author:** ![mgkuhn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mgkuhn/32/6276_2.png) [@mgkuhn](https://discourse.julialang.org/u/mgkuhn)\
**Post date:** [June 21, 2021, 9:11pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328/8 "2021-06-21T21:11:50Z")

</div>

If you set kwarg `ssh="echo"` and type the command-line(s) you see manually, what happens?

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 21, 2021, 9:12pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328/9 "2021-06-21T21:12:12Z")

</div>

I’m logging into bash on the thinkpad. On the MacBook I’m using zsh.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 21, 2021, 9:16pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328/10 "2021-06-21T21:16:46Z")

</div>

Hm, I get

```julia
julia> addprocs([("cbthinkpad", 1)]; exename="/home/crstnbr/.local/bin/julia", dir="/home/crstnbr", topology=:master_worker, ssh="echo")
exception launching on machine cbthinkpad : Base.IOError("write: broken pipe (EPIPE)", -32)
Int64[]

```

---

<div class="post-metadata">

**Author:** ![mgkuhn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mgkuhn/32/6276_2.png) [@mgkuhn](https://discourse.julialang.org/u/mgkuhn)\
**Post date:** [June 21, 2021, 9:19pm UTC](https://discourse.julialang.org/t/cant-add-distributed-workers-per-ssh-connection-timed-out/63328/11 "2021-06-21T21:19:23Z")

</div>

Never mind. If you can see the `julia --worker` process running, it’s probably not a problem with the shell command line.