Can't add distributed workers per ssh: connection timed out

General Usage
,
1

First, let me say that I’m super confused since adding workers per ssh has worked fine for me for many years. Even on the same machine everything was fine a few months back.

I’m trying to start a Julia worker on a thinkpad running ubuntu from my MacBook Pro running macOS (Julia 1.6.1 on both but I tested 1.5 as well). Although the worker process seems to get spawned correctly (see the htop screenshot from the target machine below), on my MacBook (the master) I get

julia> addprocs([("cbthinkpad", 1)]; exename=`/home/crstnbr/.local/bin/julia`, dir="/home/crstnbr", topology=:master_worker)
ERROR: TaskFailedException

    nested task error: IOError: connect: connection timed out (ETIMEDOUT)
    Stacktrace:
     [1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:1082
     [2] worker_from_id
       @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:1079 [inlined]
     [3] #remote_do#154
       @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:486 [inlined]
     [4] remote_do
       @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:486 [inlined]
     [5] kill(manager::Distributed.SSHManager, pid::Int64, config::WorkerConfig)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/managers.jl:680
     [6] create_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:593
     [7] setup_launched_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:534
     [8] (::Distributed.var"#41#44"{Distributed.SSHManager, Vector{Int64}, WorkerConfig})()
       @ Distributed ./task.jl:411

    caused by: IOError: connect: connection timed out (ETIMEDOUT)
    Stacktrace:
     [1] wait_connected(x::Sockets.TCPSocket)
       @ Sockets /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Sockets/src/Sockets.jl:532
     [2] connect
       @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Sockets/src/Sockets.jl:567 [inlined]
     [3] connect_to_worker(host::SubString{String}, port::Int64)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/managers.jl:639
     [4] connect(manager::Distributed.SSHManager, pid::Int64, config::WorkerConfig)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/managers.jl:566
     [5] create_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig)
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:589
     [6] setup_launched_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:534
     [7] (::Distributed.var"#41#44"{Distributed.SSHManager, Vector{Int64}, WorkerConfig})()
       @ Distributed ./task.jl:411
Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:369
 [2] macro expansion
   @ ./task.jl:388 [inlined]
 [3] addprocs_locked(manager::Distributed.SSHManager; kwargs::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:exename, :dir, :topology), Tuple{Cmd, String, Symbol}}})
   @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:480
 [4] addprocs(manager::Distributed.SSHManager; kwargs::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:exename, :dir, :topology), Tuple{Cmd, String, Symbol}}})
   @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:444
 [5] #addprocs#237
   @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/managers.jl:143 [inlined]
 [6] top-level scope
   @ REPL[2]:100:

Screenshot 2021-06-21 at 22.01.02
Screenshot 2021-06-21 at 22.01.022554×1362 324 KB

Initially, I thought it might be an issue with the thinkpad. But after having tried to spawn workers on two other machines (another linux desktop and a linux VM) for both of which I saw time outs as well I’m now thinking it must be something about the master, i.e. my MacBook. I just don’t have a clue, in particular given that this was working just fine in March… I found Properly connect Julia processes in multiple machines with Sockets, so maybe Julia is listening on the wrong interface? I tried disabling everything except my WiFi but still see the same error. In Addprocs with SSH and a ProxyJump times out a time out is seen when using a proxy jump. As described here, trying to spawn workers on a different linux machine which I can only access via a proxy using the “control master” approach mentioned in this thread I could trade the time out error for the following error:

ERROR: TaskFailedException

    nested task error: unable to create SSH tunnel after 100 tries. No free port?

Finally, What does this error mean? also sees a time out when trying to start a distributed worker. There, someone suggested that it might be a cross-OS issue. But given that this has always worked in the past (and I tried 1.5 where I know for sure that it worked in March) this seems unlikely to me.

Anyone has an idea what’s going on here?!

2

As suggested by @ppalmes over on Slack, I tried passing sshflags=`-vvv` to get a bit more debug information. Can anyone with more ssh knowledge decipher this?

Debug information 
julia> addprocs([("cbthinkpad", 1)]; exename=`/home/crstnbr/.local/bin/julia`, dir="/home/crstnbr", topology=:master_worker, sshflags=`-vvv`)
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/crstnbr/.ssh/config
debug1: /Users/crstnbr/.ssh/config line 2: Applying options for *
debug1: /Users/crstnbr/.ssh/config line 9: Applying options for cbthinkpad
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to cbthinkpad port 22.
debug1: Connection established.
debug1: identity file /Users/crstnbr/.ssh/id_rsa type 0
debug1: identity file /Users/crstnbr/.ssh/id_rsa-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_dsa type -1
debug1: identity file /Users/crstnbr/.ssh/id_dsa-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_ecdsa type -1
debug1: identity file /Users/crstnbr/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_ed25519 type -1
debug1: identity file /Users/crstnbr/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_xmss type -1
debug1: identity file /Users/crstnbr/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000
debug3: fd 6 is O_NONBLOCK
debug1: Authenticating to cbthinkpad:22 as 'crstnbr'
debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:39
debug3: load_hostkeys: loaded 1 keys from cbthinkpad
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: <HIDDEN>
debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:39
debug3: load_hostkeys: loaded 1 keys from cbthinkpad
debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:48
debug3: load_hostkeys: loaded 1 keys from <HIDDEN>
debug1: Host 'cbthinkpad' is known and matches the ECDSA host key.
debug1: Found key in /Users/crstnbr/.ssh/known_hosts:39
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
debug1: Will attempt key: crstnbr@cbpro.local RSA <HIDDEN>
debug1: Will attempt key: crstnbr@cbpro.fritz.box <HIDDEN>
debug1: Will attempt key: /Users/crstnbr/.ssh/id_dsa
debug1: Will attempt key: /Users/crstnbr/.ssh/id_ecdsa
debug1: Will attempt key: /Users/crstnbr/.ssh/id_ed25519
debug1: Will attempt key: /Users/crstnbr/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
debug3: sign_and_send_pubkey: <HIDDEN>
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to cbthinkpad ([<HIDDEN>]:22).
debug2: fd 7 setting O_NONBLOCK
debug2: fd 8 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 4
debug1: Remote: /home/crstnbr/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 4
debug1: Remote: /home/crstnbr/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 6 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IPV6_TCLASS 0x20
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env SSH_AUTH_SOCK
debug1: Sending env LC_TERMINAL_VERSION = 3.4.8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env COLORFGBG
debug3: Ignored env ITERM_PROFILE
debug3: Ignored env XPC_FLAGS
debug3: Ignored env PWD
debug3: Ignored env SHELL
debug3: Ignored env __CFBundleIdentifier
debug3: Ignored env SECURITYSESSIONID
debug1: Sending env LC_CTYPE = UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env PATH
debug3: Ignored env DISPLAY
debug1: Sending env LC_TERMINAL = iTerm2
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env COLORTERM
debug3: Ignored env COMMAND_MODE
debug3: Ignored env TERM
debug3: Ignored env HOME
debug3: Ignored env TMPDIR
debug3: Ignored env USER
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env LOGNAME
debug3: Ignored env LaunchInstanceID
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env ITERM_SESSION_ID
debug3: Ignored env SHLVL
debug3: Ignored env OLDPWD
debug3: Ignored env LS_COLORS
debug3: Ignored env ZSH
debug3: Ignored env PAGER
debug3: Ignored env LESS
debug3: Ignored env LSCOLORS
debug3: Ignored env CONDA_EXE
debug3: Ignored env _CE_M
debug3: Ignored env _CE_CONDA
debug3: Ignored env CONDA_PYTHON_EXE
debug3: Ignored env CONDA_SHLVL
debug3: Ignored env CONDA_PREFIX
debug3: Ignored env CONDA_DEFAULT_ENV
debug3: Ignored env CONDA_PROMPT_MODIFIER
debug1: Sending env LANG = en_US
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env TZ
debug3: Ignored env EDITOR
debug3: Ignored env GIT_EDITOR
debug3: Ignored env JULIA_EDITOR
debug3: Ignored env JULIA_NUM_THREADS
debug3: Ignored env JULIA_PKG_SERVER
debug3: Ignored env JULIA_MPI_PATH
debug3: Ignored env JILL_UPSTREAM
debug3: Ignored env GOPATH
debug3: Ignored env GOROOT
debug3: Ignored env PYTHONSTARTUP
debug3: Ignored env MICRO_TRUECOLOR
debug3: Ignored env LESS_TERMCAP_mb
debug3: Ignored env LESS_TERMCAP_md
debug3: Ignored env LESS_TERMCAP_me
debug3: Ignored env LESS_TERMCAP_se
debug3: Ignored env LESS_TERMCAP_so
debug3: Ignored env LESS_TERMCAP_ue
debug3: Ignored env LESS_TERMCAP_us
debug3: Ignored env _
debug3: Ignored env OPENBLAS_MAIN_FREE
debug3: Ignored env OPENBLAS_NUM_THREADS
debug1: Sending command: sh -l -c 'cd -- /home/crstnbr
'\\''/home/crstnbr/.local/bin/julia'\\'' --worker'
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 7 efd 9 [write])
debug2: channel 0: input open -> closed
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 8 efd 9 [write])
debug2: channel 0: output drain -> closed
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/9 sock -1 cc -1)

debug3: send packet: type 1
debug1: fd 0 clearing O_NONBLOCK
debug3: fd 1 is not O_NONBLOCK
Transferred: sent 3016, received 3168 bytes, in 62.0 seconds
Bytes per second: sent 48.6, received 51.1
debug1: Exit status 1
3

try ssh -vvv toyourhost and compare the logs with that by julia. it seems that it successfully created the socket if you look at the logs. can you ssh to your localhost without password?

4

i can see from the logs that authentication succeeded but the command to start julia seemed to fail. check if the path of julia binary is correct.

5

The path is correct. I can start Julia with /home/crstnbr/.local/bin/julia on the remote machine.

You mean the reverse direction remote → local? No, that’s not set up (and never was in the past).

6

This is what I get from ssh -vvv cbthinkpad:

➜  ~ ssh -vvv cbthinkpad
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/crstnbr/.ssh/config
debug1: /Users/crstnbr/.ssh/config line 2: Applying options for *
debug1: /Users/crstnbr/.ssh/config line 9: Applying options for cbthinkpad
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to cbthinkpad port 22.
debug1: Connection established.
debug1: identity file /Users/crstnbr/.ssh/id_rsa type 0
debug1: identity file /Users/crstnbr/.ssh/id_rsa-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_dsa type -1
debug1: identity file /Users/crstnbr/.ssh/id_dsa-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_ecdsa type -1
debug1: identity file /Users/crstnbr/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_ed25519 type -1
debug1: identity file /Users/crstnbr/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/crstnbr/.ssh/id_xmss type -1
debug1: identity file /Users/crstnbr/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000
debug3: fd 6 is O_NONBLOCK
debug1: Authenticating to cbthinkpad:22 as 'crstnbr'
debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:39
debug3: load_hostkeys: loaded 1 keys from cbthinkpad
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 <HIDDEN>
debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:39
debug3: load_hostkeys: loaded 1 keys from cbthinkpad
debug3: hostkeys_foreach: reading file "/Users/crstnbr/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/crstnbr/.ssh/known_hosts:48
debug3: load_hostkeys: loaded 1 keys from <HIDDEN>
debug1: Host 'cbthinkpad' is known and matches the ECDSA host key.
debug1: Found key in /Users/crstnbr/.ssh/known_hosts:39
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
debug1: Will attempt key: crstnbr@cbpro.local <HIDDEN>
debug1: Will attempt key: crstnbr@cbpro.fritz.box <HIDDEN>
debug1: Will attempt key: /Users/crstnbr/.ssh/id_dsa
debug1: Will attempt key: /Users/crstnbr/.ssh/id_ecdsa
debug1: Will attempt key: /Users/crstnbr/.ssh/id_ed25519
debug1: Will attempt key: /Users/crstnbr/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/crstnbr/.ssh/id_rsa <HIDDEN>
debug3: sign_and_send_pubkey: <HIDDEN>
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to cbthinkpad ([<HIDDEN>]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 4
debug1: Remote: /home/crstnbr/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 4
debug1: Remote: /home/crstnbr/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 0: request auth-agent-req@openssh.com confirm 0
debug3: send packet: type 98
debug2: fd 6 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IPV6_TCLASS 0x48
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env SSH_AUTH_SOCK
debug1: Sending env LC_TERMINAL_VERSION = 3.4.8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env COLORFGBG
debug3: Ignored env ITERM_PROFILE
debug3: Ignored env XPC_FLAGS
debug3: Ignored env PWD
debug3: Ignored env SHELL
debug3: Ignored env __CFBundleIdentifier
debug3: Ignored env SECURITYSESSIONID
debug1: Sending env LC_CTYPE = UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env PATH
debug3: Ignored env DISPLAY
debug1: Sending env LC_TERMINAL = iTerm2
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env COLORTERM
debug3: Ignored env COMMAND_MODE
debug3: Ignored env TERM
debug3: Ignored env HOME
debug3: Ignored env TMPDIR
debug3: Ignored env USER
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env LOGNAME
debug3: Ignored env LaunchInstanceID
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env ITERM_SESSION_ID
debug3: Ignored env SHLVL
debug3: Ignored env OLDPWD
debug3: Ignored env LS_COLORS
debug3: Ignored env ZSH
debug3: Ignored env PAGER
debug3: Ignored env LESS
debug3: Ignored env LSCOLORS
debug3: Ignored env CONDA_EXE
debug3: Ignored env _CE_M
debug3: Ignored env _CE_CONDA
debug3: Ignored env CONDA_PYTHON_EXE
debug3: Ignored env CONDA_SHLVL
debug3: Ignored env CONDA_PREFIX
debug3: Ignored env CONDA_DEFAULT_ENV
debug3: Ignored env CONDA_PROMPT_MODIFIER
debug1: Sending env LANG = en_US
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env TZ
debug3: Ignored env EDITOR
debug3: Ignored env GIT_EDITOR
debug3: Ignored env JULIA_EDITOR
debug3: Ignored env JULIA_NUM_THREADS
debug3: Ignored env JULIA_PKG_SERVER
debug3: Ignored env JULIA_MPI_PATH
debug3: Ignored env JILL_UPSTREAM
debug3: Ignored env GOPATH
debug3: Ignored env GOROOT
debug3: Ignored env PYTHONSTARTUP
debug3: Ignored env MICRO_TRUECOLOR
debug3: Ignored env LESS_TERMCAP_mb
debug3: Ignored env LESS_TERMCAP_md
debug3: Ignored env LESS_TERMCAP_me
debug3: Ignored env LESS_TERMCAP_se
debug3: Ignored env LESS_TERMCAP_so
debug3: Ignored env LESS_TERMCAP_ue
debug3: Ignored env LESS_TERMCAP_us
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
7

What shell are you logging into? bash? zsh? tcsh? (Asking because tcsh is not a POSIX shell and known to not work.)

8

If you set kwarg ssh="echo" and type the command-line(s) you see manually, what happens?

9

I’m logging into bash on the thinkpad. On the MacBook I’m using zsh.

10

Hm, I get

julia> addprocs([("cbthinkpad", 1)]; exename="/home/crstnbr/.local/bin/julia", dir="/home/crstnbr", topology=:master_worker, ssh="echo")
exception launching on machine cbthinkpad : Base.IOError("write: broken pipe (EPIPE)", -32)
Int64[]
11

Never mind. If you can see the julia --worker process running, it’s probably not a problem with the shell command line.