# Distributed fails to create command on ssh connection to remote worker

**URL:** <https://discourse.julialang.org/t/distributed-fails-to-create-command-on-ssh-connection-to-remote-worker/114704>\
**Category:** General Usage\
**Created:** [May 24, 2024, 9:38pm UTC](https://discourse.julialang.org/t/distributed-fails-to-create-command-on-ssh-connection-to-remote-worker/114704 "2024-05-24T21:38:31Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Teodor\_Tchalakov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/teodor_tchalakov/32/53043_2.png) [@Teodor\_Tchalakov](https://discourse.julialang.org/u/Teodor_Tchalakov)\
**Post date:** [May 24, 2024, 9:38pm UTC](https://discourse.julialang.org/t/distributed-fails-to-create-command-on-ssh-connection-to-remote-worker/114704/1 "2024-05-24T21:38:31Z")

</div>

When adding remote workers through an ssh connection using [Distributed.jl](https://github.com/JuliaLang/Distributed.jl), the SSH Manager is able to connect but for some reason a command over ssh gets improperly formatted and setting up the remote workers fails. The error I receive is `The syntax of the command is incorrect`, more details below.

I am running Julia 1.10.3 on an M1 MacBook Pro and am connecting to a Windows 10 device using ssh (OpenSSH server) on my local network. The ssh connection is established using a certificate based authentication, which I guarantee works because I can connect through a terminal and the debug from the ssh connection in Julia shows that the connection is established. Specifically, I run the command:

```julia
addprocs([("user@hostname", :auto)], shell=:wincmd, tunnel=false, exename="julia", sshflags=`-vvv`)

```

With debug output shows (some stuff omitted):

```julia
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env MallocNanoZone
debug3: Ignored env USER
debug3: Ignored env SECURITYSESSIONID
debug3: Ignored env COMMAND_MODE
debug3: Ignored env __CFBundleIdentifier
debug3: Ignored env PATH
debug3: Ignored env HOME
debug3: Ignored env SHELL
debug3: Ignored env LaunchInstanceID
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env XPC_FLAGS
debug3: Ignored env LOGNAME
debug3: Ignored env TMPDIR
debug3: Ignored env ORIGINAL_XDG_CURRENT_DESKTOP
debug3: Ignored env SHLVL
debug3: Ignored env PWD
debug3: Ignored env OLDPWD
debug3: Ignored env HOMEBREW_PREFIX
debug3: Ignored env HOMEBREW_CELLAR
debug3: Ignored env HOMEBREW_REPOSITORY
debug3: Ignored env MANPATH
debug3: Ignored env INFOPATH
debug3: Ignored env LDFLAGS
debug3: Ignored env CPPFLAGS
debug3: Ignored env PKG_CONFIG_PATH
debug3: Ignored env _
debug3: Ignored env JULIA_EDITOR
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env TERM_PROGRAM_VERSION
debug1: channel 0: setting env LANG = "en_US.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env COLORTERM
debug3: Ignored env VSCODE_ENV_REPLACE
debug3: Ignored env VSCODE_ENV_PREPEND
debug3: Ignored env VIRTUAL_ENV
debug3: Ignored env GIT_ASKPASS
debug3: Ignored env VSCODE_GIT_ASKPASS_NODE
debug3: Ignored env VSCODE_GIT_ASKPASS_EXTRA_ARGS
debug3: Ignored env VSCODE_GIT_ASKPASS_MAIN
debug3: Ignored env VSCODE_GIT_IPC_HANDLE
debug3: Ignored env TERM
debug3: Ignored env OPENBLAS_MAIN_FREE
debug3: Ignored env OPENBLAS_DEFAULT_NUM_THREADS
debug1: Sending command: pushd "/Julia" && julia --worker
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug3: client_repledge: enter
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 81
debug1: client_global_hostkeys_prove_confirm: server used untrusted RSA signature algorithm ssh-rsa for key 0, disregarding
debug3: client_repledge: enter
debug1: pledge: fork
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: rcvd ext data 41
The syntax of the command is incorrect.
debug2: channel 0: written 41 to efd 9
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: chan_shutdown_write: channel 0: (i0 o1 sock -1 wfd 8 efd 9 [write])
debug2: channel 0: output drain -> closed
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug2: chan_shutdown_read: channel 0: (i0 o3 sock -1 wfd 7 efd 9 [write])
debug2: channel 0: input open -> closed
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 [session] r0 i3/0 o3/0 e[write]/0 fd -1/-1/9 sock -1 cc -1 io 0x00/0x00)

debug3: send packet: type 1
Transferred: sent 3252, received 2932 bytes, in 0.2 seconds
Bytes per second: sent 19759.6, received 17815.3
debug1: Exit status 1
ERROR: TaskFailedException

    nested task error: Unable to read host:port string from worker. Launch command exited with error?
    Stacktrace:
     [1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64)
       @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:1093
     [2] worker_from_id
       @ /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:1090 [inlined]
     [3] remote_do
       @ /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/remotecall.jl:557 [inlined]
     [4] kill(manager::Distributed.SSHManager, pid::Int64, config::WorkerConfig)
       @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/managers.jl:731
     [5] create_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig)
       @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:604
     [6] setup_launched_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:545
     [7] (::Distributed.var"#45#48"{Distributed.SSHManager, Vector{Int64}, WorkerConfig})()
       @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:501
    
    caused by: Unable to read host:port string from worker. Launch command exited with error?
    Stacktrace:
     [1] read_worker_host_port(io::Base.PipeEndpoint)
       @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:330
     [2] connect(manager::Distributed.SSHManager, pid::Int64, config::WorkerConfig)
       @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/managers.jl:575
     [3] create_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig)
       @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:600
     [4] setup_launched_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:545
     [5] (::Distributed.var"#45#48"{Distributed.SSHManager, Vector{Int64}, WorkerConfig})()
       @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:501
Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:448
 [2] macro expansion
   @ ./task.jl:480 [inlined]
 [3] addprocs_locked(manager::Distributed.SSHManager; kwargs::@Kwargs{…})
   @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:490
 [4] addprocs_locked
   @ /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:456 [inlined]
 [5] addprocs(manager::Distributed.SSHManager; kwargs::@Kwargs{…})
   @ Distributed /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:450
 [6] addprocs
   @ /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:443 [inlined]
 [7] #addprocs#255
   @ /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Distributed/src/managers.jl:159 [inlined]
 [8] top-level scope
   @ clustermanagement.jl:18
Some type information was truncated. Use `show(err)` to see complete types.

```

My current idea is that this is likely a problem with Distributed.jl not serializing commands correct for communication between MacOS and Windows 10. There was an issue previous found [here](https://github.com/JuliaLang/julia/issues/29702) which notes that communication with Windows 10 is non-Posix, but this has been added to Distributed.jl by specifying `shell=:wincmd`. I am not sure if this is a problem with how I can configured my ssh connection, the Distributed.jl implementation, or if this is a problem with Julia’s implementation for ssh communications between MacOS and Windows 10 operating systems. Any thoughts would be helpful.

---

<div class="post-metadata">

**Author:** ![owiecc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/owiecc/32/8894_2.png) [@owiecc](https://discourse.julialang.org/u/owiecc)\
**Post date:** [February 5, 2025, 10:08am UTC](https://discourse.julialang.org/t/distributed-fails-to-create-command-on-ssh-connection-to-remote-worker/114704/2 "2025-02-05T10:08:40Z")

</div>

Did you figure out how to connect to Windows? I am fighting with the same problem right now.