Why running remote ssh cmd in threads is still blocked?

Hi there, I want to run some commands on remote machine. I use julia to manage my commands. To avoid blocking in main threads, I run the commands in a spawned thread.For example:

cmd = """
sleep 3
echo Hello
Threads.@spawn begin
    run(`ssh user@remoteip "$cmd"`)

However, it seems that the ssh still blocked the main thread. I can do nothing in the REPL. But if I run those commands locally, the main thread won’t be blocked:

Threads.@spawn begin
    run(`bash -c "$cmd")

In my julia REPL,Threads.nthreads() equals 16. And I’m sure the code above almost runs in different thread everytime. Is the ssh’s internet io communication blocked the main thread? Or have I missed any thing?
I know, we can use Distributed’s functions. Could we just use Threads package to realize my needs? Thanks.

How many threads did you start julia with? What’s the output of versioninfo()? Does connecting to your remote require entering a password or similar?

Hi Sukera,thanks for your reply. I started julia: julia -t auto. There are 16 threads. versioninfo is as follow:

Julia Version 1.9.0
Commit 8e630552924 (2023-05-07 11:25 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 16 on 16 virtual cores
  LD_LIBRARY_PATH = /usr/local/nvidia/lib:/usr/local/nvidia/lib64

I use key file for ssh authorization. No passwd needed.

Then I can’t reproduce the issue - this works for me:

julia> cmd = """
       sleep 3
       echo Hello

julia> Threads.@spawn run(`ssh user@host $cmd`); println("test")

julia> Hello

Note that the " are not needed - julia does escaping of interpolated values in commands automatically:

julia> `ssh user@host $cmd`
`ssh user@host 'sleep 3
echo Hello

See also this juliacon talk for more information.

Does the connection without Threads.@spawn work, i.e. just this:

julia> run(`ssh user@host "$cmd"`)
Process(`ssh user@host 'sleep 3
echo Hello
'`, ProcessExited(0))


Thanks, I confirmed again. Your code works for me too. The test string outputs normally. But have you found that when you run the code, the REPL stuck? I think I must misunderstand the phenomenon, so, it’s not the main thread is blocked, it’s just the REPL blocked. Sorry for the inappropriate description.

No, the REPL isn’t stuck - I can execute commands as usual. Is it actually blocked, i.e. entering things doesn’t show up? Can you run commands? Do you maybe need the -n flag to ssh, so that ssh doesn’t try to highjack your stdin? On my machine I have this:

-n      Redirects stdin from /dev/null (actually, prevents reading from stdin).  This must be used when ssh
        is run in the background.  A common trick is to use this to run X11 programs on a remote machine.
        For example, ssh -n shadows.cs.hut.fi emacs & will start an emacs on shadows.cs.hut.fi, and the X11
        connection will be automatically forwarded over an encrypted channel.  The ssh program will be put
        in the background.  (This does not work if ssh needs to ask for a password or passphrase; see also
        the -f option.)  Refer to the description of StdinNull in ssh_config(5) for details.

If you want to capture the output of that ssh command in julia, you can use read(`ssh ...`) by the way.

1 Like

Oh god, it’s true. I tried with -n option, the REPL isn’t stuck anymore. Thanks a lot. You really solved my problem.

1 Like

You’re welcome!

I’d also recommend looking at pipeline as well as the wait keyword to run, to disentangle your parent process IO with the child process.

Thanks. Yes, I’ve used pipeline for my ordinary commands execution and result retrival.