Why running remote ssh cmd in threads is still blocked?

General Usage
,
1

Hi there, I want to run some commands on remote machine. I use julia to manage my commands. To avoid blocking in main threads, I run the commands in a spawned thread.For example:

cmd = """
sleep 3
echo Hello
"""
Threads.@spawn begin
    run(`ssh user@remoteip "$cmd"`)
end

However, it seems that the ssh still blocked the main thread. I can do nothing in the REPL. But if I run those commands locally, the main thread won’t be blocked:

Threads.@spawn begin
    run(`bash -c "$cmd")
end

In my julia REPL,Threads.nthreads() equals 16. And I’m sure the code above almost runs in different thread everytime. Is the ssh’s internet io communication blocked the main thread? Or have I missed any thing?
I know, we can use Distributed’s functions. Could we just use Threads package to realize my needs? Thanks.

2

How many threads did you start julia with? What’s the output of versioninfo()? Does connecting to your remote require entering a password or similar?

3

Hi Sukera,thanks for your reply. I started julia: julia -t auto. There are 16 threads. versioninfo is as follow:

versioninfo()
Julia Version 1.9.0
Commit 8e630552924 (2023-05-07 11:25 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 16 on 16 virtual cores
Environment:
  LD_LIBRARY_PATH = /usr/local/nvidia/lib:/usr/local/nvidia/lib64
  JULIA_EDITOR = code
  JULIA_NUM_THREADS =

I use key file for ssh authorization. No passwd needed.

4

Then I can’t reproduce the issue - this works for me:

julia> cmd = """
       sleep 3
       echo Hello
       """

julia> Threads.@spawn run(`ssh user@host $cmd`); println("test")
test

julia> Hello

Note that the " are not needed - julia does escaping of interpolated values in commands automatically:

julia> `ssh user@host $cmd`
`ssh user@host 'sleep 3
echo Hello
'`

See also this juliacon talk for more information.

Does the connection without Threads.@spawn work, i.e. just this:

julia> run(`ssh user@host "$cmd"`)
Hello
Process(`ssh user@host 'sleep 3
echo Hello
'`, ProcessExited(0))

?

5

Thanks, I confirmed again. Your code works for me too. The test string outputs normally. But have you found that when you run the code, the REPL stuck? I think I must misunderstand the phenomenon, so, it’s not the main thread is blocked, it’s just the REPL blocked. Sorry for the inappropriate description.

6

No, the REPL isn’t stuck - I can execute commands as usual. Is it actually blocked, i.e. entering things doesn’t show up? Can you run commands? Do you maybe need the -n flag to ssh, so that ssh doesn’t try to highjack your stdin? On my machine I have this:

-n      Redirects stdin from /dev/null (actually, prevents reading from stdin).  This must be used when ssh
        is run in the background.  A common trick is to use this to run X11 programs on a remote machine.
        For example, ssh -n shadows.cs.hut.fi emacs & will start an emacs on shadows.cs.hut.fi, and the X11
        connection will be automatically forwarded over an encrypted channel.  The ssh program will be put
        in the background.  (This does not work if ssh needs to ask for a password or passphrase; see also
        the -f option.)  Refer to the description of StdinNull in ssh_config(5) for details.

If you want to capture the output of that ssh command in julia, you can use read(`ssh ...`) by the way.

1 Like
7

Oh god, it’s true. I tried with -n option, the REPL isn’t stuck anymore. Thanks a lot. You really solved my problem.

1 Like
8

You’re welcome!

I’d also recommend looking at pipeline as well as the wait keyword to run, to disentangle your parent process IO with the child process.

9

Thanks. Yes, I’ve used pipeline for my ordinary commands execution and result retrival.