Low-level IO with libuv

Internals & Design
1

Hi, I’m working on the Expect.jl module to interact with external
processes under a pty (https://github.com/wavexx/Expect.jl).

For those unfamiliar with expect-like modules, the current API looks
like this:

  interact(`ssh somewhere`, 10) do ssh
      expect!(ssh, " password: ")
      println(ssh, "supersecret")
      ...
  end

I need to have fine-grained control on how buffering is performed at a
lower level and I think I’ve reached a limit on what I can do using the
Julia runtime and need some guidance.

I’ll start with the easiest issue:

Julia’s Base.TTY seem to be designed only as a front-end to the main
terminal interacting with the user. This seems to stem from libuv as
well. For one, libuv tracks the first tty on which the uv_tty_set_mode
has been called, just to reset it on shutdown.

https://github.com/libuv/libuv/issues/1292

This obviously doesn’t take pseudo-ttys into consideration. Julia itself
also calls uv_tty_set_mode in jl_close_uv!

This seems broken, as if we use libuv, we should only be calling
uv_tty_reset_mode before exiting (which is intended to be called from a
signal handler) and ensure the tty reset by libuv itself is the console
(again, this can only be done currently by calling uv_tty_set_mode with
a non-normal mode at start! - another issue with the libuv api design).
Getting stuff fixed in libuv takes forever, and even more to propagate
the change to julia, so this is why I’m not directly filling PRs and
issues right away.

I see the api in libuv broken in this regard: there should be a function
call to explicitly set the tty to reset on un_tty_reset_mode.

The second issue is reading. Reads from LibuvStream are throttled by
julia, however I need to ensure the input stream coming from the
coprocess is never stopped, otherwise I can incur in a stall in a
pending write. Looking at Base.wait_readnb I see no easy way to avoid
the call to stop_reading. Ideas?

I’d love to be able to re-use the code in stream.jl, but the handling of
the read timeout is also something I want to manage directly. Is there a
“recommended” way to create a new handle directly using libuv? I need to
cram the descriptor as being watched in the main uv loop and set my own
callbacks.

Thanks for any pointer.

2

This is correct. In particular, TTY is really most representative of a TTY slave. For a TTY master, it’s perhaps closer to open this as a PipeEndpoint object on the Julia side. This can also help with avoid some of the libuv bugs surrounding its tty support (although you don’t need to use the same type for the Julia and libuv representation).

You should file duplicate issues against Julia’s libuv fork.

Yeah, this is a libuv bug. AFAICT, they/their users must not be using external processes / tty redirection much, since this use case is why we have a fork. We’ve tried to work around this by calling uv_tty_set_mode from julia_init on all (tty) file descriptors, which has been fairly successful at hiding it. But I agree we should fix it to detect / record exactly the case where we called it on stdin fd(0) during init, and otherwise not reset the tty mode.
New issue opened: improve uv_tty_reset_mode logic · Issue #21659 · JuliaLang/julia · GitHub

They aren’t stopped. They are throttled by your code whenever you don’t have a buffer assigned for reading into. Run your IO handler in an @async task, and it won’t get throttled.

There is no read timeout. It is strongly discouraged to add one as this would cause race conditions in your code. There is either data or there is not (eof / readavailable).

3

Is there a way to construct a PipeEndpoint manually from a RawFD that I don’t see?

I want to consume into a buffer indefinitely, even when there’s no explicit reader. Adding something like @async while true wait(fd) will do what I need, but is not a solution I particularly endorse.

I know there’s no read timeout in libuv. I want to generate timeouts at a higher level. This is a fundamental property of “expect”. But to do so correctly I want to use libuv callbacks directly, as most of the julia runtime makes too many assumptions in how streams are handled.

The question is what would be the best way to register my own descriptors (along with custom read/write callbacks) into the main libuv loop.

4

I want to consume into a buffer indefinitely, even when there’s no explicit reader.

A reader is a buffer. You can’t consume into a buffer, even when there is no buffer.

But to do so correctly I want to use libuv callbacks directly, as most of the julia runtime makes too many assumptions in how streams are handled.

I think you’re making too many assumptions - Julia’s runtime is pretty much a plain wrapper around the libuv async primitives.

I want to generate timeouts at a higher level.

Sure, we have some examples of this in Base already, such as timeouts on starting addprocs or in the test code. Usually this is done by starting a Timer, and if it expires, have it close the client handle, as that is the most reliable way to avoid race conditions (for unreliable connections). Although, if you’re connecting over a socket, you can usually just ask the kernel to handle this for you by configuring the SO_KEEPALIVE (either directly with setsockopt, or per application, in your .ssh/config file for instance).

Is there a way to construct a PipeEndpoint manually from a RawFD that I don’t see?

You need to make an uninitialized p = PipeEndpoint(), open it with a ccall to jl_init_pipe, then associate it with the fd handle (ccall to uv_pipe_open)

5

But to do so correctly I want to use libuv callbacks directly, as
most of the julia runtime makes too many assumptions in how streams
are handled.

I think you’re making too many assumptions - Julia’s runtime is pretty
much a plain wrapper around the libuv async primitives.

I’m talking about LibuvStream/s (PipeEndpoint or TTY are pretty similar
in that regard). They do have an internal buffer, which is managed by
the collection of stream.jl functions. uv callbacks are enabled/disabled
depending on whether there’s someone on the notify queue. Am I
interpreting stream.jl wrong here?

unreliable connections). Although, if you’re connecting over a socket,
you can usually just ask the kernel to handle this for you by
configuring the SO_KEEPALIVE (either directly with setsockopt, or per
application, in your .ssh/config file for instance).

This is unrelated to the actual connection status of the stream. Local
pipes are always reliable. These timeouts are an higher-level
abstraction to manage an unknown stream protocol where the speed of the
data rate might be relevant.

You need to make an uninitialized p = PipeEndpoint(), open it with a
ccall to jl_init_pipe, then associate it with the fd handle (ccall to uv_pipe_open)

I’ll have a look.