Reading from either stdout or stderr of spawned process

General Usage
,
1

My code runs an external process (in this case, Pari/GP) which might produce output on either its stdout or stderr file descriptors. I communicate with the process in the following way:

pin, pout = Pipe(), Pipe()
process = run(pipeline(`gp -q -D colors=no`; stdin=pin, stdout=pout); wait=false)
println(pin, "2+2")
readline(pout) # returns "4"

However, I also want to read the stderr messages from the process. If I append stderr=pout to the call above, I get them in the same Pipe object as the stdout messages, which is not desirable.

In C I would use the select(2) system call for this: given a set of FDs, it waits until one of these has data ready (and of course says which one(s) it is). I don’t see anything equivalent in the IO package, but perhaps I missed something? Is there another way to do I/O multiplexing?

2

Why not use a third Pipe?

3

OP wants to use a third Pipe. Then he wants his Task to block, until either pout or perr has a full line available. This is a quite natural request.

An obvious way to do this is to set up a channel, and two extra tasks; one of which waits on pout and pushes lines into the channel (plus metadata indicating that the line came from pout), and another one which waits on perr and does basically the same thing. OPs main task can then wait on the channel, and consume the lines.

That being said, this does feel a little heavy-weight, both on allocs/performance and on ceremony. The performance is not that bad: These tasks are green threads, not OS threads.

OP already stated another obvious way: Fire off the desired syscall. That would be quite annoying, because you’d be working against the julia runtime. OP directly asked for an appropriate julia wrapper/interface to this syscall, and I don’t know one.

In the same vein: Julia pipe objects use libuv. You could “simply” bind an event handler / callback to the pipe, bypassing julia’s machinery. But that is (1) also working against the runtime, and (2) also would work via a channel, you’d just skip the julia Task switching and instead put data into the channel directly from the callback. (or it would work by notify inside the callback on a Condition that the main task waits on. Same story.)

4

Thanks! Here is what I came up with.

module Test 
mutable struct Command{X} 
  process::X 
  in::Pipe 
  out::Channel{Tuple{Int,String}} 
end 
function put_io(process, io, n, channel) 
  while process_running(process) 
    put!(channel, (n, readline(io))) 
  end 
end 
function Command(cmd) 
  pin, pout, perr = Pipe(), Pipe(), Pipe() 
  process = run(pipeline(cmd; stdin=pin, stdout=pout, stderr=perr); 
    wait=false) 
  @assert process_running(process) 
  close(pin.out); close(pout.in); close(perr.in) 
  channel = Channel{Tuple{Int,String}}(16) 
  schedule(@task put_io(process, pout, 1, channel)) 
  schedule(@task put_io(process, perr, 2, channel)) 
  return Command(process, pin, channel) 
end 
function ask(cmd::Command, s...) 
  println(cmd.in, s...) 
  wait(cmd.out) 
  return take!(cmd.out) 
end 
end # module Test