Read stdout/stderr from subprocess asynchronously

General Usage
1

I want to call a process that might write to its stdout or stderr however it chooses. The process could take a while, and it might die (exit with nonzero status).

Conceptually, I want something like this:

cmd = `sh -c 'echo "My stdout"; echo "My stderr" 1>&2; sleep 5; exit 1'`
open(cmd) do io  # WRONG - doesn't capture stderr
    for line in eachline(io)
        @info "sub> $line"
    end
end

I want:

  1. The sub> ... outputs should appear immediately, not wait for the process to finish
  2. I don’t need to distinguish between stdout and stderr on incoming lines
  3. I do need to distinguish between a zero & nonzero exit code - but if the process had spewed some lines before dying, I need to make sure I’ve captured all of them in my eachline() loop

I toyed around with creating a custom IO subtype that just does @info "sub> $line" whenever it receives any input, and using it as both stdout and stderr for a pipeline, but I couldn’t get my IO to be full-featured enough to work in this way (it would be great if there were an interface for IO objects).

Any advice for getting this to work?

2

My colleagues and I ended up woodshedding this for a few days and it looks like this will fit the bill:

function live_subprocess(cmd::Cmd; handle_line=(line -> @info "sub> $line"), die=true)
    pipe = Pipe()
    process = run(pipeline(cmd, stdout=pipe, stderr=pipe), wait=false)
    close(pipe.in)  # To allow EOF detection on the reading side

    read_thread = Base.Threads.@spawn \
    for line in eachline(pipe)
        handle_line(line)
    end

    wait(process)
    wait(read_thread)

    if die && process.exitcode != 0
        error("Subprocess failed with exit code $(process.exitcode)")
    end

    return process.exitcode
end

I’d definitely still be interested in some peer review if anyone has comments.