Reading lines from a pipe without hanging at the end

General Usage
1

Suppose you want to run a command, read the output, and return some processed version of the output…

for example this code parses the output of “traceroute”…


function gethops(dname)

    trin = Pipe()
    trout = Pipe()
    trerr = Pipe()
    trproc = run(pipeline(`traceroute -n -w 0.5 $dname`; stdin=trin,stdout=trout,stderr=trerr),wait=false)

    hops = DataFrame()
    @async (sleep(5);close(trout));
    readline(trout) ## throw away the first line
    while true
        if(!eof(trout))
            line = lstrip(readline(trout));
            s = split(line,r"[ *]+")
            push!(hops,(n=s[1],addr=s[2],ms=s[3]))
            #@show hops
        else
            break;
        end
    end
    return hops
end

gethops("www.google.com")

Without the @async (sleep(5);close(trout) the function hangs because after the command exits the pipe is not closed. That seems weird. What’s the right way to read all the lines output by a command without hanging after the task ends?

(note, I could also do @async(wait(trproc); close(trout)) which might make more sense than a fixed timeout… though I could also imagine a process that hangs without exiting… and we want a timeout as well? another @async?)

2

Would while process_running(trproc) work for your application?

IIUC EOF means that all potential writers have exited or closed the descriptor, but Julia (or libuv?) seems to preserve pipes as zombies.

1 Like
3

This seems to make sense. The zombie pipe is a little weird though, once the process is done and all text has been read, what else could write to the pipe? Shouldn’t it be eof?

4

On second thought, my suggestion is wrong because of races. Better solutions I can think of are not properly documented.

The conventional (POSIX) approach is to call

close(Base.pipe_writer(trout))

after the run command.

It turns out that libuv provides one-sided pipes, so the alternative is

trout = Base.PipeEndpoint()

Both of these have the effect that eof(trout) only depends on the subprocess, so your logic should work.

The zombie issue is that otherwise the pipe end held by Julia isn’t closed until a finalizer runs when the Pipe object gets garbage-collected.

1 Like