Avoiding `readavailable()` when communicating with long-lived external program

This is my current implementation; it’s much shorter and simpler than before. I’d appreciate if you could take a look at it and let me know if it can still be improved:

"Start gnuplot process and start reading tasks."
function startgp()
    inp = Base.PipeEndpoint()
    out = Base.PipeEndpoint()
    err = Base.PipeEndpoint()

    process = run(`gnuplot`, inp, out, err, wait=false)

    return (process, inp, out, err)
end

"End gnuplot process `p`"
function endgp(p)
    write(p[2], "exit gnuplot\n")
    close(p[2])
    wait(p[1])
    return p[1].exitcode
end

"Send `input` to process `p` and return its response."
function communicate(p, input)
    inp = p[2]
    out = p[3]
    err = p[4]

    # send user input to gnuplot
    write(inp, input)

    # ask gnuplot to return sigils when it is done
    write(inp, """\nset print '-'
                   print 'GastonDone'
                   set print
                   print 'GastonDone'\n""")

    gpout = readuntil(out, "GastonDone\n", keep=true)
    gperr = readuntil(err, "GastonDone\n", keep=true)

    return (gpout, gperr)
end
2 Likes

For simplicity/readability purposes, you might just want startgp() to return just “process”. (::Process objects already have all 3 streams as parameters).

The problem was that they weren’t available to process::Process when you specified them in the pipeline(`gnuplot`, stdin=inp, stdout=out, stderr=err) command. Good news though: they get transferred properly when they are specified in the call to run() instead.

STDIN

In fact:

println(process, "exit gnuplot")

sends "exit gnuplot" it to gnuplot's STDIN (forwards println() to process.in).

STDOUT

And:

nextline = readline(process)

forwards the readline() command to process.out.

STDERR

Though to get data out of STDERR, you need to be a bit more explicit:

nextline = readline(process.err)
1 Like

Works beautifully and the code is more readable. Thanks a lot!

1 Like

how it this going please? I think I might have a similar problem and would appreciate seeing how your code ended up. So if you could post the working code?

I’m trying to take the output from a python script and process it in julia. I am new to julia and the code you posted in this exchange was a great help.

First of all, I’d recommend avoiding communication via stdin/stdout if at all possible (for example, use files instead, or just call the Python functions using PyCall or similar).

Having said that, here’s my code for communicating with gnuplot, in case it’s useful to you.

I start gnuplot with:

function gp_start()
    inp = Base.PipeEndpoint()
    out = Base.PipeEndpoint()
    err = Base.PipeEndpoint()
    process = run(`gnuplot`, inp, out, err, wait=false)

    return process
end

and quit with:

function gp_quit(process::Base.Process)
    write(process, "exit gnuplot\n")
    close(process.in)
    wait(process)

    return process.exitcode
end

I send a message to gnuplot using the function below. After writing the message, I ask gnuplot to echo a "sentinel" string, and wait until the echo is received.

"Send string `message` to `process` and handle its response."
function gp_send(process::Base.Process, message::String)
    message *= "\n"
    write(process, message) # send user input to gnuplot

    # ask gnuplot to return sentinel when it is done
    write(process, "set print '-'; print 'sentinel'")

    gpout = readuntil(process, "sentinel\n", keep=true)

    # handle errors
    gpout == "" && @warn "gnuplot crashed."

    return gpout
end
1 Like

Thank you @mbaz for posting the code. I learn by example and your code is something I can use to examine my 2022 approach ( first julia). My usecase is to get a stream of data from existing python scripts and process it inside julia.

I am VERY interested in PyCall and am progressing down that route today. I can certainly see other areas where your approach code be used in my approach.

thanks again

1 Like

hi there
do you have a sample julia session for running this environment please?
thank you