How to continuously communicate with an external program?

General Usage
1

Hi. I’m trying to write to the stdin of an external program and read from it’s stdout multiple times while it’s still executing.

For example, consider this C code that indefinetly reads an integer input a and outputs a+1:

#include<stdio.h>

int main()
{
    int a;
    while(1)
    {
        scanf("%d", &a);
        printf("%d\n", a + 1);
    }
    return 0;
}

Does Julia support communication with this type of program?
My current best guess is (exec is the filename of the compiled C program):

julia> proc = open(`./exec`, write=true, read=true)
Process(`./exec`, ProcessRunning)

julia> println(proc, "12")

julia> readline(proc)

From there, readline stalls. Execution resumes if I read from proc.out.buffer instead, but the buffer is empty. Also, while I can get something if I close the process (or proc.out), this prevents further communication.

This topic gave me the impression that this usecase may not be supported in Julia. Is that the case?

Thanks in advance!

1 Like
2

It is possible to do so. As examples, both Gnuplot frontends (Gnuplot.jl and my own Gaston.jl) work like this. But doing so is not straightforward. Here’s how I do it. I’ll assume gnuplot is the external executable I want to communicate with and, as a bonus, I’ll also read from its stderr :wink:

First, start the process and connect its streams to Julia pipes:

"Returns a new gnuplot process."
function gp_start()
    inp = Base.PipeEndpoint()
    out = Base.PipeEndpoint()
    err = Base.PipeEndpoint()
    process = run(config.exec, inp, out, err, wait=false)

    return process
end

Now we need a function to send a message to the process and read its response. This part is tricky because you want to be able to read the entire response, but reading when there is no output will result in blocking and stop your code. My approach is to ask gnuplot to print what I call “sigils” to its stdout and stderr. Then, I read the content of the pipes until I detect the sigils. This guarantees that the code does not block, and also gives me confidence that I have read everything that gnuplot has to say.

"Send string `message` to `process` and handle its response."
function gp_send(process::Base.Process, message::String)
    message *= "\n"
    write(process, message) # send user input to gnuplot

    # ask gnuplot to return sigils when it is done
    write(process, """set print '-'
                      print 'GastonDone'
                      set print
                      print 'GastonDone'
                      """)

    gpout = readuntil(process, "GastonDone\n", keep=true)
    gperr = readuntil(process.err, "GastonDone\n", keep=true)

    # handle errors
    gpout == "" && @warn "gnuplot crashed."

    gperr = gperr[1:end-11]
    gperr != "" && @info "gnuplot returned a message in STDERR:" gperr

    return gpout, gperr
end

Finally, I have a function to quit the process:

function gp_quit(process::Base.Process)
    write(process, "exit gnuplot\n")
    close(process.in)
    wait(process)

    return process.exitcode
end

You will certainly have to adapt this code to your use case, but I hope this example points you in the right direction.

4 Likes
3

Thanks for your answer. I had seen your former topic on the subject, but, unfortunately, I cannot control the target program so I could not add a “sigil” to the output.
What puzzles me is that the Julia code I put in the question seems “too pretty to be wrong”: it really feels like that is the intended use of open(command, stdio=devnull; write::Bool, read::Bool).

1 Like
4

Heh, I didn’t realize I had already posted my code on that older topic :sweat_smile:

I understand your puzzlement, and of course the lack of official documentation does not help.

1 Like
5

It’s the buffering of the external program’s output!
Since Julia doesn’t call the external program via a terminal, it’s output is block-buffered: it will only be writen to stdout once there is enough data to be worth the trouble.

In the end, it’s sufficient to get the IO of the external program to be line-buffered (or not buffered at all, but it would be less eficient). That is, to have the output of the program to be effectively written to it’s out-stream (proc.out) after every line break (\n). There are multiple ways to instruct the system to do this. I’ve achieved it via the stdbuf command:

julia> proc = open(`stdbuf -oL ./exec`, write=true, read=true)
Process(`stdbuf -oL ./exec`, ProcessRunning)

julia> println(proc, "12")

julia> readline(proc)
"13"

julia> println(proc, 23)

julia> readline(proc)
"24"

(Notice that all of this means that adding an fflush(stdout) to the C code in the question would fix the issue. But, in pratice, this wasn’t an option for me since cannot modify the external program.)

5 Likes
6

Excellent find! I wonder if there are similar solutions for Windows and Mac.