# How to continuously communicate with an external program?

**URL:** <https://discourse.julialang.org/t/how-to-continuously-communicate-with-an-external-program/86319>\
**Category:** General Usage\
**Tags:** question\
**Created:** [August 25, 2022, 12:10pm UTC](https://discourse.julialang.org/t/how-to-continuously-communicate-with-an-external-program/86319 "2022-08-25T12:10:09Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ArthurW](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arthurw/32/19561_2.png) [@ArthurW](https://discourse.julialang.org/u/ArthurW)\
**Post date:** [August 25, 2022, 12:10pm UTC](https://discourse.julialang.org/t/how-to-continuously-communicate-with-an-external-program/86319/1 "2022-08-25T12:10:09Z")

</div>

Hi. I’m trying to write to the stdin of an external program and read from it’s stdout multiple times while it’s still executing.

For example, consider this C code that indefinetly reads an integer input `a` and outputs `a+1`:

```nohighlight
#include<stdio.h>

int main()
{
    int a;
    while(1)
    {
        scanf("%d", &a);
        printf("%d\n", a + 1);
    }
    return 0;
}

```

Does Julia support communication with this type of program?  
My current best guess is (`exec` is the filename of the compiled C program):

```nohighlight
julia> proc = open(`./exec`, write=true, read=true)
Process(`./exec`, ProcessRunning)

julia> println(proc, "12")

julia> readline(proc)

```

From there, `readline` stalls. Execution resumes if I read from `proc.out.buffer` instead, but the buffer is empty. Also, while I can get something if I close the process (or `proc.out`), this prevents further communication.

[This topic](https://discourse.julialang.org/t/avoiding-readavailable-when-communicating-with-long-lived-external-program/61611) gave me the impression that this usecase may not be supported in Julia. Is that the case?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![mbaz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbaz/32/17295_2.png) [@mbaz](https://discourse.julialang.org/u/mbaz)\
**Post date:** [August 25, 2022, 1:14pm UTC](https://discourse.julialang.org/t/how-to-continuously-communicate-with-an-external-program/86319/2 "2022-08-25T13:14:15Z")

</div>

It is possible to do so. As examples, both Gnuplot frontends ([Gnuplot.jl](https://github.com/gcalderone/Gnuplot.jl) and my own [Gaston.jl](https://github.com/mbaz/Gaston.jl)) work like this. But doing so is not straightforward. Here’s how I do it. I’ll assume gnuplot is the external executable I want to communicate with and, as a bonus, I’ll also read from its `stderr` 😉

First, start the process and connect its streams to Julia pipes:

```julia
"Returns a new gnuplot process."
function gp_start()
    inp = Base.PipeEndpoint()
    out = Base.PipeEndpoint()
    err = Base.PipeEndpoint()
    process = run(config.exec, inp, out, err, wait=false)

    return process
end

```

Now we need a function to send a message to the process and read its response. This part is tricky because you want to be able to read the entire response, but reading when there is no output will result in blocking and stop your code. My approach is to ask gnuplot to print what I call “sigils” to its `stdout` and `stderr`. Then, I read the content of the pipes until I detect the sigils. This guarantees that the code does not block, and also gives me confidence that I have read everything that gnuplot has to say.

```julia
"Send string `message` to `process` and handle its response."
function gp_send(process::Base.Process, message::String)
    message *= "\n"
    write(process, message) # send user input to gnuplot

    # ask gnuplot to return sigils when it is done
    write(process, """set print '-'
                      print 'GastonDone'
                      set print
                      print 'GastonDone'
                      """)

    gpout = readuntil(process, "GastonDone\n", keep=true)
    gperr = readuntil(process.err, "GastonDone\n", keep=true)

    # handle errors
    gpout == "" && @warn "gnuplot crashed."

    gperr = gperr[1:end-11]
    gperr != "" && @info "gnuplot returned a message in STDERR:" gperr

    return gpout, gperr
end

```

Finally, I have a function to quit the process:

```julia
function gp_quit(process::Base.Process)
    write(process, "exit gnuplot\n")
    close(process.in)
    wait(process)

    return process.exitcode
end

```

You will certainly have to adapt this code to your use case, but I hope this example points you in the right direction.

---

<div class="post-metadata">

**Author:** ![ArthurW](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arthurw/32/19561_2.png) [@ArthurW](https://discourse.julialang.org/u/ArthurW)\
**Post date:** [August 25, 2022, 9:20pm UTC](https://discourse.julialang.org/t/how-to-continuously-communicate-with-an-external-program/86319/3 "2022-08-25T21:20:40Z")

</div>

Thanks for your answer. I had seen your former topic on the subject, but, unfortunately, I cannot control the target program so I could not add a “sigil” to the output.  
What puzzles me is that the Julia code I put in the question seems “too pretty to be wrong”: it really feels like that is the intended use of `open(command, stdio=devnull; write::Bool, read::Bool)`.

---

<div class="post-metadata">

**Author:** ![mbaz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbaz/32/17295_2.png) [@mbaz](https://discourse.julialang.org/u/mbaz)\
**Post date:** [August 25, 2022, 9:48pm UTC](https://discourse.julialang.org/t/how-to-continuously-communicate-with-an-external-program/86319/4 "2022-08-25T21:48:27Z")

</div>

> [@ArthurW](#):
>
> I had seen your former topic on the subject,

Heh, I didn’t realize I had already posted my code on that older topic 😅

I understand your puzzlement, and of course the lack of official documentation does not help.

---

<div class="post-metadata">

**Author:** ![ArthurW](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arthurw/32/19561_2.png) [@ArthurW](https://discourse.julialang.org/u/ArthurW)\
**Post date:** [August 25, 2022, 9:59pm UTC](https://discourse.julialang.org/t/how-to-continuously-communicate-with-an-external-program/86319/5 "2022-08-25T21:59:57Z")

</div>

It’s the buffering of the external program’s output!  
Since Julia doesn’t call the external program via a terminal, it’s output is _block-buffered_: it will only be writen to _stdout_ once there is enough data to be worth the trouble.

In the end, it’s sufficient to get the IO of the external program to be _line-buffered_ (or not buffered at all, but it would be less eficient). That is, to have the output of the program to be effectively written to it’s out-stream (`proc.out`) after every line break (`\n`). There are multiple ways to instruct the system to do this. I’ve achieved it via the `stdbuf` command:

```nohighlight
julia> proc = open(`stdbuf -oL ./exec`, write=true, read=true)
Process(`stdbuf -oL ./exec`, ProcessRunning)

julia> println(proc, "12")

julia> readline(proc)
"13"

julia> println(proc, 23)

julia> readline(proc)
"24"

```

(Notice that all of this means that adding an `fflush(stdout)` to the C code in the question would fix the issue. But, in pratice, this wasn’t an option for me since cannot modify the external program.)

---

<div class="post-metadata">

**Author:** ![mbaz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbaz/32/17295_2.png) [@mbaz](https://discourse.julialang.org/u/mbaz)\
**Post date:** [August 25, 2022, 11:48pm UTC](https://discourse.julialang.org/t/how-to-continuously-communicate-with-an-external-program/86319/6 "2022-08-25T23:48:09Z")

</div>

Excellent find! I wonder if there are similar solutions for Windows and Mac.