C program embedding Julia - works in terminal but breaks in pipes

General Usage
, , ,
1

I’d like to write a C program which embeds Julia code, and call this C program from an unrelated program using two-way pipes. While the C program works fine in the terminal, it breaks when called from pipes. Here’s a minimal working example, where the C program initializes the Julia runtime but does not run any actual Julia code. Instead, it runs C code which repeatedly reads integers from stdin and prints them out to stdout:

/* test1.c */
#include <julia.h>
#include <stdio.h>
JULIA_DEFINE_FAST_TLS

int main(int argc, char *argv[])
{
    int i;
    jl_init();
    
    while (!feof(stdin)) {
        scanf("%d", &i);
        printf("%d\n", i);
    }

    jl_atexit_hook(0);
    return 0;
}

I compiled it with

gcc -o test1 -fPIC -I$JULIA_DIR/include/julia -L$JULIA_DIR/lib -Wl,-rpath,$JULIA_DIR/lib test1.c -ljulia

where JULIA_DIR points to my Julia 1.8.0 installation.

The program runs fine in the terminal: whenever I type in any integer, the same integer is printed out, as expected. However, when I try to call it from another program via two-way pipes, it breaks badly. I’d like to interface with a different program, but here I’ll just show the problem by calling the program from a Julia REPL:

julia> io = open(`./test1`, "r+")
Process(`./test1`, ProcessRunning)

julia> println(io, "10")

julia> readline(io)
"32766"

You can see that the output is completely wrong and seems to be an uninitialized integer. I strongly suspects that there is some conflict between Julia’s handling of stdin / stdout and stdio.h in C, because I don’t experience this problem when my C program only performs IO by executing Julia code containing e.g., print(). Is it possible that there should be some syncing between the IO buffers in C and in Julia?

2 Likes
2

Running this with

/* test1.c */
#include <stdio.h>
#include <julia.h>
JULIA_DEFINE_FAST_TLS

int main(int argc, char *argv[])
{
    int i = 1337;
    jl_init();

    while (!feof(stdin)) {
        scanf("%d", &i);
        printf("%d\n", i);
        fflush(stdout);
    }

    jl_atexit_hook(0);
    return 0;
}

Here’s the julia code in test1.jl

# test1.jl
io = open(`./test1`, "r+")
println(io, "10")
println(readline(io))
close(io)

Then I get the following result:

$ julia test1.jl
10

In the Julia REPL, I get the following result:

julia> begin
           io = open(`./test1`, "r+")
           println(io, "10")
           println(readline(io))
           close(io)
       end
10

julia> function f(n)
           io = open(`./test1`, "r+")
           println(io, "$n")
           println(readline(io))
           close(io)
       end
f (generic function with 1 method)

julia> f(900)
900

julia> io = open(`./test1`, "r+")
Process(`./test1`, ProcessRunning)

julia> println(io, "100");

julia> readline(io)
"1337"

julia> close(io)

If I do not do jl_init() then there is not issue:

/* test2.c */
#include <stdio.h>
#include <julia.h>
JULIA_DEFINE_FAST_TLS

int main(int argc, char *argv[])
{
    int i = 1337;
    //jl_init();

    while (!feof(stdin)) {
        scanf("%d", &i);
        printf("%d\n", i);
        fflush(stdout);
    }

    //jl_atexit_hook(0);
    return 0;
}

julia> io = open(`./test2`, "r+")
Process(`./test2`, ProcessRunning)

julia> println(io, "100");

julia> readline(io)
"100"

julia> close(io)
3

I can break it with a single extra line, like this:

# test1_modified.jl
io = open(`./test1`, "r+")
sleep(1)
println(io, "10")
println(readline(io))
close(io)

Running julia test1_modified.jl now prints 1337. Adding sleep() probably makes it similar to the delay of a human user in your REPL example, which also prints out an unexpected result.

4

One brief observation is that I’m not sure if stdin is still in buffering mode. Effectively, scanf may not be blocking.

5

Introduction

The buffering modes on stdin and stdout are different upon including julia.h. They are no longer in line buffer mode. See setvbuf.

This means that we need to check the status output of scanf before we print anything. It’s possible that scanf read nothing and did not update i. Thus we need to alter the C code as follows.

Corrected code

/* test5.c */
#include <stdio.h>
#include <julia.h>
JULIA_DEFINE_FAST_TLS

int main(int argc, char *argv[])
{
    int i = 1337;
    int s = 0;
    jl_init();

    while (!feof(stdin)) {
        s = scanf("%d", &i);
        if (s > 0) {
            printf("%d\n", i);
            fflush(stdout);
        }
    }

    jl_atexit_hook(0);
    return 0;
}

Above we do the following.

  1. Check if scanf actually read a value ands stored it in i
  2. If so, print to stdout and flush stdout.

Reading the output in Julia, we now get the expected result.

julia> io = open(`./test5`, "r+")
Process(`./test5`, ProcessRunning)

julia> println(io, 10)

julia> println(readline(io))
10

julia> println(io, 30)

julia> println(readline(io))
30

julia> close(io)

Prior code analysis

Previously, scanf was basically reading nothing and not updating i. It would print many lines. Consider the following code.

/* test6.c */
#include <stdio.h>
#include <julia.h>
JULIA_DEFINE_FAST_TLS

int main(int argc, char *argv[])
{
    int i = 1337;
    jl_init();

    while (!feof(stdin)) {
        i++;
        scanf("%d", &i);
        printf("%d\n", i);
        fflush(stdout);
    }

    jl_atexit_hook(0);
    return 0;
}

If we read from this in Julia, we could get the following:

julia> io = open(`./test6`, "r+")
Process(`./test6`, ProcessRunning)

julia> readline(io)
"1338"

julia> readline(io)
"1339"

julia> readline(io)
"1340"

julia> readline(io)
"1341"

julia> readline(io)
"1342"

julia> readline(io)
"1343"

julia> readline(io)
"1344"

julia> readline(io)
"1345"

julia> begin
           io = open(`./test6`, "r+")
           for line in Iterators.take(eachline(io), 100)
               println(line)
           end
       end
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
...
2 Likes
6

Thanks for looking deep into this. That’s very illuminating. Now the slight annoyance is that after running

julia> io = open(`./test5`, "r+")

the process test5 runs with 100% CPU usage. Presumably scanf is running continuously until it finds an integer input.

I guess the workarounds are either doing all IO on the Julia side or avoiding pipes.

1 Like
7

Just use setvbuf to set the buffering to whatever you would like.

8

I added

setvbuf(stdin, NULL, _IOLBF, 16384);

right after

jl_init();

to change stdin back to line-buffering mode, but this doesn’t seem to fix the CPU usage when idle. Maybe I got it wrong? I suspect it’s necessary to dig into libuv to fully resolve this, but I know too little about it.

P.S. I can get by without pipes. Still I’m just curious about the problem.

1 Like