A minimal example with Base.redirect_stdout

General Usage
1

Hello all

I’m having a bit of difficulty making the Base.redirect_stdout(f::Function, stream) method work. What I would have thought would be a minimal working example is in fact a minimal failing example:

out = Pipe()
redirect_stdout(out) do s
    println("stick this in your pipe")
end

This gives me
ERROR: MethodError: no method matching redirect_stdout(::Pipe)

Who’s in the wrong here - me, or Julia? Any help much appreciated. It seems like a nifty function and it’s annoying not to be able to use it.

Link to the docs below:
https://docs.julialang.org/en/v1/base/io-network/#Base.redirect_stdout-Tuple{Function,%20Any}

1 Like
2

Redirection to Pipe is not supported on Julia 1.6. You can redirect to a file stream:

julia> open("/tmp/stdout", "w") do io
           redirect_stdout(io) do
               println("hello, world")
           end
       end

julia> read("/tmp/stdout", String)
"hello, world\n"

In the upcoming Julia 1.7 Pipe is supported, though (JuliaLang/julia#39132).

3

Why though? The docs very clearly state that it is:
https://docs.julialang.org/en/v1/base/io-network/#Base.redirect_stdout-Tuple{Function,%20Any}

4

Bug in the documentation (JuliaLang/julia#31159).

5

OK. Seems like a handy function when it’s ready.

6

This works in Julia 1.7-beta2:

pipe = Pipe()
output = IOBuffer()
cmd = run(pipeline(pipe, `wc -l`, result), wait=false)
redirect_stdout(pipe) do
    print(pipe, "a\nb\n")
end

close(pipe)
wait(cmd)

julia> result = String(take!(output))
"2\n"

You can also read directly from the pipe, but don’t forget that pipe writes will block if the pipe is full (because the reader is slower), so you need something like this:

pipe = Pipe()
writer = @async redirect_stdout(pipe) do
    write(stdout, String(rand(['a':'z'; '\n'], 100_000)))
    close(Base.pipe_writer(pipe))
end

result = read(pipe, String)
wait(writer)  # Not really necessary but cleaner

It’s maybe a bit hacky to use the unexported pipe_writer function, and to close the stream before the end of redirect_stdout… This should all be made simpler, see the following issues:

1 Like
7

In 1.8.3 it gives:
ArgumentError: Base.PipeEndpoint(RawFD(4294967295) init, 0 bytes waiting) is not initialized

8

Oops that’s because this code has a race condition. The pipe object must be initialized by redirect_stdout before it’s used in the read(pipe, String) call. But the order of execution is undefined here: Julia can decide to start executing the async task before read (1.7 behavior apparently), or it can start executing read before the async task (1.8 behavior).

To fix the race condition we can add some explicit synchronization:

pipe = Pipe()
started = Base.Event()
writer = @async redirect_stdout(pipe) do
    notify(started)
    write(stdout, String(rand(['a':'z'; '\n'], 100_000)))
    close(Base.pipe_writer(pipe))
end


wait(started)
result = read(pipe, String)
wait(writer) 
println(result)

(By the way, the first example in that same message had an error: it should be pipeline(pipe, `wc -l`, output) instead of pipeline(pipe, `wc -l`, result), but I can’t edit it anymore.)

1 Like
9

Is this not equivalent to not using an asynchronous task at all?

10

No: without an asynchronous task, the code in redirect_stdout has to finish before we start reading. I expect that the write will block because the pipe will be full before all data is written, so the whole program will get stuck.

With the asynchronous task, the read call can start consuming the pipe as the write call is still running and feeding more data to the pipe.

In other words: the async task with synchronization makes sure we don’t read before the producer starts. Having no async task at all would delay reading until the producer finishes (which will never happen if the data doesn’t fit in the pipe buffer).

11

I see, thank you