Issues with println buffering output when redirecting stdout to a txt file

General Usage
1

Hey all,

I’m attempting to find the julia definition for the println method as I want to replace it with my own custom println method that performs a flush(stdout) after every println

I’ve found this but it doesn’t seem to actually define the println function.

Here is what I have so far:

#Hack to work around buffered stdout prints
import Base.println
println(s::String) = begin
    println(stdout, s)
    flush(stdout)
end

This works fine… until someone deliberately redirects the stdout to dev/nul, but my func overrrides that and always prints to stdout.

Alternatively… Is there a better way to do what I’m attempting?

Jon

2

I also tried the following, but it gives a stack overflow err:

import Base.println
original_println = Base.println
println(s::String) = begin
    original_println(">$s")
    flush(stdout)
end
3

Regarding println: since it has multiple methods, you cannot find the definition. Eg methods(println) should list them all.

That said, overwriting a method in Base is almost surely not the way you want to do this. I asked a similar question a while ago, you may find the solution helpful:

Or, if this is not for display purposes, you can define a wrapper type for an <:IO that captures the \n, and flushes. This is a nice example of something similar:

4

For most output stream types, println should already be line-buffered. What are you writing to?

5

Hi Stefan/Tamas,

Thanks for taking the time to respond.

The reason i’m trying to force the flush is that I’m executing a julia compiled .EXE (PackageCompiler) in an HPC environment, where the STDOUT is captured to a .txt file

Without the println/flush hack I posted above, it will buffer all the stdout until right at the end when the process exits. Then the stdout.txt is written to.

With the flush hack, the stdout.txt is instantly written to disk as lines become available.

Here’s what I’m about to test now:

import Base.println
println(s::String) = begin
    show(s)
    flush(stdout)
end
6

When implementing any IO method, you want to implement the method that takes an initial io::IO argument. So you’d want to implement:

import Base.println
function println(io::IO, s::String)
    println(io, s)
    flush(io)
end

Of course, doing this sort of “monkey patching” is fragile (at best) because it effects everyone, even unrelated packages. In this case it’s probably fine, but it might cause hard-to-track-down surprises down the road.

7

Hey Matt - thanks for the input. I’m aware of the danger - as you said though I’m hoping that this is relatively side effect free.

Unfortunately your sample causes a stack overflow exception - is there a way of calling the original method when overwriting the method?

8

Oh of course — yes, the way that I’d go about this is to shadow println with your own definition. That will allow you to still refer to the base method and only apply to your own namespaces:

println(args...) = println(stdout, args...)
println(io::IO, args...) = Base.println(io, args...)
function println(io::IO, s::String)
    Base.println(io, s)
    flush(io)
end

Note that you must define this before referring to println as once you’ve called Base’s implementation println you’re no longer able to shadow it with your own definition.

1 Like
9

Unfortunately this stack overflows for me too - however I did have to add in import Base.println before the code otherwise it wouldn’t compile. Is this preventing the shadow?

image
image971×268 11.4 KB

10

Yes, that’s what I was saying with:

Note that you must define this before referring to println as once you’ve called Base’s implementation println you’re no longer able to shadow it with your own definition.

You’ll need to restart your Julia session or create a new module to allow the shadowing of println.

1 Like
11

Or simply call it something else (e.g. flushprintln)

1 Like
12

That would be nice, unfortunately we have a few thousand lines of julia code that i don’t want to do a find/replace in :slight_smile:

13

Unfortunately it is not very well worked out or documented which kind of streams are buffered how, and by whom (buffered on the julia side? in ios.c? by libuv?).

A spectacularly ugly workaround is to keep your code and just invoke $ julia somefile.jl | tee output.txt >/dev/null instead of $ julia somefile.jl > output.txt. This will cause stdout to be a PipeEndpoint, and PipeEndpoint is by default unbuffered. I consider this a bug, but sometimes xkcd: Workflow is the way to go.

PS. You find the definition via @less println("AA"). Apart from the @less macro, you can use

julia> methods(println)
# 3 methods for generic function "println":
[1] println(io::IO) in Base at coreio.jl:5
[2] println(io::IO, xs...) in Base at strings/io.jl:69
[3] println(xs...) in Base at coreio.jl:4

After reading the two lines in coreio.jl, you could come up with

julia> @eval Base println(xs...) = begin res = println(stdout::IO, xs...); flush(stdout); return res end;
julia> @eval Base println(io::IO) =  begin res =  print(io, '\n'); flush(io); return res end;

You can verify that this works with

$ strace julia -e 'for i=1:10_000 println("AA") end' 1>output 2>trace
$ wc -l output
10000 output
$ wc -l trace 
884 trace

$ strace julia -e '@eval Base println(xs...) = begin res = println(stdout::IO, xs...); flush(stdout); return res end; @eval Base println(io::IO) =  begin res =  print(io, '\n'); flush(io); return res end; for i=1:10_000 println("AA") end' 1>output 2>trace
$ wc -l output
10000 output
$ wc -l trace 
10884 trace
3 Likes
14

Using a wrapper type would be a clean solution. Eg (just a starting point, modify/extend as needed):

struct FlushingIO{T}
    io::T
end

function Base.print(io::FlushingIO, args...)
    print(io.io, args...)
    flush(io.io)
end

out = FlushingIO(stdout)
my_function(out, ...)
1 Like
15

@Tamas_Papp and @all Is this possible with a stderr? How to use it for example to write to a file?

1 Like