# Issues with println buffering output when redirecting stdout to a txt file

**URL:** <https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738>\
**Category:** General Usage\
**Created:** [May 1, 2019, 3:12pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738 "2019-05-01T15:12:45Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![jonbarkerlondon](https://avatars.discourse-cdn.com/v4/letter/j/9dc877/32.png) [@jonbarkerlondon](https://discourse.julialang.org/u/jonbarkerlondon)\
**Post date:** [May 1, 2019, 3:12pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/1 "2019-05-01T15:12:45Z")

</div>

Hey all,

I’m attempting to find the julia definition for the `println` method as I want to replace it with my own custom `println` method that performs a `flush(stdout)` after every `println`

I’ve found [this](https://github.com/JuliaLang/julia/blob/17c4b0213aabd654da132e8dbe62edee73de5672/base/coreio.jl) but it doesn’t seem to actually define the println function.

Here is what I have so far:

```julia
#Hack to work around buffered stdout prints
import Base.println
println(s::String) = begin
    println(stdout, s)
    flush(stdout)
end

```

This works fine… until someone deliberately redirects the stdout to dev/nul, but my func overrrides that and always prints to stdout.

Alternatively… Is there a better way to do what I’m attempting?

Jon

---

<div class="post-metadata">

**Author:** ![jonbarkerlondon](https://avatars.discourse-cdn.com/v4/letter/j/9dc877/32.png) [@jonbarkerlondon](https://discourse.julialang.org/u/jonbarkerlondon)\
**Post date:** [May 1, 2019, 3:19pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/2 "2019-05-01T15:19:52Z")

</div>

I also tried the following, but it gives a stack overflow err:

```julia
import Base.println
original_println = Base.println
println(s::String) = begin
    original_println(">$s")
    flush(stdout)
end

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 1, 2019, 3:48pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/3 "2019-05-01T15:48:31Z")

</div>

Regarding `println`: since it has multiple methods, you cannot find _the_ definition. Eg `methods(println)` should list them all.

That said, overwriting a method in `Base` is almost surely not the way you want to do this. I asked a similar question a while ago, you may find the solution helpful:

> [@Avoiding very long output](https://discourse.julialang.org/t/avoiding-very-long-output/23658):
>
> Occasionally, for custom types, I forget to define a show method that respects limit. This can happen accidentally even with the best intentions, eg with unfortunate combinations of nested types. This can lead to very long output in the REPL, which not all terminals and interfaces are happy with. I am wondering it it would be possible to somehow customize the REPL or the display infrastructure to catch this. Eg when working the REPL, somehow impose a sanity check of cutting off at eg line 100,…

Or, if this is not for display purposes, you can define a wrapper type for an `<:IO` that captures the `\n`, and flushes. This is a nice example of something similar:

> **[GitHub - KristofferC/IOIndents.jl: Simple writing of indented and aligned text](https://github.com/KristofferC/IOIndents.jl)**
>
> Simple writing of indented and aligned text. Contribute to KristofferC/IOIndents.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [May 1, 2019, 3:51pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/4 "2019-05-01T15:51:44Z")

</div>

For most output stream types, println should already be line-buffered. What are you writing to?

---

<div class="post-metadata">

**Author:** ![jonbarkerlondon](https://avatars.discourse-cdn.com/v4/letter/j/9dc877/32.png) [@jonbarkerlondon](https://discourse.julialang.org/u/jonbarkerlondon)\
**Post date:** [May 1, 2019, 3:54pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/5 "2019-05-01T15:54:59Z")

</div>

Hi Stefan/Tamas,

Thanks for taking the time to respond.

The reason i’m trying to force the flush is that I’m executing a julia compiled .EXE (PackageCompiler) in an HPC environment, where the STDOUT is captured to a .txt file

Without the println/flush hack I posted above, it will buffer all the stdout until right at the end when the process exits. Then the stdout.txt is written to.

With the flush hack, the stdout.txt is instantly written to disk as lines become available.

Here’s what I’m about to test now:

```julia
import Base.println
println(s::String) = begin
    show(s)
    flush(stdout)
end

```

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [May 1, 2019, 3:58pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/6 "2019-05-01T15:58:48Z")

</div>

When implementing _any_ IO method, you want to implement the method that takes an initial `io::IO` argument. So you’d want to implement:

```julia
import Base.println
function println(io::IO, s::String)
    println(io, s)
    flush(io)
end

```

Of course, doing this sort of “monkey patching” is fragile (at best) because it effects everyone, even unrelated packages. In this case it’s probably fine, but it might cause hard-to-track-down surprises down the road.

---

<div class="post-metadata">

**Author:** ![jonbarkerlondon](https://avatars.discourse-cdn.com/v4/letter/j/9dc877/32.png) [@jonbarkerlondon](https://discourse.julialang.org/u/jonbarkerlondon)\
**Post date:** [May 1, 2019, 4:01pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/7 "2019-05-01T16:01:04Z")

</div>

Hey Matt - thanks for the input. I’m aware of the danger - as you said though I’m hoping that this is relatively side effect free.

Unfortunately your sample causes a stack overflow exception - is there a way of calling the original method when overwriting the method?

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [May 1, 2019, 4:07pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/8 "2019-05-01T16:07:44Z")

</div>

Oh of course — yes, the way that I’d go about this is to shadow println with your own definition. That will allow you to still refer to the base method and only apply to your own namespaces:

```julia
println(args...) = println(stdout, args...)
println(io::IO, args...) = Base.println(io, args...)
function println(io::IO, s::String)
    Base.println(io, s)
    flush(io)
end

```

Note that you must define this _before_ referring to `println` as once you’ve called Base’s implementation `println` you’re no longer able to shadow it with your own definition.

---

<div class="post-metadata">

**Author:** ![jonbarkerlondon](https://avatars.discourse-cdn.com/v4/letter/j/9dc877/32.png) [@jonbarkerlondon](https://discourse.julialang.org/u/jonbarkerlondon)\
**Post date:** [May 1, 2019, 4:15pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/9 "2019-05-01T16:15:56Z")

</div>

> [@mbauman](#):
>
> println(args…) = println(stdout, args…) println(io::IO, args…) = Base.println(io, args…) function println(io::IO, s::String) Base.println(io, s) flush(io) end

Unfortunately this stack overflows for me too - however I did have to add in `import Base.println` before the code otherwise it wouldn’t compile. Is this preventing the shadow?

 ![image](https://global.discourse-cdn.com/julialang/original/3X/3/7/37bc43905c5b307f6fe1335d9bbd09915ae7b633.png)

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [May 1, 2019, 4:17pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/10 "2019-05-01T16:17:38Z")

</div>

Yes, that’s what I was saying with:

> Note that you must define this _before_ referring to `println` as once you’ve called Base’s implementation `println` you’re no longer able to shadow it with your own definition.

You’ll need to restart your Julia session or create a new module to allow the shadowing of `println`.

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [May 1, 2019, 4:40pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/11 "2019-05-01T16:40:40Z")

</div>

Or simply call it something else (e.g. `flushprintln`)

---

<div class="post-metadata">

**Author:** ![jonbarkerlondon](https://avatars.discourse-cdn.com/v4/letter/j/9dc877/32.png) [@jonbarkerlondon](https://discourse.julialang.org/u/jonbarkerlondon)\
**Post date:** [May 1, 2019, 4:41pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/12 "2019-05-01T16:41:24Z")

</div>

That would be nice, unfortunately we have a few thousand lines of julia code that i don’t want to do a find/replace in 🙂

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [May 1, 2019, 7:05pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/13 "2019-05-01T19:05:27Z")

</div>

Unfortunately it is not very well worked out or documented which kind of streams are buffered how, and by whom (buffered on the julia side? in `ios.c`? by libuv?).

A spectacularly ugly workaround is to keep your code and just invoke `$ julia somefile.jl | tee output.txt >/dev/null` instead of `$ julia somefile.jl > output.txt`. This will cause `stdout` to be a `PipeEndpoint`, and `PipeEndpoint` is by default unbuffered. I consider this a bug, but sometimes [xkcd: Workflow](https://xkcd.com/1172/) is the way to go.

PS. You find the definition via `@less println("AA")`. Apart from the `@less` macro, you can use

```julia
julia> methods(println)
# 3 methods for generic function "println":
[1] println(io::IO) in Base at coreio.jl:5
[2] println(io::IO, xs...) in Base at strings/io.jl:69
[3] println(xs...) in Base at coreio.jl:4

```

After reading the two lines in `coreio.jl`, you could come up with

```julia
julia> @eval Base println(xs...) = begin res = println(stdout::IO, xs...); flush(stdout); return res end;
julia> @eval Base println(io::IO) = begin res = print(io, '\n'); flush(io); return res end;

```

You can verify that this works with

```julia
$ strace julia -e 'for i=1:10_000 println("AA") end' 1>output 2>trace
$ wc -l output
10000 output
$ wc -l trace 
884 trace

$ strace julia -e '@eval Base println(xs...) = begin res = println(stdout::IO, xs...); flush(stdout); return res end; @eval Base println(io::IO) = begin res = print(io, '\n'); flush(io); return res end; for i=1:10_000 println("AA") end' 1>output 2>trace
$ wc -l output
10000 output
$ wc -l trace 
10884 trace

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 2, 2019, 5:22am UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/14 "2019-05-02T05:22:27Z")

</div>

> [@jonbarkerlondon](#):
>
> That would be nice, unfortunately we have a few thousand lines of julia code that i don’t want to do a find/replace in 🙂

Using a wrapper type would be a clean solution. Eg (just a starting point, modify/extend as needed):

```julia
struct FlushingIO{T}
    io::T
end

function Base.print(io::FlushingIO, args...)
    print(io.io, args...)
    flush(io.io)
end

out = FlushingIO(stdout)
my_function(out, ...)

```

---

<div class="post-metadata">

**Author:** ![Martin\_Florek](https://avatars.discourse-cdn.com/v4/letter/m/f4b2a3/32.png) [@Martin\_Florek](https://discourse.julialang.org/u/Martin_Florek)\
**Post date:** [July 23, 2019, 12:48pm UTC](https://discourse.julialang.org/t/issues-with-println-buffering-output-when-redirecting-stdout-to-a-txt-file/23738/15 "2019-07-23T12:48:03Z")

</div>

@Tamas_Papp and @all Is this possible with a stderr? How to use it for example to write to a file?