Redirect output and error to a file

fsantos · April 9, 2021, 8:40am

Hi,

[Warning: Julia noob here.]
I would like to know how to redirect any output of a given call to a file. I know that the package Suppressor.jl is an interesting entry point and does almost what I need, but for instance:

using DataFrames
using Suppressor
output = @capture_out begin
dtf = DataFrame(                 
 [8.04   9.14  7.46   6.58;    
  6.95   8.14  6.77   5.76;   
  8.33   9.26  7.81   8.47])
dtf
end

captures nothing (or more precisely, it captures "") in output. To be able to capture the output, we have to write print(dtf) instead of dtf (while the simple dtf would actually produce an output in the console). So it seems that any output cannot be captured by this macro @capture_out.

Actually, when I send a given code block to Julia, I would need to redirect to a given file all that would be displayed in the console: either normal output or error messages.

What would be the best / easiest way to achieve that? And, if possible, with minimal dependencies on packages?

Thanks!

fsantos · April 13, 2021, 8:43am

Just a little bump. Since I had no answer here, I cross-posted to SO, where I had a great answer. However, I still don’t get all the subtleties of Julia I/O…

My real goal is to retrieve external code blocks (without modifying them) and automatically redirect their REPL output to a given file. You’ll understand that I need thus a universal solution, which will work for any type of output (dataframe, dict, tuple, array, etc.). This sounds like a very basic task, but I really can’t find any universal solution in Julia, which sounds just absurd to me.

Let’s say I have the following code block:

x = [1, 2, 3, 4]
for i=1:3
    println(x[i])
end

Everyone can guess waht would be the REPL output. I would just need that this classical REPL output of such a code block could be written as is, into a file.

First try:

open("/home/fsantos/myfile.txt", "w") do io
show(io, "text/plain", begin
        x = [1, 2, 3, 4]
        for i=1:3
            println(x[i])
        end
    end)
end

This creates a file where it’s only written… nothing. The loop result 1 2 3 is still printed in the REPL only, and x is not displayed at all (neither in the file nor in the REPL). What’s the reason for that?

Second try: with the @capture_out macro from Suppressor.jl, with or without wrapping the whole code block within a print instruction. I let you see that none of those two options universally capture the real REPL output of those very elementary code blocks: sometimes you need a print, sometimes you don’t need it, but there seems to be no means of guessing whether it’s necessary or not. And none of those solutions is really satisfying:

using Suppressor

output = @capture_out begin
    x = [1, 2, 3, 4]
end
output # doesnt' work at all

output = @capture_out print(begin
    x = [1, 2, 3, 4]
end)
output # works, but with a different style than in the REPL

output = @capture_out begin
    for i=1:3
        println(i)
    end
end
output # works

output = @capture_out print(begin
    for i=1:3
        println(i)
    end
end)
output # adds a "nothing" row (why?..)

In R, there’s just the very simple function sink() to write very easily any REPL output into a file. It looks like there’s no solution at all in Julia for such a basic task?..

sijo · April 13, 2021, 10:56am

As you know I think there are two things to distinguish:

Output generated by your code
Output generated by the REPL functions

The code

dtf = DataFrame(a=1:5)
dtf

generates no output. But if you type it in the REPL, your are executing another piece of code: a REPL function that evaluates your code and prints the result.

If you want to redirect output from your code, you can use Suppressor.jl, or simply something like this:

julia> open("stdout.txt", "w") do io
           redirect_stdout(io) do
               println("Hello")
           end
       end

But I don’t know if it’s possible to redirect output generated by the REPL. For that I would pipe the script to a new Julia process, so I can redirect the output from the shell.

For example, say I have a file a.jl containing the following:

using DataFrames

dtf = DataFrame([8.04   9.14  7.46   6.58
                 6.95   8.14  6.77   5.76
                 8.33   9.26  7.81   8.47], :auto)

x = 3

If I run julia a.jl in the terminal, I get no output because the code runs without a REPL and generates no output.

However on Linux for example I can run it by calling julia <a.jl >stdout.txt : this feeds the code through standard input. Apparently this is enough to make Julia start a REPL, so it prints results as you want. And the shell redirects the results to stdout.txt:

cat stdout.txt 
3×4 DataFrame
 Row │ x1       x2       x3       x4
     │ Float64  Float64  Float64  Float64
─────┼────────────────────────────────────
   1 │    8.04     9.14     7.46     6.58
   2 │    6.95     8.14     6.77     5.76
   3 │    8.33     9.26     7.81     8.47
3

To redirect both stdout and stderr, you could use julia >output.txt 2>&1, or with a recent Bash version: julia &>output.txt. Example:

$ echo 'x = y' | julia &>output.txt

$ cat output.txt 
ERROR: UndefVarError: y not defined
Stacktrace:
 [1] top-level scope
   @ none:1

fsantos · April 13, 2021, 11:42am

Many thanks, I think that the function redirect_stdout() should work in most of the cases I need. (Your other proposition should do the trick for the remaining cases… I hope!)

This particular point really looks odd to me. I mean, is it something specific to Julia, or is it like that in most interpreted languages? (Unless I’m deeply wrong, that’s something very easy to achieve in R, nope?)
Anyway, the fact that there is no quick-and-obvious equivalent to the R function sink(), for instance, surprises me. I didn’t expect that such a basic task would require so much effort and workarounds in Julia… These “First steps” in Julia are more painful that I thought

Thanks, however!

FPGro · April 13, 2021, 12:07pm

I think it’s just something that nobody missed before It’s actually pretty easy to do, take this proof of concept: (and if somebody did already implement that before me: I’m sorry, my search-foo is pretty weak. It’s also still pretty brittle.)
copytorepl.jl

File content

struct CopyToREPL <: AbstractDisplay
	io::IO
	copyto::AbstractDisplay
	insertnewlines::Bool
end

import Base: flush, close, display, displayable 

flush(d::CopyToREPL) = flush(d.io)
close(d::CopyToREPL) = close(d.io) # not the copyto display, we don't want to tear it down too

display(d::CopyToREPL, @nospecialize x) = display(d, MIME"text/plain"(), x)
display(d::CopyToREPL, M::MIME"text/plain", @nospecialize x) = _display(d, M, x)
function _display(d::CopyToREPL, M, @nospecialize x)
	show(d.io, M, x)
	d.insertnewlines && println(d.io)
	# you may insert a flush(d.io) here if you want to be able to kill the window without loosing content
	show(d.copyto, M, x)
end

displayable(d::CopyToREPL, M::MIME) = istextmime(M)
# it's probably better not to catch every MIME, idk
#function display(d::CopyToREPL, M::MIME, @nospecialize x)
#    displayable(d, M) || throw(MethodError(display, (d, M, x)))
#    _display(d.io, M, x)
#end

function gethighesttextdisplay()
	displays = Base.Multimedia.displays
	message = "Probing displays!" 
	# because we need to find the last display that actually outputs text 
	# or we may end up backing up to a plotting pane or so. There may be a better way to do this.
	for i = length(displays):-1:1
        if Base.Multimedia.xdisplayable(displays[i], message)
            try
                display(displays[i], message)
				return displays[i]
            catch e
                isa(e, MethodError) && (e.f === display || e.f === show) ||
                    rethrow()
            end
        end
    end
    error("No text display found, that's weird.")
end

function pophighestcopyto()
	displays = Base.Multimedia.displays
	for i = length(displays):-1:1
        if displays[i] isa CopyToREPL
            return splice!(displays, i)
        end
    end
    throw(KeyError(CopyToREPL))
end

function startREPLcopy(noIO; append = false)
	io = open(noIO, write = true, append = append)
	startREPLcopy(io)
end

stdoutbackup = []

function startREPLcopy(io::IO; insertnewlines = true)
	# insertnewlines because show(...) usually doesn't insert newlines after display,
	# so the output is kinda mashed together. There may be cleaner way to do this.
	backupdisplay = gethighesttextdisplay()
	cpdisplay = CopyToREPL(io, backupdisplay, insertnewlines)
	@info "Started copying REPL output to the specified location!"
	pushdisplay(cpdisplay)
	# TODO: It would be better to check here if the stdout hadn't been redirected before
	# So technically, this may snatch a stdout and display more than the REPL
	push!(stdoutbackup, Base.stdout)
	Base.redirect_stdout(io) 
	# TODO I think this does only work for IOStreams, so maybe unwrap contexts and such
	# TODO: may or may not be desirable to handle stderr
	nothing
end

function endREPLcopy()
	cpdisplay = try
		pophighestcopyto()
	catch e
		isa(e, KeyError) || rethrow()
		@warn "Couldn't stop copying the REPL output because it wasn't even started."
	end
	flush(cpdisplay)
	oldstdout = pop!(stdoutbackup)
	Base.redirect_stdout(oldstdout)
	close(cpdisplay)
	@info "Copying of REPL output stopped!"
	nothing
end

and a demonstration:

julia> include("copytorepl.jl")
endREPLcopy (generic function with 1 method)

julia> startREPLcopy("testlog.log")
"Probing displays!"
[ Info: Started copying REPL output to the specified location!

julia> test = 2
2

julia> test
2

julia> println("Bla-Keks")

julia> show("αβ↑")

julia> notdefined
ERROR: UndefVarError: notdefined not defined

julia> # note that the error does not print

julia> rand(8)
8-element Vector{Float64}:
 0.48184594197528385
 0.4667812261893638
 0.43096133172880347
 0.791219818003811
 0.12263780916474376
 0.15859996424283396
 0.28204060573062506
 0.3664465404400239

julia> "Bye"
"Bye"

julia> endREPLcopy()
[ Info: Copying of REPL output stopped!

gives:
testlog.log

2
2
Bla-Keks
"αβ↑"8-element Vector{Float64}:
 0.48184594197528385
 0.4667812261893638
 0.43096133172880347
 0.791219818003811
 0.12263780916474376
 0.15859996424283396
 0.28204060573062506
 0.3664465404400239
"Bye"

May need some more formatting and cleanup but it is doable in principle.

fsantos · April 13, 2021, 12:21pm

I totally understand that one. (Actually, I’m trying to implement a lightweight literate-programming interface for Julia in Emacs, so that REPL-capture stuff like this is really crucial for me. But I fully understand that it’s not a common need for Julia users, yep.)

Also, many thanks for your function! I have to study and understand it, but it will definitely be very useful for me!

FPGro · April 13, 2021, 12:53pm

Cool, good luck!

A lot of that is actually just the basic TextDisplay with doubled output:

github.com

JuliaLang/julia/blob/f9720dc2ebd6cd9e3086365f281e62506444ef37/base/multimedia.jl#L239


      
          
          
# simplest display, which only knows how to display text/plain
          
          
"""
              TextDisplay(io::IO)
          
          
Returns a `TextDisplay <: AbstractDisplay`, which displays any object as the text/plain MIME type
          (by default), writing the text representation to the given I/O stream. (This is how
          objects are printed in the Julia REPL.)
          """
          struct TextDisplay <: AbstractDisplay
              io::IO
          end
          display(d::TextDisplay, M::MIME"text/plain", @nospecialize x) = show(d.io, M, x)
          display(d::TextDisplay, @nospecialize x) = display(d, MIME"text/plain"(), x)
          
          
# if you explicitly call display("text/foo", x), it should work on a TextDisplay:
          displayable(d::TextDisplay, M::MIME) = istextmime(M)
          function display(d::TextDisplay, M::MIME, @nospecialize x)
              displayable(d, M) || throw(MethodError(display, (d, M, x)))
              show(d.io, M, x)

but if you need more intricate behaviour, you may have to look at the REPL source. Fortunately, it’s all just Julia code.

Henrique_Becker · April 13, 2021, 12:58pm

No idea. But I think it is hardly common that someone wants that. If you are saving the outputs, and only the outputs, of a REPL then you have a bunch of answers without the questions. I fail to see the usefulness of that.

In Bash you can do:

[henrique AreaDeTrabalho]$ julia | tee repl_log.txt
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.3 (2020-11-09)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> "teste1"
"teste1"

julia> println("teste2")
teste2

julia>

And this will save all the log to the file, somehow even the colors are also saved. I do not know if this helps you, but if I ever had a similar need it would be this that I would do.