Redirect stderr/out to IOBuffer()

General Usage
1

I have a Julia script that runs on a remote machine. The main julia process running this script launches local julia worker processes and waits for AMQP messages that contain input data for a computation. The results of the computation are then packed into an AMQP message and returned. We’re trying to set things up so if one of the julia worker processes on the remote machine errors, we can exit gracefully and send the error and a backtrace back via AMQP. In some cases we can just wrap everything in a try/catch block, intercept exceptions that way and then send the error message to a logger that writes to an IOBuffer that we can put in an AMQP message.

However, this doesn’t always work. In particular, if one of the julia worker processes ccalls an external library and that crashes, then an error message seems to get sent to stderr but we don’t seem to be able to capture that as a log event. I’m wondering if instead, we can just redirect stderr to an IOBuffer or some julia object such that we can grab everything being sent to stderr while still inside julia and then use Julia to pack it into an AMQP message to be sent off somewhere.

I’m aware of the redirect_stderr function but I’m having trouble using it. I don’t seem to be able to redirect to an IOBuffer(). The docs say that I can only redirect to a TTY, a pipe or a socket. I haven’t been able to find enough documentation on pipes and sockets to figure out how I can capture this output inside julia. Any suggestions on a solution or pointers on where to look for documentation would be much appreciated. I’m sorry I haven’t been able to distill this down to some sort of MWE.

Thanks!

1 Like
2

There may be a better way to pipe things together, but this appears to work:

julia> b = IOBuffer();

julia> @async while true
           write(b, redirect_stderr()[1])
       end
Task (runnable) @0x00007fdd1a3c2230

julia> write(stderr, "test 1\n")
7

julia> write(stderr, "test 2\n")
7

julia> print(String(take!(b)))
test 1
test 2
1 Like
3

@capture_err from https://github.com/JuliaIO/Suppressor.jl?

4

That will create a new pipe and redirect stderr to it for each write call, which is probably not a good idea. redirect_stderr() can be moved outside the loop.

5

Thanks very much for the responses all. They’ve helped me get a much better handle on dealing with stderr and stdout and I think I’m getting more comfortable with pipes now. A couple of additional notes:

  1. The write method (currently defined at line 576 of base/io.jl) is undocumented. Is it not intended for general use?

  2. A note to people who are mixed up like I have been: It turns out that I was confused not just because I didn’t understand pipes but also because I didn’t realize that unhandled exceptions don’t get immediately written to stderr. This is a separate issue from how to redirect stderr but I was conflating the two issues, leading to my long question.

6

Type ?write and you will find that it is documented.

Basically, you use write for writing raw binary data and print for writing textual representations. write(io, "foo") and print(io, "foo") do the same thing for most io objects, because "foo" is already text, but potentially the io object might change the text encoding if you are using print. See also this thread.

Edit: Oh, I see that you are referring specifically to the write(to::IO, from::IO) method. Yes, that is undocumented right now, but maybe we should document it? A PR would be a welcome place to discuss it.

1 Like
7

Thanks for the reply. Yes, I just meant the write(to::IO, from::IO) method. Sorry, I should have been a bit clearer. I will put in a PR.