I’ve been using the amazing
IOCapture quite extensively, and I was considering abusing for a situation where I want to siphon off (
tee, in unix jargon) the
stdout data while also allowing it to be printed in the terminal or a notebook as normal.
Are there some IO experts with tips for how to go about this? I haven’t started playing around with this, but I’m guessing something in line 108:
buffer_redirect_task = @async write(output, pipe)
I don’t think
buffer_redirect_task = @async (write(output, pipe) && write(default_stdout, pipe)
will work, because
pipe can probably only be consumed once, right?
Are there other low-level considerations I should be aware of while trying to implement this?
A requirement would be that the normal
stdout should be as undisturbed as possible. For example, running in a Jupyter notebook, it should be printed continuously as a cell runs.
What I ultimately want to do is to have some long-running optimization (on the order of days) that produces significant convergence information output (possibly a couple of hundred MB worth of text).
This calculation is wrapped here:
I’d like to wrap the highlighted line in something that captures the first and the last N bytes of whatever the
optimize function prints to
stderr. That data would then be stored in the dumped
.jld file alongside the
"result". When I run the
optimize_or_load function again (where the
.jld file with the result already exists), it could not just return the cached result, but also print an abbreviated
<beginning of stdout>
<end of stdout>
to have some record of what convergence data was printed during the optimzation. The beginning and end of the output are the most interesting, I don’t want to capture the hundred MB of full output in the
It seemed like wrapping the highlighted line in
IOCapture would work great for this, except that I’d actually like the normal
stdout to be undisturbed, hence this question.
The other part is that I only want to capture the beginning and end of the data, but that seems more straightforward: I’d probably just have to replace the
output = IOBuffer() in line 107 with a custom
IOBuffer-like object that discards the data I don’t want to keep.
Are there any other solutions or packages that might be of use? The basic functionality I would need is to split (
tee) the standard
stdout/stderr into a custom
IOBuffer-like object while leaving it unaffected otherwise.
Oh, and I have seen Write to file and stdout, which might also be useful. It just seemed like
IOCapture is a lot more polished already and might easier to adapt.