Is there a way to set up a pipe between two Julia process that are already running, assuming that the two already have another method of communication to execute code (example: worker process on the same machine started via Distributed.jl). Specifically, I want to redirect the “worker” process’ stdout and stderr to the parent process’ stdout and stderr (again, noting that both are already running).
1 Like
I don’t know if it’s possible; std{in,out,err} are usually connected when one process starts another. However, you may use a FIFO for this purpose (at least on Linux, maybe on Mac as well); see man fifo.
Right, I guess I could do something platform-specific, like
julia> using Distributed; addprocs(1);
julia> @fetchfrom 2 (println("xyz"); flush(stdout))
From worker 2: xyz
julia> @fetchfrom 2 (redirect_stdio(stdout=devnull, stderr=devnull); println("xyz"); flush(stdout))
julia> pid = getpid();
julia> @fetchfrom 2 begin
parent_stdout = open("/proc/$pid/fd/1", "w")
parent_stderr = open("/proc/$pid/fd/2", "w")
redirect_stdio(stdout=parent_stdout, stderr=parent_stderr)
println("xyz")
flush(stdout)
end
xyz
But I was hoping that there was some interface within Julia for this?
Why not use a socket?
It seems wasteful/fragile to use networking IO when all I want is a file descriptor/pipe between two local processes… But I guess it’s good to keep in mind as a last resort.
I can’t find it in the manual right now, but I know for a fact that there is a platform agnostic Pipe object fulfilling exactly that purpose. The various redirect_std* functions should also be able to take them, according to their docs. That’s also how the Distributed stdlib does it, IIRC.
Other than that, the named pipe/UNIX socket from the Sockets stdlib linked above should also work. It’s not really related to networking, the name “Socket” is not exclusive to that.
For some reason, it seems that the Pipe docstring is missing from the HTML documentation. I’m aware of Pipe, but it doesn’t seem like there’s a (documented) way to set one up after a process has already been started? The full Pipe docstring is
help?> Pipe Construct an uninitialized Pipe object. The appropriate end of the pipe will be automatically initialized if the object is used in process spawning. This can be useful to easily obtain references in process pipelines, e.g.: julia> err = Pipe() # After this `err` will be initialized and you may read `foo`'s # stderr from the `err` pipe. julia> run(pipeline(pipeline(`foo`, stderr=err), `cat`), wait=false)
I don’t think Distributed uses Pipes. It simply calls open(cmd, "r+"):
It “redirects” output by continuously reading from the Process and printing the result to a different stream in an @async task:
Of course, this can only be done when the Process can be accessed directly and is already readable.
Oh, sorry, I must have missed the two entries in that list referring to these. That still leaves me having to do the redirection “manually” (like the Distributed method above), but doesn’t require access to the Process object. Another drawback is that it requires picking a path name. But I guess it does what I asked for (a pipe between the processes)!