Named pipes

General Usage
,
1

Is there something like “named pipes” in Julia? (I just learned that those things exist).

The goal is to provide a “file” as input to an external program while avoiding explicitly writing the file to disc.

Does that exist?

2

Just use shell mode to create one?

shell> mkfifo mynamedpipe

julia> write("mynamedpipe", "whateverdatayouwant")
... (blocked here)

And then in another terminal, do

cat mynamedpipe
whateverdatayouwant

Back in the REPL, write would have returned

julia> write("mynamedpipe", "whateverdatayouwant")  # Executed before
19

When writing scripts, you can just do run(`mkfifo mynamedpipe`).

1 Like
3

But then mkfifo is something that must be available (I guess probably it isn’t in general - for all platforms). I actually want to do that inside a library.

I think that’s the question: is there a mkfifo in Julia?

4

I think you can use the Sockets standard library.

https://docs.julialang.org/en/v1/stdlib/Sockets/#Sockets.listen-Tuple{AbstractString}

3 Likes
5

In Julia 1.11 there is Base.Libc.mkfifo which wraps the relevant system call.

3 Likes
6

Any idea if this is/will be in Compat or if it is already in some package?

7

Here is a small example about named pipes in the docs:
https://julia-doc.readthedocs.io/en/latest/manual/networking-and-streams/#a-simple-tcp-example

1 Like
8

This functionality is already provided by the Sockets stdlib. There’s no need to manually wrangle Libc.

9

Can someone give me a simple example on how to use Sockets for that? From the documentation I cannot figure it out.

For example, let us suppose I want to use grep to extract some lines from a string (I’m not actually willing to do that, this is just a simple example to illustrate what I want):

julia> str = """
           abc 
           def
       """
"    abc \n    def\n"

julia> open("str.txt", "w") do io
           println(io, str)
       end

julia> gr = read(pipeline(`grep "abc" str.txt`), String)
"    abc \n"

Here, I’m writing to disc an intermediate file str.txt. How can avoid that using Sockets?

10

Here is an stripped down minimal example from the docs. Open up two REPLs, REPL1 is the data receiving part, and REPL2 is the sending part in this example:

REPL1:

using Sockets
server = listen("\\\\.\\pipe\\testsocket")
while true
   sock = accept(server)
   while isopen(sock)
	   println(readline(sock, keep=true))
   end
end

REPL2:

using Sockets
clientside = connect("\\\\.\\pipe\\testsocket")
println(clientside,"Hello World from the Echo Server")
println(clientside,"Hello World from the Echo Server")
println(clientside,"Hello World from the Echo Server")

You will see the “Hello World” text send through the named pipe and printed on REPL1 three times.

1 Like
11

I appreciate that, but I do not understand how that relates to the problem of not wanting to generate an intermediate file in the for an external program.

ps: It may be that this is not possible in general, and I having called this thread “Named pipes” just cause confusion.

12

Oh, sorry, missed that point about intermediate file.

On the other hand this is not a regular file despite it looks like because of the path string "\\\\.\\pipe\\testsocket". You can see that as a special file just like the unix files in /proc/…

Let us exactly know what you need, perhaps named pipes is really not the right solution. Let’s try to avoid the xy-problem :wink:

13

What I need is something like I posted above: Named pipes - #9 by lmiq

Just the program is not grep, but another program which is compiled as a binary in Yggdrasil.
The programs reads an input file and returns a result. I currently create this input file explicitly as a temporary file in the hard disc, but I would like to avoid that.

It seems to me (this is new to me) that the mkfifo tool is the standard solution for what I want.

14

Is this a given premise?
If yes, than you have to create a file.
If no, which means you are able to change this binary, than you are probably free to do whatever you want as interprocess communication.

For me it’s still not clear what problem we try to solve. The grep example above is special as grep accepts a stream from STDIO instead of a existing file. So you have an option without creating a new file. Does your binary accepts STDIO stream as well?

15

No, it does not.

Well, yes and no. There is a library which could be used directly without the intermediate file, but would require the implementation of the interface directly to the library in Julia which is some additional work, particularly because everything is written in C++ and I don’t readily understand what’s going on in that interface. And there is the driver (main program) which is easy to use but requires the file as input.

For now I’m just writing the file to disc. I would be happy if I could avoid that, and it seemed to me that mkfifo was a mechanism to exactly do that, to create “a file” in the memory that can be used “as if it was a regular disc file” is these cases. But I may have misunderstood that.

(my tries with mkfifo in Julia 1.11 failed for now).

16

From https://www.gnu.org/software/coreutils/manual/html_node/mkfifo-invocation.html#mkfifo-invocation

A FIFO is a special file type that permits independent
processes to communicate. One process opens the
FIFO file for writing, and another for reading, after which
data can flow as with the usual anonymous pipe in shells or
elsewhere.

I am afraid, as far as I understand, that this is not what you can use as an shortcut to avoid creating a real file, but I have never tried. As far as I can see you only have the option to give the “io channel” a name, hence “named pipe”. This is not a file name. The OS is doing the real thing under the hood and gives access using this name. There is probably somewhere in the file system a special file, perhaps in /tmp/… on unix, which is the visible named pipe, but still it’s a special file. If you can open this with your binary just pretending its a normal file is questionable but something you can try.

On which OS are you working? I want to do some experiments this evening with your issue (if my evening expectations will come true).

17

I’m on Linux (ideally I would like a cross-platform solution, of course).

Try having kids :rofl:

2 Likes
18

here is that example using mkfifo, in case it’s helpful:

julia> str = """
           abc
           def
       """
"    abc \n    def\n"

julia> run(`mkfifo my_named_pipe`)
Process(`mkfifo my_named_pipe`, ProcessExited(0))

julia> t1 = Threads.@spawn open("my_named_pipe", "w") do io
           println(io, str)
       end
Task (runnable) @0x000000032063aef0

julia> t2 = Threads.@spawn read(`grep "abc" my_named_pipe`, String)
Task (runnable) @0x000000032063b080

julia> wait.([t1, t2])
2-element Vector{Nothing}:
 nothing
 nothing

julia> fetch(t2)
"    abc \n"

(I know the discussion has moved on to sockets though)

2 Likes
19

My experiments told me that mkfifo indeed works as you are expecting it, but only for linux/unix.
mkfifo creates a special file (in linux). This file can just be read just like a regular file (checked with grep and cat). You can write into the named pipe, e.g.:

mkfifo /tmp/myNamedPipe
echo "Hello World" > /tmp/myNamedPipe

Now you can read from this named pipe with any program which expects opening a file with given name:

cat /tmp/myNamedPipe

(you need a second shell for this simple example, as echo waits until all data is read from the pipe with cat).

For windows I don’t see an equivalent similar easy pendant. Cross platform is not easy. At least, together with @ericphanson solution, for linux the way to go should be feasible now.

1 Like