# Strange interaction between \`Channel()\` and pipes

**URL:** <https://discourse.julialang.org/t/strange-interaction-between-channel-and-pipes/7623>\
**Category:** General Usage\
**Created:** [December 8, 2017, 1:48pm UTC](https://discourse.julialang.org/t/strange-interaction-between-channel-and-pipes/7623 "2017-12-08T13:48:25Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![davidavdav](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidavdav/32/1065_2.png) [@davidavdav](https://discourse.julialang.org/u/davidavdav)\
**Post date:** [December 8, 2017, 1:48pm UTC](https://discourse.julialang.org/t/strange-interaction-between-channel-and-pipes/7623/1 "2017-12-08T13:48:26Z")

</div>

Hello,

I am trying to make a generator for reading a list of `files’, where a file may be an actual file or a subprocess. For the interface it is nice if the result can be a file descriptor, as the interpretation of the content depends on the caller.

I make the generator with `Channel()`–`push!()` and the subprocess using `open(``command``)` [where ```` is a single backtick but I don’t know how to enter this in here].

The problem is that the subprocess’s file descriptor produces 0 bytes when read from by the caller. The problem is maybe best demonstrated by the following code:

```julia
genpipe(file, pipe=false) = Channel() do c
	for i in 1:2
		if pipe
			fd, process = open(`cat $file`, "r")
		else
			fd = open(file, "r")
		end
		push!(c, fd)
		close(fd)
	end
end

open("out.txt", "w") do fd
	println(fd, "Hello, pipe")
end

for pipe in [false, true]
	for fd in genpipe("out.txt", pipe)
		println("pipe ", pipe, " len ", length(read(fd)))
	end
end

```

Result (julia-0.6):

```julia
pipe false len 12
pipe false len 12
pipe true len 0
pipe true len 0

```

The last two lines are unexpected.

I’ve also tried the `open() do fd` construct within the generator, but this gives the same result.

Cheers,

—david

---

<div class="post-metadata">

**Author:** ![jameson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jameson/32/23_2.png) [@jameson](https://discourse.julialang.org/u/jameson)\
**Post date:** [December 10, 2017, 10:57pm UTC](https://discourse.julialang.org/t/strange-interaction-between-channel-and-pipes/7623/2 "2017-12-10T22:57:44Z")

</div>

Since you’re calling `close(fd)` after a very short pause, it seems unsurprising that `cat` didn’t have enough time to copy the results out, but loading the file directly did.

---

<div class="post-metadata">

**Author:** ![davidavdav](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidavdav/32/1065_2.png) [@davidavdav](https://discourse.julialang.org/u/davidavdav)\
**Post date:** [December 12, 2017, 3:55pm UTC](https://discourse.julialang.org/t/strange-interaction-between-channel-and-pipes/7623/3 "2017-12-12T15:55:27Z")

</div>

Thanks,

Well, I don’t really understand the intricacies of `Channel()`–`push!()`, bu so far I’ve been using it as in a generator structure similar to python generators.

But to my understanding and experience, the `close(fd)` (either explicit, as here, or implicit if coded in a `do` construct) does not get run until the next iteration from the caller.

For now, I can solve this by pushing `IOBuffer(read(fd))` instead of `fd`, but this reads the entire file / process output into memory which might defeat the purpose.

—david

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [December 12, 2017, 7:34pm UTC](https://discourse.julialang.org/t/strange-interaction-between-channel-and-pipes/7623/4 "2017-12-12T19:34:11Z")

</div>

You are expecting the `push!` to the channel to block?  
The default `Channel` size is 32 and a `push!` will only block once the channel is full.  
`Channel() do ... end` will actually create an unbuffered `Channel` (`csize=0`), so `push!` will never block.

So since the `push!` doesn’t block, `close` will run immediately.

---

<div class="post-metadata">

**Author:** ![davidavdav](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidavdav/32/1065_2.png) [@davidavdav](https://discourse.julialang.org/u/davidavdav)\
**Post date:** [December 13, 2017, 8:59am UTC](https://discourse.julialang.org/t/strange-interaction-between-channel-and-pipes/7623/5 "2017-12-13T08:59:27Z")

</div>

OK, then, what is the ultimate Julia equivalent of a python generator?

—david

---

<div class="post-metadata">

**Author:** ![davidavdav](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidavdav/32/1065_2.png) [@davidavdav](https://discourse.julialang.org/u/davidavdav)\
**Post date:** [December 13, 2017, 9:32am UTC](https://discourse.julialang.org/t/strange-interaction-between-channel-and-pipes/7623/6 "2017-12-13T09:32:20Z")

</div>

> You are expecting the `push!` to the channel to block? Channel() do … end will actually create an unbuffered `Channel (csize=0)`, so push! will never block.

The online help of `Channel()` says

> Channel(0) constructs an unbuffered channel. `put!` blocks until a matching `take!` is called. And vice-versa.

which is what I actually want, but is the opposite of what you stated.

—david

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [December 13, 2017, 5:25pm UTC](https://discourse.julialang.org/t/strange-interaction-between-channel-and-pipes/7623/7 "2017-12-13T17:25:22Z")

</div>

You are right, I misread the documentation. The issue is slightly more complicated…

If you take a look at [https://github.com/JuliaLang/julia/blob/217e059808a4e04900a48f9e2f03069a7038af32/base/channels.jl#L289](https://github.com/JuliaLang/julia/blob/217e059808a4e04900a48f9e2f03069a7038af32/base/channels.jl#L289)

There is the potential for a race here

```julia
Task1 Task2
   | |
  push! |
   | |   
   | take!
   | <--- yieldto!
 yield(v) ---> |
   | |
 close read

```

There is **no guaranteed order** in which `close` and `read` are going to run in your code.  
It _tries_ to hand over execution to `Task2` preferentially, but if there is a `yield` point in `Task2`  
`Task1` will continue to execute.

```julia
for pipe in [false, true]
    for fd in genpipe("out.txt", pipe)
        yield()          
        println("pipe ", pipe, " len ", length(read(fd)))
    end
end

```

---

<div class="post-metadata">

**Author:** ![davidavdav](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidavdav/32/1065_2.png) [@davidavdav](https://discourse.julialang.org/u/davidavdav)\
**Post date:** [December 14, 2017, 9:33am UTC](https://discourse.julialang.org/t/strange-interaction-between-channel-and-pipes/7623/8 "2017-12-14T09:33:47Z")

</div>

Thanks, I see the problem now. It is, in a way, strange that the construct works so well for non-piped iobuffers.

I started writing the generator with a `do` block construct, hoping that the implicit `close()` would be triggered by destruction of the `IO` object, which would then happen when the caller iterates to the next item. However, that didn’t work either, so I guess my assumption about how the `open() do` construct works is false.