Detecting a closed channel?


#1

i have a very simple producer/consumer situation.

    chan = Channel{Array{Float64,1}}(1)
    f1() = use_data(chan, npoints)
    t = @task f1()
    schedule(t)
    yield()
    
    while true
        data = calc()
        put!(chan, data)
   end

When the task t has processed enough data, i.e. npoints of data, it closes the channel. I was assuming that it would be really easy for me to put something, e.g. isopen(chan), in the while loop to check for the closed channel and break.
That doesn’t seem to work.
What’s the best way to handle this ? Set up a 2nd channel and put! a value when the consumer is finishe ?


#2

I flailed around on a an approach resembling yours in my MWE here and hit the wall where there is no isopen() function nor an open() for RemoteChannel. In that MWE code above, I had to detect that the channel was closed by catching the error and exiting the loop. Crude, and there was no way to restart the thing because there is no open()

Later, I set up code to put zero-flagged jobs onto the job queue, and the workers would exit when they saw a zero job. There was a bit of flushing the pipes but it worked.

Then, I abandoned the whole approach and went to a variant of the scheduler in the example code. This works better in my case, because emit() was causing too much data traffic, and it was better to accumulate map results in the worker before returning a result. I’m happy with this method.

So, the answer is that you have to catch the error, but there may be more elegant ways to do it.


#3

Sorry to revive this old conversation, @pasha, but I am running into the same problems with RemoteChannel that you describe. I’m interested in the solution you ended up with, but the links in your paragraph about your variant of a scheduler no longer work, and I’m afraid I don’t understand the solution you ended up with. If you have time, could you please update the link or explain the scheduler in more detail?


#4

I found the documentation that you linked to in the v0.6 docs. The example there was billed as a ‘simple pmap’, which led me to look through the pmap code and do some simple experiments. I hadn’t understood, until now, that pmap already feeds jobs to workers, as workers complete their jobs. For some reason I thought the jobs were split up evenly amongst workers.