Empty remotechannel and resizing slurm jobs

[vancleve]

I am using RemoteChannels and Distributed for a multicore simulation with the template provided here:
https://docs.julialang.org/en/v1/manual/parallel-computing/index.html#Channels-and-RemoteChannels-1

Each process can take a few hours so I’d like to resize my Slurm job after the RemoteChannel with the data/parameters to use becomes empty and some of the workers become idle.

Is there any way to do this within julia (i.e., determining when a worker is idle)? I found this:

How to check a RemoteChannel is open?

I wasn’t able to find anything about this in the documentation either. However, I ended up with the following pattern. addprocs(3) println(nprocs()) arr = [i for i=1:4] function producer(c::Channel) for item in arr put!(c, item) end end jobs = RemoteChannel(()->Channel(producer)); results = RemoteChannel(()->Channel(32)); @everywhere function do_work(jobs, results) while true x = try take!(jobs) end if x == nothing break end …

Is this the best way to get the worker to complete the do_work function after the RemoteChannel with the data/parameters becomes empty? I don’t know exactly how RemoteChannels work, so I was wondering if there is some chance that do_work here would complete with items still in the RemoteChannel.

If this does work, then I suppose I could have another channel that contains the IDs of the worker and each worker puts its ID there before do_work finishes. Then, the main process could use Slurm to resize the job…

Thoughts?