Dear Julia users,
I am currently coding an algorithm that performs the same task on several pieces of a dataset in parallel using pmap
, and then gathers the results in a SharedArray. Due to the exploratory nature of the algorithm, the execution time of the task can highly vary depending on the part of the dataset on which it executes, and I have no way to predict it.
Hence when running my code on a cluster, at some point, it turns out that a few workers are still busy, while all the others are idle.
My question is: is it possible for busy workers to somehow “request the help” of idle workers, since other parts of the task to be performed can be parallelized?
In other words: can a worker process call in turn pmap
? How to define the pool of idle workers inside a pmap
call?
I am kind of a newbie with Julia and parallel computing, so I hope my question is clear enough.
Thank you in advance for your help!