Dear Julia users, I am currently coding an algorithm that performs the same task on several pieces of a dataset in parallel using pmap, and then gathers the results in a SharedArray. Due to the exploratory nature of the algorithm, the execution time of the task can highly vary depending on the part…

You are looking for redefining pmap since built in works on a chunks. Fortunately in documentation that is already done https://docs.julialang.org/en/stable/manual/parallel-computing :slight_smile:

Thanks for your answer! :slightly_smiling_face: Are you talking about this piece of code? function pmap(f, lst) np = nprocs() # determine the number of processes available n = length(lst) results = Vector{Any}(n) i = 1 # function to produce the next work item from the queue. …

It’s OK to pmap a function that inside executes an other pmap of a subfunction on more granular computing tasks. Under the hood, the top level pmap divides the array into small chunks. And this chunks get divided once more into the more granular computing tasks and those are qeued. Then, all worker…

Yes, that is the code. There would be no idle workers if one keeps feeding them one by one when they are finished with previous task.

That alternative pmap function code, which I’ve used and modified myself, is good at scheduling new jobs to idle workers. But it can’t take the work from an already-busy worker and somehow give some of that work to an idle work. It wouldn’t know how to divide up your task or how to move data effic…

The pmap does nothing magical - magics happens with @async. Here you give the control to the task scheduler. And with nested pmap you do it once more. And even the nested pmap sees the whole worker pool…

Thank you all for your answers! Indeed, I’ll look to my code to see if I can divide up the tasks more efficiently. However, about the fact that the nested pmap sees the whole worker pool, I tried the following code to test it: addprocs(6) pmap((i)->(sleep(i);println(default_worker_pool().workers)…

I wonder what situation do you have in which you would like to submit a task (some function to calculate) from a worker process?

Also see this thread. [image] Nested pmap() functions don not work as I expect Performance Nested pmap seems to work fine. See my notebook for an example.

Requesting idle workers to speed up unbalanced processes with pmap

General Usage

tk3369 March 21, 2018, 8:12pm 10

Also see this thread.

Topic		Replies	Views
Issue with nested pmap Julia at Scale bug , pmap	2	754	June 11, 2019
Pmap using fewer workers than expected after some time Julia at Scale pmap	5	651	February 10, 2021
Dynamically schedule work onto pmap worker pool? General Usage	6	955	January 22, 2021
Nested pmap() functions don not work as I expect Performance pmap	3	1512	March 19, 2018
Behavior of worker pool in pmap Performance pmap	2	946	November 25, 2018

Requesting idle workers to speed up unbalanced processes with pmap

Related topics