Hi! I have a pretty simple problem: I have about 50 tasks I would like to do simultaneously, and I would like to do exploit shared memory parallelism on each task. I have control of about 50 nodes with about 40 CPUs each, so I would like to assign 40 CPUs (all of the CPUs on a particular node) to e…

Thank you! I have a follow-up question: since pmap only assigns to tasks to workers, wouldn’t I essentially be wasting an entire node with 40 CPUs when running a pmap in the outer loop? If so, is there any way to avoid this?

Well, you can start the workers wherever you want. The master process doesn’t have to be alone on that first node :slight_smile:

Hmm I see, I’ll try to get that to work! I have encountered one tricky complication when trying to mix parallelism with distributed. I am running many iterations of a complicated function, and inside the function there are two steps: A distributed for loop over around 10000 array values A paralle…

If your problem can be expressed in a functional manner (no mutation of values shared across tasks), then I’d recommend trying Dagger.jl instead. Dagger is designed to support multithreading and multiprocessing simultaneously, and instead of having to manually express your problem in terms of threa…

Shared memory computation inside of each task of parallel map

General Usage Performance

carstenbauer November 4, 2021, 8:04am 2

Take a look at Multi-threaded worker processes - #6 by Pbellive and the other posts in this thread.

Topic		Replies	Views
Is Pmap _both_ distributed and threaded? Performance multithreading , distributed , pmap	4	495	December 28, 2021
Attaching workers to cores General Usage distributed	10	565	July 22, 2020
Pmap but with control over which workers do the tasks? Julia at Scale question	0	431	January 12, 2020
Lack of improvement from distributed pmap, understanding a simple example New to Julia distributed , pmap	6	214	October 29, 2024
Running a function over multiple nodes on a julia cluster, from within a julia script Julia at Scale hpc , cluster , distributed , slurm	5	297	August 14, 2024

Shared memory computation inside of each task of parallel map

Related topics