Hello, I am struggling to understand why pmap is slow. I use the following fast function rand(). using Distributed addprocs(4) @everywhere using Random @everywhere g(x) = rand() @time map(g,1:100_000_000); 0.697132 seconds (10 allocations: 762.940 MiB, 3.11% gc time) @time pmap(g,1:100_000_000…

Can you communicate with those distributed processes? Are they running on your local machine? I guess so… Is it possible that you have used a lot of memory and have started swapping on your local machine?

[image] johnh: Can you communicate with those distributed processes? Are they running on your local machine? I guess so You are right. Single machine multiprocessors.

It may have to do with the overhead of pmap. I’ve heard in general that pmap should be used when the function g does a large amount of work to offset the cost of overhead. In your case, apparently, the function g is much faster than the time it takes to send out the work to different processors. If …

What do these retuen after you addprocs ? nprocs() workers()

@affans makes an excellent point. If you are going to parallelize by distrributing functions you need to give each worker a ‘decent’ amount of work to do. May apology to non-English native speakers. As @affans says the ration of the time taken to do the task should be greater than the time to commi…

Using your suggestions, I got it to work better (not for the MWE posted here) by making g computing more. Something along those lines: @everywhere g(x) = rand(1000_000) @time pmap(g,1:100);

I still think that pmap is not the way to go here (in the context of your function). See this excerpt from the documentation: Julia’s pmap is designed for the case where each function call does a large amount of work. In contrast, @distributed for can handle situations where each iteration is ti…

This thread may help too, it echos what was said above: [image] Weird behavior of pmap General Usage The following suboptimal behavior of the parallel map has been observed: 53.927 s (82132566 allocations: 2.81 GiB) 2.011 s (8000041 allocations: 167.85 MiB) 18.922 …

Struggling with pmap

New to Julia

raminammour September 5, 2019, 4:45pm 9

This thread may help too, it echos what was said above:

Topic		Replies	Views
Why is the parallel map so slow? General Usage parallel , optimization , pmap	2	3310	May 10, 2020
Weird behavior of pmap General Usage	5	1456	July 2, 2019
Pmap usage Performance question , parallel	1	385	December 13, 2020
Lack of improvement from distributed pmap, understanding a simple example New to Julia distributed , pmap	6	213	October 29, 2024
Poor speed gain using `pmap` Performance parallel , pmap	17	1307	August 6, 2021

Struggling with pmap

Related topics