I’m running pmap with 16 workers and batch_size=1 to process hundreds of files. Each file takes over an hour to work through and the files are relatively homogeneous but don’t take exactly the same time. For the first 14 hours or so the computer was using 16 cores at 100% usage. However, at some tim…

Pmap using fewer workers than expected after some time

Luapulu September 16, 2020, 3:44pm 4

Ok, simple explanation: there was an error. It waited until each worker was done to stop the program. I guess I was confused because you don’t see the error until each worker is done.

Topic		Replies	Views
Pmap use of processor cores Julia at Scale question , pmap , load-balancing	13	2250	June 12, 2019
Requesting idle workers to speed up unbalanced processes with pmap General Usage pmap	9	1569	March 21, 2018
Weird behavior of pmap General Usage	5	1446	July 2, 2019
Problems using pmap(), and doubt about the number of workers/processes to use General Usage pmap	3	1181	February 7, 2019
Behavior of worker pool in pmap Performance pmap	2	918	November 25, 2018

Pmap using fewer workers than expected after some time

Related topics