# Pmap using fewer workers than expected after some time

**URL:** <https://discourse.julialang.org/t/pmap-using-fewer-workers-than-expected-after-some-time/46715>\
**Category:** Julia at Scale\
**Tags:** pmap\
**Created:** [September 16, 2020, 3:27pm UTC](https://discourse.julialang.org/t/pmap-using-fewer-workers-than-expected-after-some-time/46715 "2020-09-16T15:27:12Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Luapulu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/luapulu/32/17669_2.png) [@Luapulu](https://discourse.julialang.org/u/Luapulu)\
**Post date:** [September 16, 2020, 3:27pm UTC](https://discourse.julialang.org/t/pmap-using-fewer-workers-than-expected-after-some-time/46715/1 "2020-09-16T15:27:12Z")

</div>

I’m running pmap with 16 workers and `batch_size=1` to process hundreds of files. Each file takes over an hour to work through and the files are relatively homogeneous but don’t take exactly the same time. For the first 14 hours or so the computer was using 16 cores at 100% usage. However, at some time in the last few hours the number of fully used cpus dropped to 8. The other 8 processes are still alive, they’re just shown as sleeping.

When I check the logs I find that the other (not-working) workers finished processing their last assigned file without error. However, they didn’t start on the next file.

Any ideas why this would happen? If no error occured, why wouldn’t a worker load the next task?

Actually, now they’re down to 6 workers.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [September 16, 2020, 3:33pm UTC](https://discourse.julialang.org/t/pmap-using-fewer-workers-than-expected-after-some-time/46715/2 "2020-09-16T15:33:43Z")

</div>

I’m not 100% sure, but I think pmap does prebatching, i.e. the work gets split up into 16 chunks (if you have 16 workers) and those workers then work on their batch of the whole work. If, by chance, some tasks that take longer are aggregated in a few of the 16 chunks, those workers will have to run for longer than others.

---

<div class="post-metadata">

**Author:** ![Luapulu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/luapulu/32/17669_2.png) [@Luapulu](https://discourse.julialang.org/u/Luapulu)\
**Post date:** [September 16, 2020, 3:36pm UTC](https://discourse.julialang.org/t/pmap-using-fewer-workers-than-expected-after-some-time/46715/3 "2020-09-16T15:36:01Z")

</div>

Oooh, so if it’s trying to combine the results from each worker every so often, then it will wait until the batch it’s expecting is completely finished, meaning it has to wait for the last worker to finish the last piece of the batch?

This would explain a few things actually. I also just realised that I did a stupid thing. My pmap isn’t really a map, it’s more like a “for each”. I don’t want to return anything, I just want to save the results from each file to a different file. However, what I think I did was return the entire result of each file, meaning it has to combine all the results rather than returning an array of `nothing`.

---

<div class="post-metadata">

**Author:** ![Luapulu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/luapulu/32/17669_2.png) [@Luapulu](https://discourse.julialang.org/u/Luapulu)\
**Post date:** [September 16, 2020, 3:44pm UTC](https://discourse.julialang.org/t/pmap-using-fewer-workers-than-expected-after-some-time/46715/4 "2020-09-16T15:44:23Z")

</div>

Ok, simple explanation: there was an error. It waited until each worker was done to stop the program. I guess I was confused because you don’t see the error until each worker is done.

---

<div class="post-metadata">

**Author:** ![nvenkov1](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nvenkov1/32/43679_2.png) [@nvenkov1](https://discourse.julialang.org/u/nvenkov1)\
**Post date:** [February 10, 2021, 11:35am UTC](https://discourse.julialang.org/t/pmap-using-fewer-workers-than-expected-after-some-time/46715/5 "2021-02-10T11:35:08Z")

</div>

My understanding is different. I think @distributed does static scheduling as you describe, but pmap is dynamic, although not necessarily fault tolerant.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [February 10, 2021, 12:29pm UTC](https://discourse.julialang.org/t/pmap-using-fewer-workers-than-expected-after-some-time/46715/6 "2021-02-10T12:29:31Z")

</div>

It depends. If there are no remote workers, it’s not batched, but if there are and `batch_size` is set to something other than `1`, it is. See [the source](https://github.com/JuliaLang/julia/blob/ce5bd1c3dadcc7c733508a19a4b692b744d59a3d/stdlib/Distributed/src/pmap.jl#L99) for more info - it’s written in julia, so should be quite readable! 🙂