# Choosing the number of worker processes

**URL:** <https://discourse.julialang.org/t/choosing-the-number-of-worker-processes/63389>\
**Category:** General Usage\
**Tags:** distributed\
**Created:** [June 22, 2021, 5:36pm UTC](https://discourse.julialang.org/t/choosing-the-number-of-worker-processes/63389 "2021-06-22T17:36:29Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [June 22, 2021, 5:36pm UTC](https://discourse.julialang.org/t/choosing-the-number-of-worker-processes/63389/1 "2021-06-22T17:36:29Z")

</div>

I have a job that needs to run in multiple processors. How should I decide what `n` should be in `addprocs(n)`? I assume the right number will depend on the tasks and the number of cores and RAM.

Is there a way to automate the decision for common cases?

---

<div class="post-metadata">

**Author:** ![Salmon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/salmon/32/22968_2.png) [@Salmon](https://discourse.julialang.org/u/Salmon)\
**Post date:** [June 22, 2021, 7:00pm UTC](https://discourse.julialang.org/t/choosing-the-number-of-worker-processes/63389/2 "2021-06-22T19:00:33Z")

</div>

There is no ideal number of workers in general as it depends on the system that you’re running on as well as the program. However afaik it does not really gain much to create more workers than there are **physical** processors on your machine.  
You can get the total number of cores (including virtual ones) via

```julia
Sys.CPU_THREADS

```

If you can make use of all cores still depends on your program though

---

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [June 22, 2021, 7:03pm UTC](https://discourse.julialang.org/t/choosing-the-number-of-worker-processes/63389/3 "2021-06-22T19:03:54Z")

</div>

It would be nice to have a way of limiting parallelism to account for memory pressure.

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [June 22, 2021, 7:15pm UTC](https://discourse.julialang.org/t/choosing-the-number-of-worker-processes/63389/4 "2021-06-22T19:15:52Z")

</div>

> [@Salmon](#):
>
> **physical** processors on your machine.  
> You can get the total number of cores (including virtual ones) via

Hwloc is good for this:

```julia
julia> using Hwloc

julia> Sys.CPU_THREADS
20

julia> num_virtual_cores()
20

julia> num_physical_cores()
10

```

Although on the Apple M1, I’ve always found 4 (the number of big cores) to be faster than using 8 (the number of physical cores):

```julia
julia> using Hwloc

julia> Sys.CPU_THREADS
8

julia> num_virtual_cores()
8

julia> num_physical_cores()
8

julia> versioninfo()
Julia Version 1.8.0-DEV.54
Commit 6d2c0a7766* (2021-06-19 00:28 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin20.5.0)
  CPU: Apple M1

```

so while I know the number of big cores is 4, I don’t know of any way to actually query this, unfortunately.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [September 27, 2021, 9:40pm UTC](https://discourse.julialang.org/t/choosing-the-number-of-worker-processes/63389/5 "2021-09-27T21:40:28Z")

</div>

Sorry to revive an old thread, but I have one additional question. Do I need to reserve one physical core for the main process? In other words, if I have `n` physical cores, should I create `n-1` workers or `n` workers? This is in the context of using `@distributed` or `pmap`.

Based on the following quote from the manual, I suspect that the answer is to create `n` workers, but I just want to double check.

> Starting with julia -p n provides n worker processes on the local machine. Generally it makes sense for n to equal the number of CPU threads (logical cores) on the machine.

That quote appears in [this section](https://docs.julialang.org/en/v1/manual/distributed-computing/#Multi-processing-and-Distributed-Computing) of the manual.

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [September 27, 2021, 10:50pm UTC](https://discourse.julialang.org/t/choosing-the-number-of-worker-processes/63389/6 "2021-09-27T22:50:44Z")

</div>

If your main process is simply going to wait for `pmap` to return, then you can go ahead and spawn `n` workers. Otherwise, you are just letting a core “go to waste”

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [September 27, 2021, 11:08pm UTC](https://discourse.julialang.org/t/choosing-the-number-of-worker-processes/63389/7 "2021-09-27T23:08:49Z")

</div>

Thanks!