Choosing the number of worker processes

General Usage
1

I have a job that needs to run in multiple processors. How should I decide what n should be in addprocs(n)? I assume the right number will depend on the tasks and the number of cores and RAM.

Is there a way to automate the decision for common cases?

1 Like
2

There is no ideal number of workers in general as it depends on the system that you’re running on as well as the program. However afaik it does not really gain much to create more workers than there are physical processors on your machine.
You can get the total number of cores (including virtual ones) via

Sys.CPU_THREADS

If you can make use of all cores still depends on your program though

1 Like
3

It would be nice to have a way of limiting parallelism to account for memory pressure.

4

Hwloc is good for this:

julia> using Hwloc

julia> Sys.CPU_THREADS
20

julia> num_virtual_cores()
20

julia> num_physical_cores()
10

Although on the Apple M1, I’ve always found 4 (the number of big cores) to be faster than using 8 (the number of physical cores):

julia> using Hwloc

julia> Sys.CPU_THREADS
8

julia> num_virtual_cores()
8

julia> num_physical_cores()
8

julia> versioninfo()
Julia Version 1.8.0-DEV.54
Commit 6d2c0a7766* (2021-06-19 00:28 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin20.5.0)
  CPU: Apple M1

so while I know the number of big cores is 4, I don’t know of any way to actually query this, unfortunately.

4 Likes
5

Sorry to revive an old thread, but I have one additional question. Do I need to reserve one physical core for the main process? In other words, if I have n physical cores, should I create n-1 workers or n workers? This is in the context of using @distributed or pmap.

Based on the following quote from the manual, I suspect that the answer is to create n workers, but I just want to double check.

Starting with julia -p n provides n worker processes on the local machine. Generally it makes sense for n to equal the number of CPU threads (logical cores) on the machine.

That quote appears in this section of the manual.

6

If your main process is simply going to wait for pmap to return, then you can go ahead and spawn n workers. Otherwise, you are just letting a core “go to waste”

1 Like
7

Thanks!