I have a job that needs to run in multiple processors. How should I decide what
n should be in
addprocs(n)? I assume the right number will depend on the tasks and the number of cores and RAM.
Is there a way to automate the decision for common cases?
There is no ideal number of workers in general as it depends on the system that you’re running on as well as the program. However afaik it does not really gain much to create more workers than there are physical processors on your machine.
You can get the total number of cores (including virtual ones) via
If you can make use of all cores still depends on your program though
It would be nice to have a way of limiting parallelism to account for memory pressure.
Hwloc is good for this:
julia> using Hwloc
Although on the Apple M1, I’ve always found 4 (the number of big cores) to be faster than using 8 (the number of physical cores):
julia> using Hwloc
Julia Version 1.8.0-DEV.54
Commit 6d2c0a7766* (2021-06-19 00:28 UTC)
OS: macOS (arm64-apple-darwin20.5.0)
CPU: Apple M1
so while I know the number of big cores is 4, I don’t know of any way to actually query this, unfortunately.
Sorry to revive an old thread, but I have one additional question. Do I need to reserve one physical core for the main process? In other words, if I have
n physical cores, should I create
n-1 workers or
n workers? This is in the context of using
Based on the following quote from the manual, I suspect that the answer is to create
n workers, but I just want to double check.
Starting with julia -p n provides n worker processes on the local machine. Generally it makes sense for n to equal the number of CPU threads (logical cores) on the machine.
That quote appears in this section of the manual.
If your main process is simply going to wait for
pmap to return, then you can go ahead and spawn
n workers. Otherwise, you are just letting a core “go to waste”