Controlling thread affinity for distributed multithreading

Specific Domains Julia at Scale
1

I have a question regarding the setup of hybrid parallelism which combines multithreading and distributed computations.

Let’s say I want to run some computation on a machine with Non-Uniform Memory Access. In this case, it is a good idea to have a julia process per NUMA node, each of the processes employing multiple threads which run on the same NUMA node.

I know that addprocs allows one to specify the cpu affinity of the additional julia processes: these way I can pin the processes to different NUMA nodes. Besides that, one can pass to addprocs special flag to use multithreading for these processes. My question is: how does one specifies the affinity of these threads? (So that they are pinned to the same NUMA node as the process they belong to.)

2 Likes
2

I have been using numactl to set the thread affinity like

numactl --physcpubind=0-$(cpus_per_task - 1) -- $(exe) -O3 $(input) -E 'run(`numactl -s`)'

where exe was the path to my Julia executable and input the script to be executed. Last time I checked on our local cluster, the processor affinity carried over to the threads controlled by different Julia processes, pinning one thread to one core on every compute node. It’s been a while though …

3

See also Thread affinitization: pinning Julia threads to cores

4

Wasn’t aware that JULIA_EXCLUSIVE exists, thanks for linking the other issue.

1 Like