# Poor scaling results with \`ImplicitGlobalGrid.jl\`

**URL:** <https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170>\
**Category:** Julia at Scale\
**Created:** [July 23, 2021, 12:49pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170 "2021-07-23T12:49:20Z")\
**Posts on this page:** 9\
**Page:** 2

<div class="post-metadata">

**Author:** ![Eurel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eurel/32/35982_2.png) [@Eurel](https://discourse.julialang.org/u/Eurel)\
**Post date:** [July 28, 2022, 12:34pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/21 "2022-07-28T12:34:19Z")

</div>

Didn’t wait for the scaling experiment on Rome to finish.  
This is very odd, the scaling is as bad on Rome  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/d/d/dde879842faff57dcb57d184ea7909cafd76ff1c.png)  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/f/ffb74df09a7423aeb4d273371ec35189a95dea1a.png)

Milan delivers a better scaling but it still doesn’t make sense anyway in this situation

---

<div class="post-metadata">

**Author:** ![samo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samo/32/35398_2.png) [@samo](https://discourse.julialang.org/u/samo)\
**Post date:** [July 28, 2022, 3:52pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/22 "2022-07-28T15:52:24Z")

</div>

This shows that something fundamentally wrong with how you are launching the jobs. I suppose you are not launching the amount of threads that you intend to launch and/or the threads are not each pinned 1-to-1 to a core.

---

<div class="post-metadata">

**Author:** ![samo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samo/32/35398_2.png) [@samo](https://discourse.julialang.org/u/samo)\
**Post date:** [July 28, 2022, 3:54pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/23 "2022-07-28T15:54:14Z")

</div>

If I remember right, @carstenbauer has created some tools around thread affinity. @carstenbauer would you mind to chime in?

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [July 28, 2022, 7:49pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/24 "2022-07-28T19:49:13Z")

</div>

You can check on which cores your Julia threads (of each MPI rank) are running with [ThreadPinning.jl](https://github.com/carstenbauer/ThreadPinning.jl)’s `getcpuids()`. (And pin them with `pinthreads` if necessary.)

However, I suspect that this is a (arguably subtle) NUMA / [first-touch policy](https://queue.acm.org/detail.cfm?id=2513149#:~:text=The%20first%2Dtouch%20policy%20refers,policy%20associated%20with%20a%20task.) issue. [AFAICS](https://github.com/omlins/ParallelStencil.jl/blob/36a3d57d300d66724fdc85d4fa9763145a858cb5/src/ParallelKernel/allocators.jl#L97), for the CPU case, `@zeros(nx, ny, nz)` just evaluates to a regular `zeros(nx, ny, nz)`. That’s likely an issue because the first thread alone will initialise the array(s) with zeros and hence “first touch” all the elements. Consequently, the memory pages will get established in one NUMA domain - the one that the core running this first Julia thread belongs to - and accessing (parts of) the array(s) from other threads later in the parallel stencil computation will be slow because they’ll need to grab from this single (potentially far away) NUMA domain. The memory bandwidth of the single NUMA domain will become the bottleneck (the other NUMA domains are note used at all).  
Note that this issue is particularly pronounced on AMD clusters because they natively have, e.g., 4 NUMA domains per CPU, i.e. 8 NUMA domains per node (while a regular Intel CPU node has 2 NUMA domains, one per CPU).

**How to fix it?** Ideally, you should initialise (that is fill with zeros) the array in exactly the same manner as you will later access it in parallel. See [STREAMBenchmark.jl](https://github.com/JuliaPerf/STREAMBenchmark.jl) for an example:

Initialisation: [STREAMBenchmark.jl/benchmarks.jl at master · JuliaPerf/STREAMBenchmark.jl · GitHub](https://github.com/JuliaPerf/STREAMBenchmark.jl/blob/master/src/benchmarks.jl#L44-L52)  
Parallel computation: [STREAMBenchmark.jl/kernels\_nthreads.jl at master · JuliaPerf/STREAMBenchmark.jl · GitHub](https://github.com/JuliaPerf/STREAMBenchmark.jl/blob/master/src/kernels_nthreads.jl#L28-L35)

Note that if you would replace the parallel initialisation by simple `zeros` calls the memory bandwidth obtained by STREAMBenchmark.jl will be dramatically lower (i.e. a factor of 5 or similar!).

Of course, in real applications, the parallel kernel might be much more complicated in which case it isn’t at all obvious which parallel initialisation scheme (and thus which NUMA mapping) is optimal. Good news is that in many cases any parallel initialisation is already dramatically better than serial initialisation. Example: [julia-dgemm-noctua/mkl\_job\_128.out at master · carstenbauer/julia-dgemm-noctua · GitHub](https://github.com/carstenbauer/julia-dgemm-noctua/blob/master/Noctua2/MKL/mkl_job_128.out). Here, I perform a regular matrix multiplication (dgemm) on an AMD Milan system using Intel MKL and 128 threads (2 CPUs with 64 threads each).  
Naive initialisation (i.e. serial): ~668 GFLOPs  
Linear parallel initialisation: ~2831 GFLOPs  
Note that a parallel matmul is certainly accessing the input matrices (and the output matrix) in a non-linear way.

Hope this helps.

(BTW, AMD gives one the option to tune the effective number of NUMA domains per socket, see the “NPS” setting [here](https://www.amd.com/system/files/TechDocs/56873_0.80_PUB.pdf)).

---

<div class="post-metadata">

**Author:** ![samo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samo/32/35398_2.png) [@samo](https://discourse.julialang.org/u/samo)\
**Post date:** [July 29, 2022, 8:01am UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/25 "2022-07-29T08:01:20Z")

</div>

> [@carstenbauer](#):
>
> Note that this issue is particularly pronounced on AMD clusters because they natively have, e.g., 4 NUMA domains per CPU, i.e. 8 NUMA domains per node (while a regular Intel CPU node has 2 NUMA domains, one per CPU).

In any case, you should always run a process per CPU as you did. On Intel CPUs you are fine that way. For the AMD CPU case, we will need to add something along the lines what Carsten noted. Until we have it implemented (I think we can add it to the next release of ParallelStencil), you can just initialize yourself the arrays in the CPU case (following Carsten’s notes).  
An alternative for the AMD CPUs would be to also run a process per NUMA domain, with the amount of threads matching the amount of cores available per NUMA domain. Of course, it needs careful pinning of the threads to the right cores.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [July 29, 2022, 11:45am UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/26 "2022-07-29T11:45:13Z")

</div>

> [@samo](#):
>
> An alternative for the AMD CPUs would be to also run a process per NUMA domain, with the amount of threads matching the amount of cores available per NUMA domain.

Indeed, should’ve mentioned this above as a workaround as well.

---

<div class="post-metadata">

**Author:** ![Eurel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eurel/32/35982_2.png) [@Eurel](https://discourse.julialang.org/u/Eurel)\
**Post date:** [August 3, 2022, 11:53am UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/27 "2022-08-03T11:53:12Z")

</div>

Taking the NUMA architecture of the CPU into account, I’ve done my experiments again, launching multiple MPI processes per CPU and the scaling is perfectly linear and near Ideal now !! 😃  
The default pining policy was apparently sufficient.

 ![image](https://global.discourse-cdn.com/julialang/original/3X/9/3/930e1f762444a0ac162a41feb1e3ca0209fc120f.png)

Thank you very much @samo and @carstenbauer for your help!

---

<div class="post-metadata">

**Author:** ![t-bltg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/t-bltg/32/25526_2.png) [@t-bltg](https://discourse.julialang.org/u/t-bltg)\
**Post date:** [August 3, 2022, 12:11pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/28 "2022-08-03T12:11:23Z")

</div>

> [@Eurel](#):
>
> The default pining policy was apparently sufficient

Could you share your launch sequence i.e. how you pinned your processes ?

---

<div class="post-metadata">

**Author:** ![Eurel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eurel/32/35982_2.png) [@Eurel](https://discourse.julialang.org/u/Eurel)\
**Post date:** [August 3, 2022, 12:34pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/29 "2022-08-03T12:34:49Z")

</div>

> how you pinned your processes ?

I did not tune it myself, I did let the resources manager take care of it. The default process distribution policy being Block by node and Block by socket.

The launching sequence is something along the lines of:

```julia
mpirun -p <my cluster> -x -n <nb processes> -c <cores per process> julia -t <cores per process>

```

[Previous page](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170.md?page=1)