# Poor scaling results with \`ImplicitGlobalGrid.jl\`
**URL:** https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170
**Category:** Julia at Scale
**Created:** [July 23, 2021, 12:49pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170 "2021-07-23T12:49:20Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@francispoulin](https://discourse.julialang.org/u/francispoulin)
#### Post date: [July 23, 2021, 12:49pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/1 "2021-07-23T12:49:20Z")
I have tried running one of the examples of `ImplicitGlobalGrid.jl` on 1, 2 and 4 gpus to reproduce some scaling results. I started wtih `diffusion3D_multigpu_CuArray_novis.jl` with a coarser grid to see how it does. Below you will find my results.
For this weak-scaling problem I compute the efficiencies on 2 and 4 cores to be 57 and 35 percent respectively. I realize this is much coarser but is this surprising?
I have ensured that the system does have CUDA-aware MPI so that would not be a problem, I don’t think.
```julia
$ mpiexec -np 1 julia --project diffusion3D_multigpu_CuArrays_novis.jl
Global grid: 32x32x32 (nprocs: 1, dims: 1x1x1)
Simulation time = 41.547354999
$ mpiexec -np 2 julia --project diffusion3D_multigpu_CuArrays_novis.jl
Global grid: 62x32x32 (nprocs: 2, dims: 2x1x1)
Simulation time = Simulation time = 72.528739102
72.494481903
$ mpiexec -np 4 julia --project diffusion3D_multigpu_CuArrays_novis.jl
Global grid: 62x62x32 (nprocs: 4, dims: 2x2x1)
Simulation time = Simulation time = Simulation time = Simulation time = 116.549162142
116.549022381
```
I was informed by @luraess that there are two problems with what I have done. Below is a copy of the points from the issue that I initially posted.
> The weak scaling timings you report are to be expected since:
>
> - you are running a very small local problem size (32x32x32) -the ratio of boundary points over total grid points is large and communication time is no longer negligible- and;
> - the `diffusion3D_multigpu_CuArray_novis.jl` does not implement the `@hide_communication` feature to overlap and hide MPI communication by inner domain grid points computations.
I am no retrying it with a larger grid, 256^3, as the code was originally set up so I know that should improve things. However, the second point is a bit harder to figure out. Is there an example of this code that uses @hide\_communication that I could run instead?
---
### Author:  [@luraess](https://discourse.julialang.org/u/luraess)
#### Post date: [July 23, 2021, 2:19pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/2 "2021-07-23T14:19:13Z")
Hi @francispoulin, thanks for your question and interest in implicitGlobalGrid !
You can find examples of using the `@hide_communication` feature in the multixpu miniapp codes from [ParallelStencil.jl](https://github.com/omlins/ParallelStencil.jl), e.g. the [acoustic3D](https://github.com/omlins/ParallelStencil.jl/blob/24032a3b814d197e2a85f544ed24146df4f08493/miniapps/acoustic_waves_multixpu/acoustic3D_multixpu.jl#L63), and in the diffusion example from the JuliaCon2021 workshop on [solving PDEs on GPUs](https://github.com/luraess/parallel-gpu-workshop-JuliaCon21/blob/e78c7295e303bff1c47d9c319ffb57b50b29c4a0/scripts/diffusion_2D_damp_perf_multixpu.jl#L90).
---
### Author:  [@francispoulin](https://discourse.julialang.org/u/francispoulin)
#### Post date: [July 23, 2021, 2:28pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/3 "2021-07-23T14:28:22Z")
Thanks again @luraess.
I find the example below that seems to do what I want using `@hide_communication` and I’m going to play with that to see how I can get good scaling for the diffusion equation.
[https://github.com/omlins/ParallelStencil.jl/blob/main/examples/diffusion3D\_multigpucpu\_hidecomm\_novis.jl](https://github.com/omlins/ParallelStencil.jl/blob/main/examples/diffusion3D_multigpucpu_hidecomm_novis.jl)
---
### Author:  [@francispoulin](https://discourse.julialang.org/u/francispoulin)
#### Post date: [July 23, 2021, 5:39pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/4 "2021-07-23T17:39:48Z")
I wanted to shape an update.
I added the following line `select_device()` into the diffusion code and ran it on the same server with 1, 2 and 4 GPUs with a grid of `256x256x256`.
I’m happy to say that for both 2 and 4 GPUs the efficiency was 97%. I didn’t use any special optimization and didn’t use `@hide_communication` so maybe it can be made even better, but I am very happy with the results.
Is the `select_device()` line something that you would want to include in the code as it might help others you are looking for optimized code?
---
### Author:  [@luraess](https://discourse.julialang.org/u/luraess)
#### Post date: [July 24, 2021, 12:15pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/5 "2021-07-24T12:15:28Z")
Thanks for sharing your results. Regarding your feedback:
> for both 2 and 4 GPUs the efficiency was 97%
Good to read you achieved good performance !
> didn’t use `@hide_communication` so maybe it can be made even better
Would be interesting to see how it changes using `@hide_communication` as you can maybe have close to 100% efficiency.
> Is the `select_device()` line something that you would want to include in the code
Not fo now I guess. The `select_device()` is only useful in configurations where one server hosts multiple GPUs, as then the GPUs can be assigned to the local MPI ranks using shared memory pool. If it turns out that in the future such GPU arrangement is the most standard, we may indeed update the examples to include it “by default”.
Here some further suggestions to ensure your multi-GPU code runs as expected:
- check `nvidia-smi` or print the device ID to ensure you are running on different GPUs when running multiple MPI processes
- use Nvidia visual profiler if you want to ensure that the communication-computation overlap functions properly using `@hide_communication`
Note that using `@hide_communication` may not lead to significant performance increase when running multiple GPU-MPI processes on a single node as there inter-process communication may be fast given the high-connectivity of the GPUs (e.g. Nvlink). However, when running on physically different nodes, communication (over Ethernet or Infiniband) network may be significantly slower bringing up the need to hide communication time.
---
### Author:  [@francispoulin](https://discourse.julialang.org/u/francispoulin)
#### Post date: [July 24, 2021, 3:48pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/6 "2021-07-24T15:48:24Z")
Thanks for your insights @luraess .
Next, I will try the other diffusion3D code that uses `@hide_communication`. If it performs better, then I will certainly use that approach in the future. But I must say that 97% is pretty excellent by my standards. However, I imagine this would decrease more rapidly when going on to more gpus, and probably would not perform near as well in going to +5000 cores, like you have done before.
Thanks again for writing such good software that is fun to play with!
---
### Author:  [@francispoulin](https://discourse.julialang.org/u/francispoulin)
#### Post date: [July 24, 2021, 4:19pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/7 "2021-07-24T16:19:49Z")
One more question for @luraess . In the mini-course yesterday you give a lot of different code to solve the diffusion equation in 3D, which was a lot of fun. I know that you have shown that multi-GPU can be efficient on thousands of cores when using MPI. Do you know how high of efficiency we can get with multi-threading?
We have done some preliminary tests and even with large matrices, 512^3, we seem to get low efficiencies and saturation at 16 cores. I am tempted to try playing with the threaded code you discussed yesterday but thought I would ask you first, if you have any experience you can share.
Thank you again.
---
### Author:  [@luraess](https://discourse.julialang.org/u/luraess)
#### Post date: [July 25, 2021, 9:03pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/8 "2021-07-25T21:03:15Z")
Thanks for your feedback !
> Do you know how high of efficiency we can get with multi-threading?
Wouldn’t you mind opening another Discourse thread for this different topic? This would help other users to hit relevant responses upon future searches. Doing so, you could also ping @samo who would be able to give you some feedback on that. Thanks!
---
### Author:  [@francispoulin](https://discourse.julialang.org/u/francispoulin)
#### Post date: [July 26, 2021, 12:51pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/9 "2021-07-26T12:51:17Z")
Very good point @luraess. I will do that right now.
---
### Author:  [@Eurel](https://discourse.julialang.org/u/Eurel)
#### Post date: [May 20, 2022, 1:12pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/10 "2022-05-20T13:12:43Z")
@luraess I am having trouble getting good performance on [this example](https://github.com/omlins/ParallelStencil.jl/blob/main/examples/diffusion3D_multigpucpu_hidecomm.jl) which uses the hide-comms functionnality, I am runing it on a few nodes composed of four 16core CPUs each, having 128 threads per nodes. (USE\_GPU=false)
With some script like this
```julia
for N in {1,2,3}
do
mpirun -n ${N} julia -t 128 ../travail/julia-heatdiff/scripts/run.jl
done
```
I get the following results, the bandwidth decreases when I increase the number of compute nodes
```julia
Global grid: 512x512x512 (nprocs: 1, dims: 1x1x1)
time_s=5.118296146392822 T_eff=56.64195353063296
Global grid: 1022x512x512 (nprocs: 2, dims: 2x1x1)
time_s=5.26022481918335 T_eff=55.11366955699977
Global grid: 1532x512x512 (nprocs: 3, dims: 3x1x1)
time_s=8.300725936889648 T_eff=34.92589620283643
Global grid: 1022x1022x512 (nprocs: 4, dims: 2x2x1)
time_s=8.083126068115234 T_eff=35.86611046728351
```
Am i launching it wrong?
---
### Author:  [@luraess](https://discourse.julialang.org/u/luraess)
#### Post date: [May 20, 2022, 1:45pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/11 "2022-05-20T13:45:24Z")
Thanks @Eurel for reporting this. Nothing wrong on your side I guess.
The relevant doc section from the [ParallelStencil README](https://github.com/omlins/ParallelStencil.jl#seamless-interoperability-with-communication-packages-and-hiding-communication) never explicitly state that the hide communication feature is currently only enabled when executing code on Nvidia GPUs, but it should. Because then, one can actually take advantage of async execution of streams as a concise way to prioritise concurrent subdomain execution in order to realise the communication/computation overlap. I will fix it here [https://github.com/omlins/ParallelStencil.jl/issues/56](https://github.com/omlins/ParallelStencil.jl/issues/56)
---
### Author:  [@Eurel](https://discourse.julialang.org/u/Eurel)
#### Post date: [May 24, 2022, 2:23pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/12 "2022-05-24T14:23:22Z")
Thank you for your quick answer and sorry for my slow responsiveness…
Makes sense for the hide\_communication clause, thank you for making that clear to me now.
But it wasn’t exactly what i had in mind, as i didn’t express myself correctly, I am so sorry for that.
When I increase the number of procs, shouldn’t the compute time decrease?
Is it why the global grid increase?
That behavior isn’t exactly clear to me as I expected the global grid to be distributed amongst each processes, the grid is instead duplicated is that right?
I am sorry in advance if I ask trivial questions.
---
### Author:  [@luraess](https://discourse.julialang.org/u/luraess)
#### Post date: [May 24, 2022, 11:31pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/13 "2022-05-24T23:31:26Z")
Hi @Eurel , no worries. I guess you suggested it.
> When I increase the number of procs, shouldn’t the compute time decrease?
> Is it why the global grid increase?
The behaviour you describe may occur in a strong scaling configuration, i.e., if the global problem size is fixed; the more procs one adds, the smaller the local problem size (one each proc) becomes and one may observe some (limited) speed-up.
> That behavior isn’t exactly clear to me as I expected the global grid to be distributed amongst each processes, the grid is instead duplicated is that right?
On GPUs in particular, one gets the best performance for a specific local problem size. In that case, it makes sense to look at the distributed memory problem in a weak scaling approach; that’s what is done in ImplicitGlobalGrid. You define the optimal local problem size and then scale it the resources for your global problem, ensuring optimal execution by construction. Hope this helps ([see, e.g., this figure](https://github.com/eth-cscs/ImplicitGlobalGrid.jl/blob/master/docs/images/implicit_global_grid.png)).
---
### Author:  [@Eurel](https://discourse.julialang.org/u/Eurel)
#### Post date: [July 25, 2022, 2:10pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/14 "2022-07-25T14:10:59Z")
I’ve done some scaling experiments on up to 32 64 cores CPUs (2048 threads in total) and 12 GPUs and whilst GPU scaling is excellent, the CPU scaling is way not as good whilst the code is the same.
As on the github page, the scaling is reported on GPUs, Is it expected to be the case for multi CPUs
experiments?
here are some of my results
Thanks or your time, help and all of your work with this Library 😃
---
### Author:  [@Eurel](https://discourse.julialang.org/u/Eurel)
#### Post date: [July 25, 2022, 2:12pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/15 "2022-07-25T14:12:06Z")
meant to reply to @luraess
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 27, 2022, 6:01pm UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/16 "2022-07-27T18:01:16Z")
Hi @Eurel, just with the plots it is a bit difficult to get an understanding of the issue. Can you give some more information?
- How are you using the cores of each CPU? Are you using multi-threading? How have you set `JULIA_NUM_THREADS`?
- Are you running 32 MPI processes or more?
- what performance do you get with 1 CPU (meaning using all 64 cores)?
- how many CPUs (meaning physical - with 64 cores) are on one compute node?
---
### Author:  [@Eurel](https://discourse.julialang.org/u/Eurel)
#### Post date: [July 28, 2022, 9:35am UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/17 "2022-07-28T09:35:35Z")
I am using ParallelStencil.jl for multi-threading, I set the number of threads to 64 with the -t command line argument, I did not set the environment variable
On these experiments, I ran up to 32 MPI processes.
The kind processor I use is a 64core (hyperthreading disabled) AMD milan [[https://www.amd.com/fr/products/cpu/amd-epyc-7763](https://www.amd.com/fr/products/cpu/amd-epyc-7763)] (when I ploted the graph the milan partitions were out) and i get from 100 GB/s to 120 GB/s on one CPU (one MPI process) out of the 204GB/s announced on the vendor sheet.
Each node is composed of 2 CPUs (dualsocket)
I do not specify to the batch scheduler (custom Slurm ) how many nodes i require, but how many processes I need.
I would typycally lauch it like this
```julia
mpirun -n -c 64 -x julia -t 64 benchmark.jl
```
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 28, 2022, 9:57am UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/18 "2022-07-28T09:57:47Z")
two more questions about the setup please:
- is it 2-D or 3-D?
- what resolution do you solve per CPU?
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 28, 2022, 10:14am UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/19 "2022-07-28T10:14:30Z")
In addition, can you comment all the `update_halo!` calls in your code and **leave all the rest exactly as it is** and redo the scaling experiment? That way you have an embarrassingly parallel application without communication, which should scale ideally.
This way we know if the performance loss is in the halo updates or if there is something fundamentally wrong with how you are launching the job.
---
### Author:  [@Eurel](https://discourse.julialang.org/u/Eurel)
#### Post date: [July 28, 2022, 11:27am UTC](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170/20 "2022-07-28T11:27:32Z")
It is an explicit 2D heat diffusion, in which settled for a 2048\*2048 tile resolution.
Here’s a graph of the throughput of all sizes.
> In addition, can you comment all the `update_halo!` calls in your code and **leave all the rest exactly as it is** and redo the scaling experiment?
Thanks that is a very good idea. Gonna run the experiment without `update_hallo!` right away !
[Next page](https://discourse.julialang.org/t/poor-scaling-results-with-implicitglobalgrid-jl/65170.md?page=2)