# Julia code becomes slower on running on supercomputers and does not scale well when parallelizing with Base.Threads

**URL:** <https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439>\
**Category:** Julia at Scale\
**Tags:** fortran, parallel, linearalgebra, threads\
**Created:** [January 6, 2024, 2:11pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439 "2024-01-06T14:11:51Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [January 17, 2024, 2:02pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/43 "2024-01-17T14:02:50Z")

</div>

> [@ufechner7](#):
>
> > [@Uranium238](#):
> >
> > It is about `2.84%`.
> 
> Well, if that is correct garbage collection would not be the limiting factor.

There is the following issue: When the runtime decides to run GC, then it needs to wait until all threads have reached a GC safepoint. The compiler makes no attempt to guarantee a maximal amount of compute between safepoints (as opposed to eg JVM).

Hence you can run into the situation where you have 80 threads, one of which has a long-running tight loop without safepoints, and 79 are waiting for this one to finish the loop until they can finally GC.

It used to be that time spent waiting was not counted towards the GC percentage.

I am not 100% sure that this is still correct.

TL;DR: Even if you get reported 0.1% GC time on an 80 thread machine, this tiny amount of CPU time may account for effective `98.75%` of wall-clock time.

You will see this by running `top` or `htop` as ~100% CPU utilization (instead of the ideal 8000%). You can also see this by running `time julia ...` and looking at the times reported by the kernel.

Can you @Uranium238 post the output of `time julia ...`? That should show us whether the non-ideal scaling is due to threads waiting on locks.

PS. In an output of `time` like the following

```julia
real	0m0.107s
user	0m0.061s
sys	0m0.056s

```

you can compare `real` (wall-clock-time) to `user` (cpu-seconds) to gauge CPU utilization (ideal multithreading means `user == real * ncpus`). If `sys` (cpu-seconds spent in kernel that are attributed to you) is large, then something is very broken and you need to think hard about what went wrong. You will e.g. get this if you write single characters to unbuffered stdout. You start debugging that via `strace`.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [January 17, 2024, 2:07pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/44 "2024-01-17T14:07:34Z")

</div>

Slightly off topic - it is alwasy interesting to run the ‘lstopo’ utility then get a graphical view of the internal layour of your servers  
[Portable Hardware Locality (hwloc) (open-mpi.org)](https://www.open-mpi.org/projects/hwloc/lstopo/)

Also agree with @foobar_lv2 - run top or htop

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [January 17, 2024, 2:44pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/45 "2024-01-17T14:44:10Z")

</div>

FWIW, this is what I get here:

Original version:

```julia
leandro@m3g:~/Downloads/uranium% time julia -t 10 --project pal-rv6.jl 
Number of threads = 10
The rate constant is 2.3048765809185653e9
343.218980 seconds (101.98 M allocations: 3.746 TiB, 1.95% gc time, 13.97% compilation time: <1% of which was recompilation)

real 5m45,960s
user 52m58,936s
sys 0m8,036s

```

Memory-efficient version:

```julia
leandro@m3g:~/Downloads/uranium% time julia -t 10 --project pal-rv6_2.jl 
Number of threads = 10
311.750008 seconds (37.96 M allocations: 20.854 GiB, 0.17% gc time, 15.86% compilation time)
The rate constant is 2.304876582130397e9
312.690927 seconds (40.85 M allocations: 21.053 GiB, 0.18% gc time, 16.10% compilation time: <1% of which was recompilation)

real 5m15,405s
user 46m39,963s
sys 0m2,472s

```

(using `dd = 1000`)

I’m somewhat intrigued by the compilation time, which seems to increase (in absolute terms) with system size.

---

<div class="post-metadata">

**Author:** ![Uranium238](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uranium238/32/209706_2.png) [@Uranium238](https://discourse.julialang.org/u/Uranium238)\
**Post date:** [January 18, 2024, 10:24am UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/46 "2024-01-18T10:24:56Z")

</div>

> [@ufechner7](#):
>
> I just tried:

I tried the exact same code on the supercomputer in question. It took 41.769s !

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [January 18, 2024, 10:32am UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/47 "2024-01-18T10:32:14Z")

</div>

Did you try both versions, with and without `using MKL` ?

On my desktop PC (Ryzen 7950X) I get:

```julia
ufechner@ufryzen:~/repos/bench$ julia --project bench.jl 
  3.475 s (6 allocations: 767.90 MiB)

```

with MKL.

---

<div class="post-metadata">

**Author:** ![Uranium238](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uranium238/32/209706_2.png) [@Uranium238](https://discourse.julialang.org/u/Uranium238)\
**Post date:** [January 18, 2024, 10:51am UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/48 "2024-01-18T10:51:40Z")

</div>

> [@ufechner7](#):
>
> Did you try both versions, with and without `using MKL` ?
> 
> On my desktop PC (Ryzen 7950X) I get:

My reported value was without using `MKL` , I shall report the value while `using MKL` as soon as I am able to get access to the cluster.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [January 18, 2024, 2:09pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/49 "2024-01-18T14:09:14Z")

</div>

I tried the exact same code on the supercomputer in question. It took 41.769s !

I would honestly say to start by running ‘top’ or ‘htop’ when this application is running.  
Look for any other applications running on the compute node at the same time.  
Look for processes hopping around between different CPU cores.

Are you running into ‘time to first X’ here, maybe you have a very slow storage system ?

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [January 18, 2024, 2:30pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/50 "2024-01-18T14:30:38Z")

</div>

> [@johnh](#):
>
> Are you running into ‘time to first X’ here, maybe you have a very slow storage system ?

Cannot be, my benchmark uses the `@btime` macro:

```julia
# using MKL
using BenchmarkTools

n=10000
m=rand(n,n)
@btime inv($m)

```

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [January 18, 2024, 2:58pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/51 "2024-01-18T14:58:43Z")

</div>

@ufechner7 my question was directed to @Uranium238  
Apologies

---

<div class="post-metadata">

**Author:** ![Uranium238](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uranium238/32/209706_2.png) [@Uranium238](https://discourse.julialang.org/u/Uranium238)\
**Post date:** [January 18, 2024, 3:53pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/52 "2024-01-18T15:53:55Z")

</div>

> [@johnh](#):
>
> You are comparing threaded code with MPI code.  
> Fortran can used MPI or can use OpenMP For threaded code.  
> Julia can use MPI or can use threads.  
> I would say you chould compare MPI to MPI or threaded to threaded.

The Fortran code (as you might have seen from the git project I linked in this thread) uses MPI (Not OpenMP). As I stated before, I completely realize that it is not right for me to compare threaded (`Base.Threads`) performance with the Fortran code using MPI, I should have used MPI.jl .

As as has been stated in this thread, `Base.Threads` has the lowest overhead out of all parallelization techniques that can be employed in Julia. So switching to MPI.jl should definetly not improve the performance. And if the threaded Julia code is slower than the MPI Fortran code, I do not feel that switching to MPI.jl would serve my primary purpose which is to make the Julia code run faster than Fortran code while using the same number of processors.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [January 18, 2024, 4:03pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/53 "2024-01-18T16:03:31Z")

</div>

We do not have enough information to really know what is going on on the cluster, but your code bottleneck is the matrix multiplication, by far. That suggests communication among process is not really what might be making your code slower, so I would not bet on MPI vs. Threads differences here. The Julia code might become slower because you have _a lot_ of memory usage in the original code, which has to be constantly garbage-collected. Thus the need to check if the more memory-friendly version there solves the scalability issue.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 18, 2024, 4:23pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/54 "2024-01-18T16:23:48Z")

</div>

> [@Uranium238](#):
>
> my code becomes slower ! taking `52` seconds instead of `47` seconds. […] I think it is of paramount importance

A 10% difference when you go from one machine to another (with the same number of cores?) is not unusual and may not be too important.

> [@Uranium238](#):
>
> The Fortran code uses MPI parallelization, whereas my Julia code relies on the features of `Base.Threads`

Even on shared-memory hardware, if you aren’t careful about memory allocation & access, then using threads with shared memory can hurt you because you have more cache contention (compared to MPI with completely separate address spaces for each process). And this effect varies depending on the machine and its memory architecture, so it could easily change dramatically between your desktop and your server.

You are also allocating zillions of temporary arrays in your inner loop rather than re-using buffers and working in-place, which hurts performance (both serial and parallel). Note also that slices in Julia , like `MJSJ[mp,:]`, allocate copies unless you mark your code with `@views` (which can be applied to the whole function). This is all covered in the [Julia performance tips](https://docs.julialang.org/en/v1/manual/performance-tips/#man-performance-tips).

(All of your type declarations accomplish exactly nothing for performance, by the way. But missing `const` on globals like `T` and `t` may hurt you. By the way, there is no point in using `plan_fft` if you are going to use the plan only once, but this does not look performance critical for you anyway.)

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [January 18, 2024, 5:37pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/55 "2024-01-18T17:37:46Z")

</div>

In my “supercomputer” (a 12-year old Xenon cluster), I can test the scalability in a machine with 16 real cores. I get:

### Original version (with dd = 100):

With 8 cores:

```julia-auto
leandro@gn003:~/Downloads/uranium% julia -t8 --project pal-rv6.jl
Number of threads = 8
The rate constant is 2.175134435831123e9
129.216527 seconds (18.28 M allocations: 384.064 GiB, 3.39% gc time, 71.59% compilation time)

```

With 16 cores:

```julia-auto
leandro@gn003:~/Downloads/uranium% julia -t16 --project pal-rv6.jl 
Number of threads = 16
The rate constant is 2.175134435831123e9
 87.336051 seconds (18.43 M allocations: 384.077 GiB, 3.95% gc time, 210.67% compilation time)

```

A an improvement of 1.4x

#### The memory-optimized version (with dd = 100):

```julia-auto
leandro@gn003:~/Downloads/uranium% julia -t8 --project pal-rv6_2.jl
Number of threads = 8
119.344040 seconds (12.83 M allocations: 2.667 GiB, 0.52% gc time, 90.66% compilation time)
The rate constant is 2.1751344364095407e9

leandro@gn003:~/Downloads/uranium% julia -t16 --project pal-rv6_2.jl 
Number of threads = 16
 69.198384 seconds (12.96 M allocations: 2.679 GiB, 0.83% gc time, 303.68% compilation time)
The rate constant is 2.1751344364095407e9

```

An improvement of 1.72x for 2x the number of cores.

With a larger system that is probably better. Those compilation times do not make sense (but that’s a separate issue I’m trying to report).

---

<div class="post-metadata">

**Author:** ![abraemer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abraemer/32/51403_2.png) [@abraemer](https://discourse.julialang.org/u/abraemer)\
**Post date:** [January 18, 2024, 9:49pm UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/56 "2024-01-18T21:49:48Z")

</div>

> [@lmiq](#):
>
> Those compilation times do not make sense (but that’s a separate issue I’m trying to report).

I’ve seen similar things with threads. I think the (absolute) compilation time is essentially multiplied by the number of threads (perhaps each thread compiles its own version?). This is then compared to the runtime of the function (not multiplied by the thread count). Not sure about this though! I think I experimented at some point and for that example the math worked out but I can’t recreate anything unfortunately.

---

<div class="post-metadata">

**Author:** ![Uranium238](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uranium238/32/209706_2.png) [@Uranium238](https://discourse.julialang.org/u/Uranium238)\
**Post date:** [January 19, 2024, 3:37am UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/57 "2024-01-19T03:37:36Z")

</div>

> [@stevengj](#):
>
> All of your type declarations accomplish exactly nothing for performance, by the way.

Could you please elaborate why that is the case ? As far as I know it is always better to declare the type of every variable that is going to be passed onto any function.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [January 19, 2024, 3:50am UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/58 "2024-01-19T03:50:13Z")

</div>

No. Julia specializes the code based on the types you call the function with. The annotations on functions only control dispatch.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [January 19, 2024, 3:56am UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/59 "2024-01-19T03:56:58Z")

</div>

Type declarations can be helpful, but they do not actually help performance directly in Julia.

Indirectly, they can help ensure your code is type stable, but the same type stable code without typed arguments or type assertions would execute just the same. The main effect of adding type to arguments is restricting the types of potential arguments or multiple dispatch.

Julia uses a late binding call model so functions are specialized based on the arguments they are called with, not the arguments they are declared with.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [January 19, 2024, 4:09am UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/60 "2024-01-19T04:09:12Z")

</div>

An increase in execution time on a HPC multisocket compute node can be expected in code that is constrained by memory limitations. This is due to the shared memory model on multisocket systems called Non-Uniform Memory Access.

I have found that the issue is particularly pronounced on Intel Skylake and Cascade Lake Xeons. If you have access to the hardware, you might find that a single Xeon system is actually faster than multiple Xeons. In my experience, I got a speedup by removing one Xeon from a two Xeon system.

From a pure software perspective, you can reduce this effect by reducing memory allocations.

Here are few reports about poor memory performance with dual socket Xeons:

> <https://serverfault.com/questions/915091/memcpy-bandwidth-1-6x-faster-on-1-vs-2-socket-intel-scalable-skylake>

> **[Single-threaded memory performance for dual socket Xeon E5-\* systems](https://community.intel.com/t5/Software-Tuning-Performance/Single-threaded-memory-performance-for-dual-socket-Xeon-E5/m-p/978734)**
>
> Hi all, I had originally asked this question in a separate Intel community forum (https://communities.intel.com/thread/50808), but it was suggested that I repost here. There is also a stackoverflow question from another user linked in the other...

---

<div class="post-metadata">

**Author:** ![Uranium238](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uranium238/32/209706_2.png) [@Uranium238](https://discourse.julialang.org/u/Uranium238)\
**Post date:** [January 19, 2024, 8:38am UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/62 "2024-01-19T08:38:17Z")

</div>

> [@mkitti](#):
>
> I have found that the issue is particularly pronounced on Intel Skylake and Cascade Lake Xeons.

I am trying to run the same two codes on AMD systems to get an idea of how much of the problem is caused by the Xeon chipset.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [January 19, 2024, 9:18am UTC](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439/63 "2024-01-19T09:18:02Z")

</div>

You are getting ahead of your skis here.  
Take my advice please - ask your system admins for help in running the ‘lstopo’ utility to produce graphical images of the internal layout of the Xeon and AMD systems.  
Dual socket Xeon and AMD both have NUMA setups.  
The exact layout in NUMA terms can be altered by BIOS settings.  
[PowerEdge: NUMA Nodes per Socket | CPU Best Practices | Dell Technologies Info Hub](https://infohub.delltechnologies.com/l/cpu-best-practices-3/poweredge-numa-nodes-per-socket-1/)

Yes the NPS Setting does have an effect on performance (I cannot remember if Skylake has an NPS setting)

I would be very surprised if the effects you are seeing are down to the difference in AMD and Xeon chipsets.

Please consider the effects of hyperthreading - which you have enabled.  
You really only have 40 physical caores, not the 80 which the sysem presents.  
I cant say how much hyperthreading behaves with your code.  
Also please consider and tell us the BLAS Libraries being used on each of your systems. From the above discussions BLAS may be dominating things here.

[Previous page](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439.md?page=2)

[Next page](https://discourse.julialang.org/t/julia-code-becomes-slower-on-running-on-supercomputers-and-does-not-scale-well-when-parallelizing-with-base-threads/108439.md?page=4)