# Why does increasing the number of identical parallel operations increases each operation time?
**URL:** https://discourse.julialang.org/t/why-does-increasing-the-number-of-identical-parallel-operations-increases-each-operation-time/57014
**Category:** Performance
**Tags:** linearalgebra, distributed
**Created:** [March 12, 2021, 11:08am UTC](https://discourse.julialang.org/t/why-does-increasing-the-number-of-identical-parallel-operations-increases-each-operation-time/57014 "2021-03-12T11:08:36Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@matthiasbe](https://discourse.julialang.org/u/matthiasbe)
#### Post date: [March 12, 2021, 11:08am UTC](https://discourse.julialang.org/t/why-does-increasing-the-number-of-identical-parallel-operations-increases-each-operation-time/57014/1 "2021-03-12T11:08:36Z")
Hi,
I am investigating on the scaling performance of my distributed linear algebra application. For this purpose I made very basic examples of identical operations done in parallel.
In the following example, I don’t understand **why the execution time of each operation is increasing even though the number of flops is the same.** **Is there something I missed for the cluster configuration ? Is there some multi-threading setting to turn off ?** (I tried `LinearAlgebra.BLAS.set_num_threads(1)` without success.)
Having a constant execution time is important for me to time correctly my scaling experiments.
- Every operation has the same amount of flops.
- There is the same amount of operations than the number of cores subscribed for the program.
## Example 1 : Matrix multiplication
I execute the following code with a different amount of processes
```julia
res = [@spawnat p @elapsed rand(5000,5000) * rand(5000,5000) for p in workers()]
display(mean(map(fetch, res)))
```
## Example 2: Matrix permutation
```julia
res = [@spawnat p @elapsed permutedims(rand(10000,10000)) for p in workers()]
display(mean(map(fetch, res)))
```
## Additional information
I’m running this on a node of a SLURM managed cluster. It is a 32 cores node (2x Cacade Lake Intel Xeon 5218, 16 cores, 2.4GHz).
The initialisation of the processes is done with ClusterManager.jl through a `sbatch` subscription as follows:
```julia-auto
using ClusterManagers
using Distributed
ncores = parse(Int, ENV["SLURM_NTASKS"])
addprocs_slurm(ncores)
```
If you need any extra information I would be happy to provide them.
Thank you for you help.
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [March 12, 2021, 12:07pm UTC](https://discourse.julialang.org/t/why-does-increasing-the-number-of-identical-parallel-operations-increases-each-operation-time/57014/2 "2021-03-12T12:07:23Z")
Part of the problem may be that `rand` accesses variables which are shared. Independently of other problems, it might be a good idea to first generate the matrices, and then benchmark the computations with them (or, alternatively, generate non-random matrices with independent computations on each thread).
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [March 12, 2021, 1:03pm UTC](https://discourse.julialang.org/t/why-does-increasing-the-number-of-identical-parallel-operations-increases-each-operation-time/57014/3 "2021-03-12T13:03:18Z")
> [@matthiasbe](#):
>
> why the execution time of each operation is increasing even though the number of flops is the same.
Memory contention. If you are running on a shared-memory machine, then all the cores are all accessing the same main memory (and may also share some caches), which slows them down as you add more processes.
(You shouldn’t see this on problems that are not memory-bound. For example, try a problem where each processor is just running a long expensive loop but is not accessing any large array or allocating any memory.)
> [@lmiq](#):
>
> Part of the problem may be that `rand` accesses variables which are shared.
I’m not sure what you’re referring to?
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [March 12, 2021, 1:30pm UTC](https://discourse.julialang.org/t/why-does-increasing-the-number-of-identical-parallel-operations-increases-each-operation-time/57014/4 "2021-03-12T13:30:21Z")
> [@stevengj](#):
>
> I’m not sure what you’re referring to?
I am not sure either, but I have experienced problems with multi-threading and random number generators before, and I have seen this arising repeatedly here. I think the problem can be associated with the fact that if one wants to generate a single sequence of random numbers, all threads will be accessing the same seed sequentially, thus there is a problem there if the generation of the random numbers is a limiting step. Anyway, I am not sure when this actually applies, and when it does not.
Here is one post which I could find referring to this kind of problem:
> [@Number of threads available seems random (using @spawn)](https://discourse.julialang.org/t/number-of-threads-available-seems-random-using-spawn/44594/5):
>
> I think the poor scaling that you’re seeing is because the default global rng can suffer from poor performance when called from multiple threads. IIRC, the reason was due to cache invalidation (the default global rng for each thread is stored next to the others in an array without any padding, so when one thread changes the rng state, it forces the cache on all other cores to reload… or something like that.) [me@redmi ~]$ time ~/julia/bin/julia -t2 /tmp/error.jl 2 [DateTime("2020-08-09T15:24:2…
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [March 12, 2021, 1:33pm UTC](https://discourse.julialang.org/t/why-does-increasing-the-number-of-identical-parallel-operations-increases-each-operation-time/57014/5 "2021-03-12T13:33:53Z")
> [@lmiq](#):
>
> think the problem can be associated with the fact that if one wants to generate a single sequence of random numbers, all threads will be accessing the same seed sequentially,
That’s not how `rand` works in Julia; it’s generating “independent” pseudorandom streams (i.e. starting from different seeds) on different processes or threads, so no synchronization is involved.
---
### Author:  [@matthiasbe](https://discourse.julialang.org/u/matthiasbe)
#### Post date: [March 12, 2021, 1:50pm UTC](https://discourse.julialang.org/t/why-does-increasing-the-number-of-identical-parallel-operations-increases-each-operation-time/57014/6 "2021-03-12T13:50:27Z")
I have redone the experiments with the random generation beforehand:
For the first example I replaced with the following code
```julia
@everywhere A = rand(10000,10000)
@everywhere B = rand(10000,10000)
res = [@spawnat p @elapsed A*B for p in workers()]
times = map(fetch, res)
println(mean(times))
```
but the resulting times are still increasing:
For the example 2 the code becomes
```julia
@everywhere A = rand(10000,10000)
res = [@spawnat p @elapsed permutedims(A) for p in workers()]
times = map(fetch, res)
println(mean(times))
```
and the obtained times are constant (0.6 second for 1,…32 processes). I don’t know if the scaling issue came from the RNG or from something else though.
Regarding exemple 1: from the [documentation](https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=192454) I think the node has a shared L3 cache for all the cores. I’m surprised that this can affect the performance that much.
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [March 12, 2021, 2:00pm UTC](https://discourse.julialang.org/t/why-does-increasing-the-number-of-identical-parallel-operations-increases-each-operation-time/57014/7 "2021-03-12T14:00:21Z")
> [@matthiasbe](#):
>
> I don’t know if the scaling issue came from the RNG or from something else though.
I created a [new thread](https://discourse.julialang.org/t/random-number-and-parallel-execution/57024) here about the possible issues with the random number generator, which may or may not be related to what you are seeing. I am not sure either. Let us see what we learn there 🙂