# Poor Distributed performance for independent linear algebra operators

**URL:** <https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480>\
**Category:** Performance\
**Created:** [January 7, 2024, 8:51pm UTC](https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480 "2024-01-07T20:51:11Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![hshackle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hshackle/32/47114_2.png) [@hshackle](https://discourse.julialang.org/u/hshackle)\
**Post date:** [January 7, 2024, 8:51pm UTC](https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480/1 "2024-01-07T20:51:11Z")

</div>

I’m trying to run some embarrasingly parallel code using `Distributed` but am seeing poor scaling when scaling up the number of cores. The actual code is rather complicated but for the most part just involves a lot of linear algebra - matrix multiplication, determinants, and similar operators with relatively small (~50x50) matrices. I’ve come up with a MWE with some similar operations that mostly captures the scaling behavior (although the actual code performs slightly worse). The code runs 8 copies of a function which does a bunch of matrix multiplication, and this is spread across _n_ cores (_n_ = 1, 2, 4, 8).

```julia
using Distributed
using BenchmarkTools
ENV["OPENBLAS_NUM_THREADS"] = "1"

@everywhere function f(i::Int)
    for i in 1:100000
        A = rand(30, 30)
        A = A * A
    end
    return
end
@btime res = pmap(f, 1:8)

```

I run this on an AMD Ryzen 7 3700X 8-Core Processor (16 threads, 8 physical CPUs) using `-p n` (_n_ = 1, 2, 4, 8) and get the following output:

```julia
  3.774 s (339 allocations: 13.44 KiB)
  2.239 s (344 allocations: 13.94 KiB)
  1.888 s (354 allocations: 14.94 KiB)
  1.887 s (374 allocations: 16.94 KiB)

```

which only constitutes a ~2x speedup for 8 cores as well as an unusual lack of speedup from 4 to 8 cores. I get roughly similar behavior running on an HPC node with an Intel Xeon Gold 6148 processor. I understand that perfect scaling should not be expected due to shared resources like RAM, cache, etc, but I am wondering if there is any way to improve this given the particular computations of interest. I don’t think this is solely the fault of `Distributed`, as I observe qualitatively similar slowdowns if I just run one process but with multiple instances of Julia through GNU `parallel`.

---

<div class="post-metadata">

**Author:** ![ctkelley](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ctkelley/32/10684_2.png) [@ctkelley](https://discourse.julialang.org/u/ctkelley)\
**Post date:** [January 7, 2024, 10:31pm UTC](https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480/2 "2024-01-07T22:31:24Z")

</div>

30 x 30 seems like a very small problem. Do your results change if you increase the size of the matrix? Does `rand` thread well in that loop? What happens if you replace `rand` with `ones`?

---

<div class="post-metadata">

**Author:** ![hshackle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hshackle/32/47114_2.png) [@hshackle](https://discourse.julialang.org/u/hshackle)\
**Post date:** [January 7, 2024, 11:02pm UTC](https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480/3 "2024-01-07T23:02:51Z")

</div>

Thanks for the suggestions. Changing `rand` to `ones` does not appear to change anything. I cranked the code up to run 1000 multiplications of 300x300 matrices gives the following performance (for 1, 2, 4, and 8 cores):

```julia
  9.603 s (339 allocations: 13.44 KiB)
  4.895 s (344 allocations: 13.94 KiB)
  2.727 s (354 allocations: 14.94 KiB)
  2.057 s (374 allocations: 16.94 KiB)

```

So the scaling does actually appear to improve with larger matrices (around 5x speedup for 8 cores). Unfortunately the code that I’m working with does actually involve large numbers of small matrix multiplications, so I’m hoping to find some way of improving the scaling in that limit.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [January 7, 2024, 11:06pm UTC](https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480/4 "2024-01-07T23:06:46Z")

</div>

Does anything change if you use `BLAS.set_num_threads(1)`? I haven’t seen `ENV["OPENBLAS_NUM_THREADS"] = "1"` before as a way to set th enumber of threads to 1 and I’m not fully convinced that it does anything.

You also might want to try `using MKL` which won’t help parallelism, but will likely speed things up.

---

<div class="post-metadata">

**Author:** ![hshackle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hshackle/32/47114_2.png) [@hshackle](https://discourse.julialang.org/u/hshackle)\
**Post date:** [January 7, 2024, 11:21pm UTC](https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480/5 "2024-01-07T23:21:26Z")

</div>

No, I tried `BLAS.set_num_threads(1)` and nothing changes (note that I can only seem to set this if I add `using LinearAlegbra`, otherwise I get an error that `BLAS` isn’t recognized - I’m a bit confused by this since all the examples I’ve found online don’t seem to mention this dependency, but this probably isn’t significant).

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [January 8, 2024, 9:24am UTC](https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480/6 "2024-01-08T09:24:12Z")

</div>

Could it have to do with allocations? What happens if you do the multiplications in-place with `mul!`?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 8, 2024, 9:43am UTC](https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480/7 "2024-01-08T09:43:50Z")

</div>

> [@Oscar\_Smith](#):
>
> I haven’t seen `ENV["OPENBLAS_NUM_THREADS"] = "1"` before as a way to set th enumber of threads to 1 and I’m not fully convinced that it does anything

[OpenBLAS/README.md at 7a6a24647df49bb0797dfe3d0f43f4dc1389aa41 · OpenMathLib/OpenBLAS · GitHub](https://github.com/OpenMathLib/OpenBLAS/blob/7a6a24647df49bb0797dfe3d0f43f4dc1389aa41/README.md#setting-the-number-of-threads-using-environment-variables) But more than else the problem is setting the environment variable inside the Julia process, I’m not sure that’s useful at that point, I presume that’s only read when the library is initialised at startup.

> [@Oscar\_Smith](#):
>
> You also might want to try `using MKL` which won’t help parallelism, but will likely speed things up.

On an AMD CPU MKL is more likely to have the opposite effect.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [January 8, 2024, 10:22am UTC](https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480/8 "2024-01-08T10:22:07Z")

</div>

> [@Oscar\_Smith](#):
>
> Does anything change if you use `BLAS.set_num_threads(1)`? I haven’t seen `ENV["OPENBLAS_NUM_THREADS"] = "1"` before as a way to set th enumber of threads to 1 and I’m not fully convinced that it does anything.

> [@giordano](#):
>
> But more than else the problem is setting the environment variable inside the Julia process, I’m not sure that’s useful at that point, I presume that’s only read when the library is initialised at startup.

As @giordano anticipated, setting the environment variable only has an effect if you do it before you start Julia. In a running session, you need to use `LinearAlgebra.BLAS.set_num_threads`.

> [@hshackle](#):
>
> ```julia
> for i in 1:100000
> A = rand(30, 30)
> A = A * A
> end
> 
> ```

Emphasising @gdalle’s point: You have many allocations here (200000, i.e. 2 per loop iteration). What scaling do you observe if you allocate `A` only once and reuse it in each loop iteration (with `mul!` from `LinearAlgebra`)?

BTW, which Julia version are you using?

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [January 8, 2024, 12:10pm UTC](https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480/9 "2024-01-08T12:10:01Z")

</div>

Little benchmark on one of our compute nodes ( 2x [AMD Milan 7763](https://www.amd.com/de/products/cpu/amd-epyc-7763)). I pinned the workers to cores in different memory domains.

```julia
using Distributed
using BenchmarkTools
using LinearAlgebra
using DelimitedFiles
using Plots
using ThreadPinning

BLAS.set_num_threads(1)

times = Float64[]
nw = Int[]

for n in 1:8
    addprocs(n)

    @everywhere begin
        using ThreadPinning
        w = findfirst(==(myid()), sort!(workers()))
        # !isnothing(w) && pinthread(w - 1)
        !isnothing(w) && pinthread(first.(cpuids_per_numa())[w])

        function f(i::Int)
            for i in 1:100000
                A = rand(30, 30)
                A = A * A
            end
            return
        end

    end

    t = @belapsed pmap(f, 1:8) samples = 3 evals = 1
    push!(times, t)
    push!(nw, nworkers())
    println(nworkers(), " -> ", t)
    rmprocs(workers())
end

writedlm("results.csv", hcat(nw, times, times_opt), ',')
data = readdlm("results.csv", ',')

plot(data[:, 1], data[1, 2] ./ data[:, 2]; marker=:circle, legend=false, frame=:box, xlabel="nworkers", ylabel="speedup")
plot!([1, 10], [1, 10]; ls=:dash)
savefig("speedup.png")

```

![speedup](https://global.discourse-cdn.com/julialang/original/3X/b/9/b95279fce54f092475d23d7b8f0f77f4a24e1924.png)

I’m not quite sure I understand the plateau for `3 < nworkers < 8` (or the peak at 8) right now but apart from it there is perfect scaling across memory domains.

If we pin the workers naively (to the first N cores, i.e. all in the same memory domain), we get this:  
 ![speedup](https://global.discourse-cdn.com/julialang/original/3X/6/c/6c63cd80089de6fb43d3a19aa60c2b373c56be96.png)

---

<div class="post-metadata">

**Author:** ![hshackle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hshackle/32/47114_2.png) [@hshackle](https://discourse.julialang.org/u/hshackle)\
**Post date:** [January 10, 2024, 9:29pm UTC](https://discourse.julialang.org/t/poor-distributed-performance-for-independent-linear-algebra-operators/108480/10 "2024-01-10T21:29:05Z")

</div>

Thanks! This is really helpful. I would guess the plateau between 4 and 8 is because you get maximum speedup when the number of cores evenly divides 8 (although using that logic I’m surprised 3 cores does as well as it does). Indeed, I can see similar performance gains on my cluster.

Regarding allocations, cutting down on allocations by doing things in-place with `mul!` does appear to recover more or less a perfect scaling. This has been helpful to keep in mind with the actual code that I’m trying to optimize; I’ve had success gradually improving the scaling by reducing allocations. This wasn’t obvious to me initially because the code running on one worker doesn’t see too significant of an improvement by being deliberate about in-place operations. But it does seem to be important for the scaling, which makes sense.