# Lack of improvement from distributed pmap, understanding a simple example

**URL:** <https://discourse.julialang.org/t/lack-of-improvement-from-distributed-pmap-understanding-a-simple-example/121880>\
**Category:** New to Julia\
**Tags:** distributed, pmap\
**Created:** [October 28, 2024, 3:14pm UTC](https://discourse.julialang.org/t/lack-of-improvement-from-distributed-pmap-understanding-a-simple-example/121880 "2024-10-28T15:14:52Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ma-y](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@ma-y](https://discourse.julialang.org/u/ma-y)\
**Post date:** [October 28, 2024, 3:14pm UTC](https://discourse.julialang.org/t/lack-of-improvement-from-distributed-pmap-understanding-a-simple-example/121880/1 "2024-10-28T15:14:52Z")

</div>

Dear all,

I am relatively new to Julia and have spent the last weeks trying to parallelize my code, without much success. Therefore I would like to understand the following minimal example.  
My computer has 16 cores, and I will write down a very simple and parallelizable task, which does not scale well to even 8 workers. The question is why. My real problem is much more complicated, involved the creation of many arrays, and runs much longer, but since I can’t even scale the simple example well, I want to understand it first.  
Say I have a function called heavy\_task that I would like to execute 160 times:

```julia
using Distributed
addprocs(8) #compare 1, 2, 4, 8, 16
workers()

@everywhere function heavy_task(i)
    b = inv(rand(2000, 2000))
    return b[1, 1]
end

```

I aim to do this with pmap and measure the time:

```julia
@time res = pmap(i -> heavy_task(i), 1:160)

```

The timing I get is:

```julia
# 1 worker: 71s
# 2 worker: 42s 
# 4 worker: 28s
# 8 workers: 23s
# 16 workers: 25s

```

My question is why? I tried to make it as simple and parallel as can be, yet the speedup is so minimal. What am I doing wrong?

Thank you so much for your help and many greetings!

---

<div class="post-metadata">

**Author:** ![eldee](https://avatars.discourse-cdn.com/v4/letter/e/b5a626/32.png) [@eldee](https://discourse.julialang.org/u/eldee)\
**Post date:** [October 28, 2024, 6:28pm UTC](https://discourse.julialang.org/t/lack-of-improvement-from-distributed-pmap-understanding-a-simple-example/121880/2 "2024-10-28T18:28:34Z")

</div>

Hi, and welcome to the Julia community!

> [@ma-y](#):
>
> My computer has 16 cores

First of all, if you want to run your code on a single computer, I would recommend multithreading over (distributed) multiprocessing, as this should have less overhead. If you aren’t aware, check out (the [documentation](https://docs.julialang.org/en/v1/manual/multi-threading/) on) `@threads`, `Task`s, and packages such as OhMyThreads.jl.

> [@ma-y](#):
>
> My question is why? I tried to make it as simple and parallel as can be, yet the speedup is so minimal. What am I doing wrong?

At least in your simple example it’s important to note that `inv` is already multithreaded.

```julia-repl
julia> using BenchmarkTools

julia> M = rand(2000, 2000);

julia> @btime inv($M);
  190.308 ms (5 allocations: 31.51 MiB)

julia> using LinearAlgebra; BLAS.get_num_threads()
4

julia> BLAS.set_num_threads(1)

julia> @btime inv($M);
  378.011 ms (5 allocations: 31.51 MiB)

```

If you have 16 cores/threads, you would then expect to get an improvement only up to 4 workers. If you have 32 threads, possibly up to 8.

On my 4C/8T machine, with `BLAS.set_num_threads(1)` (`@everywhere`) I get as timings

```julia
1 worker: 66 s
2 workers: 39 s
4 workers: 26 s
8 workers: 25 s

```

while with `BLAS.set_num_threads(4)` (`@everywhere`) this becomes

```julia
1 worker: 36 s
2 workers: 29 s
4 workers: 33 s
8 workers: 36 s

```

---

<div class="post-metadata">

**Author:** ![ma-y](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@ma-y](https://discourse.julialang.org/u/ma-y)\
**Post date:** [October 28, 2024, 6:40pm UTC](https://discourse.julialang.org/t/lack-of-improvement-from-distributed-pmap-understanding-a-simple-example/121880/3 "2024-10-28T18:40:35Z")

</div>

Thank you so much for your kind reply, this is very helpful.

If we consider the setup with `BLAS.set_num_threads(1) (@everywhere)`, what is the reason that we can’t scale efficiently to 4 workers and that there are almost no gain from going from 4 to 8?

Is it because it’s a memory bound problem or is there something else I am missing (communication is deliberately kept to a minimum in the example, but I do need to create arrays in the problem that matters for me)?

---

<div class="post-metadata">

**Author:** ![Noel\_Araujo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/noel_araujo/32/4862_2.png) [@Noel\_Araujo](https://discourse.julialang.org/u/Noel_Araujo)\
**Post date:** [October 29, 2024, 11:33am UTC](https://discourse.julialang.org/t/lack-of-improvement-from-distributed-pmap-understanding-a-simple-example/121880/4 "2024-10-29T11:33:06Z")

</div>

Hello, from my experience, the problem of scaling rises from the CPU bandwidth utilization. Linear Algebra operations can easy use all bandwidth, and I usually experience some (maybe linear) scaling only using different nodes/computers, or, with matrices sizes that were “smal” - “big” and “smal” are problem and hardware dependent.

On a single computer, I had to balance between number of `process`, number of `blas threads`, and matrices sizes. Maybe all that you need is a single process, running all cores to solve a large matrix, than trying to solve many matrices in parallel.

Just some extra comment, you can setup the number of threads and blas when creating the process

```julia
using Distributed
addprocs(4, 
	exeflags = `--threads 4`, 
	enable_threaded_blas=true)

```

---

<div class="post-metadata">

**Author:** ![sgaure](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sgaure/32/14779_2.png) [@sgaure](https://discourse.julialang.org/u/sgaure)\
**Post date:** [October 29, 2024, 12:07pm UTC](https://discourse.julialang.org/t/lack-of-improvement-from-distributed-pmap-understanding-a-simple-example/121880/5 "2024-10-29T12:07:43Z")

</div>

> [@ma-y](#):
>
> If we consider the setup with `BLAS.set_num_threads(1) (@everywhere)`, what is the reason that we can’t scale efficiently to 4 workers and that there are almost no gain from going from 4 to 8?
> 
> Is it because it’s a memory bound problem or is there something else I am missing (communication is deliberately kept to a minimum in the example, but I do need to create arrays in the problem that matters for me)?

You can measure the memory bandwidth requirements for a single process with LIKWID.jl or LinuxPerf.jl. Then compare it to the memory bandwidth of your computer. Also, some multicore CPUs come with some “performance cores” and … whatever the other cores are. But advertise them just as a bunch of cores.

In general, linear algebra done by BLAS (which most languages do), will easily utilize most resources in the cpu, so things like virtual cores/threads etc. may not work very well.

---

<div class="post-metadata">

**Author:** ![ma-y](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@ma-y](https://discourse.julialang.org/u/ma-y)\
**Post date:** [October 29, 2024, 2:24pm UTC](https://discourse.julialang.org/t/lack-of-improvement-from-distributed-pmap-understanding-a-simple-example/121880/6 "2024-10-29T14:24:38Z")

</div>

> [@Noel\_Araujo](#):
>
> `ads`, and matrices sizes. Maybe all that you need is a single process, running all cores to solve a large matrix, than trying to solve many m

Thank you very much, this is very helpful for me!

---

<div class="post-metadata">

**Author:** ![ma-y](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@ma-y](https://discourse.julialang.org/u/ma-y)\
**Post date:** [October 29, 2024, 2:25pm UTC](https://discourse.julialang.org/t/lack-of-improvement-from-distributed-pmap-understanding-a-simple-example/121880/7 "2024-10-29T14:25:31Z")

</div>

> [@sgaure](#):
>
> S (which most languages do), will easily utilize most resources in the cpu, so things like virtual cores/t

Thank you so much, this is very helpful for me, I will try to measure the bandwidth as you said.