# Dot product not parallelized on cluster

**URL:** <https://discourse.julialang.org/t/dot-product-not-parallelized-on-cluster/92472>\
**Category:** Performance\
**Tags:** linearalgebra\
**Created:** [January 3, 2023, 11:16pm UTC](https://discourse.julialang.org/t/dot-product-not-parallelized-on-cluster/92472 "2023-01-03T23:16:49Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nbakas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nbakas/32/10615_2.png) [@nbakas](https://discourse.julialang.org/u/nbakas)\
**Post date:** [January 3, 2023, 11:16pm UTC](https://discourse.julialang.org/t/dot-product-not-parallelized-on-cluster/92472/1 "2023-01-03T23:16:49Z")

</div>

I am trying to optimize a code involving a dot product which is called many times.

I run the code on a cluster’s node with 256 threads, using the built-in dot.

I check with htop the performance of the threads, and only one or two (out of 256) seem to work.

Also, I use

`BLAS.set_num_threads(32)`

As I’ve found that this is the maximum BLAS threads I can use. Is this true? or could I use all available threads with BLAS?

When I benchmark the dot product only (and not the entire loop), the (32, not 256) threads work but only for big vectors, with ~\>1E7 elements.

I tried many things like custom loops with @turbo, @simd, etc., but nothing seems to improve performance and make all threads work.

Any ideas?  
Many thanks

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [January 3, 2023, 11:20pm UTC](https://discourse.julialang.org/t/dot-product-not-parallelized-on-cluster/92472/2 "2023-01-03T23:20:56Z")

</div>

> [@nbakas](#):
>
> with ~\>1E7 elements.

yeah I don’t think you benefit from multiple threads when the array is small…

1e7 elements is tiny amount

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [January 3, 2023, 11:41pm UTC](https://discourse.julialang.org/t/dot-product-not-parallelized-on-cluster/92472/3 "2023-01-03T23:41:20Z")

</div>

`dot` products are memory bottle-necked so adding cores doesn’t help.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [January 4, 2023, 2:42am UTC](https://discourse.julialang.org/t/dot-product-not-parallelized-on-cluster/92472/4 "2023-01-04T02:42:30Z")

</div>

it doesn’t scale indefinitely but surely 2 is better than 1?

```julia
julia> const a = rand(10^9);

julia> const b = rand(10^9);

julia> using LinearAlgebra

julia> BLAS.set_num_threads(1)

julia> @btime dot(a,b)
  529.519 ms (0 allocations: 0 bytes)
2.5000706579325372e8

julia> BLAS.set_num_threads(2)

julia> @btime dot(a,b)
  357.474 ms (0 allocations: 0 bytes)
2.5000706579323804e8

```

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [January 4, 2023, 3:12am UTC](https://discourse.julialang.org/t/dot-product-not-parallelized-on-cluster/92472/5 "2023-01-04T03:12:00Z")

</div>

Yeah. It depends a lot on the CPU and memory. In general, a rough guideline is 1 core per channel of ram (this is very rough).