# Matrix Addition with multithreads!

**URL:** <https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674>\
**Category:** Numerics\
**Created:** [September 1, 2017, 4:25pm UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674 "2017-09-01T16:25:51Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![usefulhyun](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/usefulhyun/32/1770_2.png) [@usefulhyun](https://discourse.julialang.org/u/usefulhyun)\
**Post date:** [September 1, 2017, 4:25pm UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674/1 "2017-09-01T16:25:51Z")

</div>

Hi, there!

I’d like to do matrix addition with multithreads.  
I can do matrix multiplication with multithreads via BLAS.set\_num\_threads(t).  
But I do not know how to do this with matrix addition.  
Can Anyone help me?

---

<div class="post-metadata">

**Author:** ![RoyiAvital](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/royiavital/32/571_2.png) [@RoyiAvital](https://discourse.julialang.org/u/RoyiAvital)\
**Post date:** [September 1, 2017, 5:57pm UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674/2 "2017-09-01T17:57:33Z")

</div>

Why wouldn’t you let the BLAS library decide what’s best to do?

Is it experiment or you want the best performance?

I’m not an expert but I’d assume Matrix Addition is a memory bounded operation (When vectorized) hence Multi Threading isn’t going to help.

---

<div class="post-metadata">

**Author:** ![ksmcreynolds](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ksmcreynolds/32/860_2.png) [@ksmcreynolds](https://discourse.julialang.org/u/ksmcreynolds)\
**Post date:** [September 1, 2017, 7:45pm UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674/3 "2017-09-01T19:45:58Z")

</div>

I’m not sure how much speed you will get, but you can use `@threads` and write your own addition function like the one below. Be sure to set the number of threads before starting `julia` with the environment variable `JULIA_NUM_THREADS`. See the [Multi-Threading section](https://docs.julialang.org/en/stable/manual/parallel-computing/#Multi-Threading-(Experimental)-1) in the docs for more info.

```julia
function matrixadd(A::Matrix{T}, B::Matrix{T}) where T
    C = similar(A)
    Threads.@threads for i = 1:length(A)
        @inbounds C[i] = A[i] + B[i]
    end
    C
end

```

Of course preallocating the output always helps. Just using `@. C = A + B` is probably going to give you pretty good performance.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 1, 2017, 10:34pm UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674/4 "2017-09-01T22:34:10Z")

</div>

It’s been mentioned before that native Julia matches BLAS at least for level 1 BLAS, so I would assume @ksmcreynolds’s solution gets close or at least very close to using `axpy!` when allocating. But I would make a few changes:

```julia
function matrixadd!(C::AbstractArray{T},A::AbstractArray{T}, B::AbstractArray{T}) where T
   @boundscheck checkbounds(C,eachindex(B))
   @boundscheck checkbounds(C,eachindex(A))
    Threads.@threads for i = 1:eachindex(C)
        @inbounds C[i] = A[i] + B[i]
    end
end
matrixadd(A::AbstractArray{T}, B::AbstractArray{T}) where T = (C = similar(A); matrixadd!(C,A,B); C)

```

---

<div class="post-metadata">

**Author:** ![tkoolen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkoolen/32/1603_2.png) [@tkoolen](https://discourse.julialang.org/u/tkoolen)\
**Post date:** [September 2, 2017, 1:51am UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674/5 "2017-09-02T01:51:27Z")

</div>

Small correction: `1:eachindex(C)` should just be `eachindex(C)`, right?

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [September 2, 2017, 2:52am UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674/6 "2017-09-02T02:52:03Z")

</div>

you can also use [GPUArrays.jl](https://github.com/JuliaGPU/GPUArrays.jl) which has a threaded backend!

---

<div class="post-metadata">

**Author:** ![usefulhyun](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/usefulhyun/32/1770_2.png) [@usefulhyun](https://discourse.julialang.org/u/usefulhyun)\
**Post date:** [September 2, 2017, 4:38am UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674/7 "2017-09-02T04:38:36Z")

</div>

Thank you for your answering.  
I tried your example and I learned `@inbounds` is able to make my code faster.

---

<div class="post-metadata">

**Author:** ![usefulhyun](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/usefulhyun/32/1770_2.png) [@usefulhyun](https://discourse.julialang.org/u/usefulhyun)\
**Post date:** [September 2, 2017, 4:45am UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674/8 "2017-09-02T04:45:50Z")

</div>

Thank you for your answer!  
It’s amazing to use the where keyword.

---

<div class="post-metadata">

**Author:** ![usefulhyun](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/usefulhyun/32/1770_2.png) [@usefulhyun](https://discourse.julialang.org/u/usefulhyun)\
**Post date:** [September 2, 2017, 4:47am UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674/9 "2017-09-02T04:47:56Z")

</div>

Thanks to your answer that properly correct the above code.

---

<div class="post-metadata">

**Author:** ![usefulhyun](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/usefulhyun/32/1770_2.png) [@usefulhyun](https://discourse.julialang.org/u/usefulhyun)\
**Post date:** [September 2, 2017, 4:49am UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674/10 "2017-09-02T04:49:18Z")

</div>

Thank you for your answer!

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [September 2, 2017, 6:01am UTC](https://discourse.julialang.org/t/matrix-addition-with-multithreads/5674/11 "2017-09-02T06:01:41Z")

</div>

```julia
julia> Sys.CPU_CORES
4
julia> Threads.nthreads()
4
julia> A = randn(4000,4000); B = randn(4000,4000); C = similar(A);
julia> function matrixadd!(C::AbstractArray{T},A::AbstractArray{T}, B::AbstractArray{T}) where T
                 @boundscheck checkbounds(C,eachindex(B))
                 @boundscheck checkbounds(C,eachindex(A))
                 @inbounds Threads.@threads for i = eachindex(C)
                      C[i] = A[i] + B[i]
                  end
              end
julia> @benchmark matrixadd!($C, $A, $B)
BenchmarkTools.Trial: 
  memory estimate: 48 bytes
  allocs estimate: 1
  --------------
  minimum time: 28.962 ms (0.00% GC)
  median time: 29.081 ms (0.00% GC)
  mean time: 29.289 ms (0.00% GC)
  maximum time: 32.234 ms (0.00% GC)
  --------------
  samples: 171
  evals/sample: 1

julia> @benchmark $C .= $A .+ $B
BenchmarkTools.Trial: 
  memory estimate: 0 bytes
  allocs estimate: 0
  --------------
  minimum time: 30.073 ms (0.00% GC)
  median time: 31.142 ms (0.00% GC)
  mean time: 31.496 ms (0.00% GC)
  maximum time: 53.334 ms (0.00% GC)
  --------------
  samples: 159
  evals/sample: 1

```

My CPU% was almost 4x higher with the former.  
EDIT:  
I do see better performance when doing something more expensive than addition. But, if that is all you do, I would stick with single threaded. No need to cook your processor (possibly slowing other applications down until it cools down) for barely any gain.