# Parallel processing using FLoops

**URL:** <https://discourse.julialang.org/t/parallel-processing-using-floops/80893>\
**Category:** Performance\
**Created:** [May 11, 2022, 3:23pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893 "2022-05-11T15:23:04Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![devanshu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devanshu/32/37104_2.png) [@devanshu](https://discourse.julialang.org/u/devanshu)\
**Post date:** [May 11, 2022, 3:23pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/1 "2022-05-11T15:23:04Z")

</div>

Hi, I am trying to enhance the performance of my Julia code by using all the cores on my computer. For this, I am using the FLoops package. My code is simple: I am generating 100 instances of a random density matrix of size 1000 x 1000 and calculating the mean entanglement entropy. Unfortunately, I cannot get the desired speed up for the below code:

 ![pic1](https://global.discourse-cdn.com/julialang/original/3X/9/c/9c350923e3bce2de936d2dbc8c96e132b4bda01d.png)

I am using the Distributions for generating random matrices, and the QuantumInformation package for calculating the entropy. Is there any way to speed it up, or I have got it wrong?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [May 11, 2022, 3:37pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/2 "2022-05-11T15:37:01Z")

</div>

You likely won’t get significant speedups from paralellism, since the matrix multiplication and matrix logarithm are already multithreaded (by BLAS). That said, I think the algorithm can likely be sped up significantly.

---

<div class="post-metadata">

**Author:** ![devanshu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devanshu/32/37104_2.png) [@devanshu](https://discourse.julialang.org/u/devanshu)\
**Post date:** [May 11, 2022, 3:40pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/3 "2022-05-11T15:40:37Z")

</div>

> [@Oscar\_Smith](#):
>
> That said, I think the algorithm can likely be sped up significantly.

Can you elaborate on this, please?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 11, 2022, 3:48pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/4 "2022-05-11T15:48:18Z")

</div>

is there not a way to do this without materialize a huge matrix and then another one and a third one?

---

<div class="post-metadata">

**Author:** ![devanshu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devanshu/32/37104_2.png) [@devanshu](https://discourse.julialang.org/u/devanshu)\
**Post date:** [May 11, 2022, 3:51pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/5 "2022-05-11T15:51:36Z")

</div>

I am not really sure what you mean, but since what we have is a random matrix, we need to take several instances of it and then take the average.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [May 11, 2022, 3:55pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/6 "2022-05-11T15:55:30Z")

</div>

For one, thing `C*C'` can be sampled directly (it’s a Wishart distribution). I’m not actually convinced that there isn’t a relatively simple scalar distribution you can sample directly that would give the same result.

---

<div class="post-metadata">

**Author:** ![devanshu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devanshu/32/37104_2.png) [@devanshu](https://discourse.julialang.org/u/devanshu)\
**Post date:** [May 11, 2022, 3:58pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/7 "2022-05-11T15:58:46Z")

</div>

Well as far as I know, this is the procedure to generate a Wishart ensemble, because we have to make sure that the matrix is positive semidefinite, and any positive semidefinite matrix has this form `C*C'`.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [May 11, 2022, 4:00pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/8 "2022-05-11T16:00:18Z")

</div>

[https://juliastats.org/Distributions.jl/stable/matrix/#Distributions.Wishart](https://juliastats.org/Distributions.jl/stable/matrix/#Distributions.Wishart)

---

<div class="post-metadata">

**Author:** ![devanshu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devanshu/32/37104_2.png) [@devanshu](https://discourse.julialang.org/u/devanshu)\
**Post date:** [May 11, 2022, 4:08pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/9 "2022-05-11T16:08:23Z")

</div>

But I still think even `Distributions.Wishart` internally would implement `X*X'`. But, I was expecting to parallelize the code like this: I need to generate 100 instances of the matrix, and I have 10 threads on my computer. Is it not possible that the work is parallelly divided among all the threads such that each gets to handle 10 matrices and I get a 10x speed up? I am new to this, so this might sound naive.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 11, 2022, 4:13pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/10 "2022-05-11T16:13:31Z")

</div>

> [@devanshu](#):
>
> Well as far as I know, this is the procedure to generate a Wishart ensemble

> <https://github.com/JuliaStats/Distributions.jl/blob/dd6ae8f4eac304f404b0069540a6c3bb1c667f92/src/matrix/wishart.jl#L206>

you might be correct.

> [@devanshu](#):
>
> all the threads such that each gets to handle 10 matrices and I get a 10x speed up

no, as Oscar said, the `C * C'` is already multi-threaded because BLAS is multi-threaded, so dividing like this shouldn’t give you linear speed up

---

<div class="post-metadata">

**Author:** ![devanshu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devanshu/32/37104_2.png) [@devanshu](https://discourse.julialang.org/u/devanshu)\
**Post date:** [May 11, 2022, 4:15pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/11 "2022-05-11T16:15:47Z")

</div>

Thanks @Oscar_Smith and @jling . I get it now.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 11, 2022, 4:37pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/12 "2022-05-11T16:37:16Z")

</div>

```julia
julia> function g()
           e = 0.0
           C = Matrix{Float64}(undef, 1000, 1000)
           ρ = similar(C)
           for _ = 1:100
               rand!(Normal(), C)
               LinearAlgebra.mul!(ρ, C, C')
               ρ ./= tr(ρ)
               e += vonneumann_entropy(ρ)/log(2)
           end
           e
       end

```

doesn’t seem to make it faster but at least it reduces memory allocation by 70%

---

<div class="post-metadata">

**Author:** ![devanshu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devanshu/32/37104_2.png) [@devanshu](https://discourse.julialang.org/u/devanshu)\
**Post date:** [May 11, 2022, 4:48pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/13 "2022-05-11T16:48:18Z")

</div>

why do I get `rand! not defined` error?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 11, 2022, 4:49pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/14 "2022-05-11T16:49:26Z")

</div>

```julia
using Distributions, QuantumInformation, Random, LinearAlgebra

```

---

<div class="post-metadata">

**Author:** ![devanshu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devanshu/32/37104_2.png) [@devanshu](https://discourse.julialang.org/u/devanshu)\
**Post date:** [May 11, 2022, 4:52pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/15 "2022-05-11T16:52:03Z")

</div>

ok, I see thanks again! Also can you please suggest any article which explains the usage of `!` in functions in julia?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 11, 2022, 4:59pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/16 "2022-05-11T16:59:27Z")

</div>

[https://docs.julialang.org/en/v1/manual/variables/#Stylistic-Conventions](https://docs.julialang.org/en/v1/manual/variables/#Stylistic-Conventions)

it’s nothing special, just the name of the function ending with `!` is a hint that it modifies one of its arguments, that’s it, just a naming convention for functions.

---

<div class="post-metadata">

**Author:** ![devanshu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devanshu/32/37104_2.png) [@devanshu](https://discourse.julialang.org/u/devanshu)\
**Post date:** [May 11, 2022, 5:01pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/17 "2022-05-11T17:01:42Z")

</div>

Oh Alright! Thank you!

---

<div class="post-metadata">

**Author:** ![enweg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/enweg/32/24159_2.png) [@enweg](https://discourse.julialang.org/u/enweg)\
**Post date:** [May 18, 2022, 6:06pm UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/18 "2022-05-18T18:06:32Z")

</div>

I am not sure if this helps, but you can tell BLAS manually how many threads to use. Doing this in conjunction with using @floop resulted in much faster code for me:

```julia
function baseline()
    e = 0
    for _ in 1:100
        C = rand(Normal(), 1000, 1000)
        p = C*C'
        p = p / tr(p)
        e += vonneumann_entropy(p)/log(2)
    end
    e/100
end

function baseline_floop()
    e = 0
    @floop for _ in 1:100
        C = rand(Normal(), 1000, 1000)
        p = C*C'
        p = p / tr(p)
        @reduce e += vonneumann_entropy(p)/log(2)
    end
    e/100
end

```

I obtain the following when setting `BLAS.set_num_threads(1)`

```julia
@time baseline_floop()
  6.060738 seconds (2.23 k allocations: 3.017 GiB, 0.15% gc time)

```

Compared to what I get when BLAS uses 8 threads (the default on my machine)

```julia
@time baseline_floop()
 11.550902 seconds (2.23 k allocations: 3.017 GiB, 0.09% gc time)

```

My Julia was started with 4 threads.

---

<div class="post-metadata">

**Author:** ![devanshu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devanshu/32/37104_2.png) [@devanshu](https://discourse.julialang.org/u/devanshu)\
**Post date:** [May 19, 2022, 3:47am UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/19 "2022-05-19T03:47:49Z")

</div>

Hi, thanks for your response, it does help. But I am just wondering why setting the num of threads for BLAS to 1 is giving such a speed up? Shouldn’t we increase the number of threads to get speed up?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [May 19, 2022, 3:53am UTC](https://discourse.julialang.org/t/parallel-processing-using-floops/80893/20 "2022-05-19T03:53:49Z")

</div>

Blas doesn’t give perfect parallelization so if the floops can parallelize perfectly, that will be more efficient than using BLAS threading.

[Next page](https://discourse.julialang.org/t/parallel-processing-using-floops/80893.md?page=2)