# Parallelization of several independent loops

**URL:** <https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803>\
**Category:** General Usage\
**Tags:** question, parallel\
**Created:** [August 16, 2022, 12:48am UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803 "2022-08-16T00:48:46Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![iangilan](https://avatars.discourse-cdn.com/v4/letter/i/b38774/32.png) [@iangilan](https://discourse.julialang.org/u/iangilan)\
**Post date:** [August 16, 2022, 12:48am UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803/1 "2022-08-16T00:48:46Z")

</div>

Hello all,

How could I parallelize the following independent loops?

```julia
for i=1:N
     x1=A1\b1
end
for i=N+1:2*N
     x2=A2\b2
end
for i=2*N+1:3*N
     x3=A3\b3
end
for i=3*N+1:4*N
     x4=A4\b4
end
```

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [August 16, 2022, 1:36am UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803/2 "2022-08-16T01:36:57Z")

</div>

How does `i` come in to play in the loops?

---

<div class="post-metadata">

**Author:** ![iangilan](https://avatars.discourse-cdn.com/v4/letter/i/b38774/32.png) [@iangilan](https://discourse.julialang.org/u/iangilan)\
**Post date:** [August 16, 2022, 2:18am UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803/4 "2022-08-16T02:18:44Z")

</div>

It’s something like this:

```julia
psi = zeros(4*N,M)
for i=1:N
     x1=A1\b1
     psi[i,:] = x1'
end
for i=N+1:2*N
     x2=A2\b2
     psi[i,:] = x2'
end
for i=2*N+1:3*N
     x3=A3\b3
     psi[i,:] = x3'
end
for i=3*N+1:4*N
     x4=A4\b4
     psi[i,:] = x4'
end

```

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [August 16, 2022, 2:30am UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803/5 "2022-08-16T02:30:27Z")

</div>

I suppose something like this might work:

```julia
    while true
        ...
        rref = @spawnat p <kick off some work on process p> 
        push!(rrefs, rref) 
        if <all work parceled out>
            break
        end
    end
    for r in rrefs
        results = fetch(r)
        <do something with the results>
    end

```

Using a certain number of workers, in your case 4.

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [August 16, 2022, 2:48pm UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803/6 "2022-08-16T14:48:07Z")

</div>

I’m not very experienced in multithreading, so there is probably a good reason to do the channel solution that @PetrKryslUCSD suggested. Could you explain why not just use `@spawn` like below?

```julia
psi = zeros(4*N,M)
@sync
Threads.@spawn for i=1:N
     x1=A1\b1
     psi[i,:] = x1'
end
Threads.@spawn for i=N+1:2*N
     x2=A2\b2
     psi[i,:] = x2'
end
Threads.@spawn for i=2*N+1:3*N
     x3=A3\b3
     psi[i,:] = x3'
end
Threads.@spawn for i=3*N+1:4*N
     x4=A4\b4
     psi[i,:] = x4'
end
end

```

---

<div class="post-metadata">

**Author:** ![iangilan](https://avatars.discourse-cdn.com/v4/letter/i/b38774/32.png) [@iangilan](https://discourse.julialang.org/u/iangilan)\
**Post date:** [August 18, 2022, 12:51am UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803/7 "2022-08-18T00:51:34Z")

</div>

I don’t know the answer but I was trying to use the code similar to yours. It never worked out for me. Does it work for you? [mrufsvold](https://discourse.julialang.org/u/mrufsvold)

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [August 18, 2022, 3:02am UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803/8 "2022-08-18T03:02:20Z")

</div>

I don’t have definitions for the As and Bs so I couldn’t text the code, but I was able to kick off several loops in different threads using the strategy!

Here is a reproducible version

```julia
N = M = 10
psi = zeros(4*N,M)
A1 = ones(M)
A2 = A1 .+ A1
A3 = A2 + A1
A4 = A3 + A1

@sync begin 
Threads.@spawn begin
    println("Running a loop on thread $(Threads.threadid())")
    for i=1:N
         psi[i,:] = A1
    end
end
Threads.@spawn begin
    println("Running a loop on thread $(Threads.threadid())")
    for i=N+1:2*N
         psi[i,:] = A2
    end
end
Threads.@spawn begin
    println("Running a loop on thread $(Threads.threadid())")
    for i=2*N+1:3*N
         psi[i,:] = A3
    end
end
Threads.@spawn begin
    println("Running a loop on thread $(Threads.threadid())")
    for i=3*N+1:4*N
         psi[i,:] = A4
    end
end
end
psi

```

Maybe worth noting that you need to start Julia with `julia --threads auto` or replace auto with an integer for the number of threads you want to use. Otherwise, Julia starts with one thread by default.

---

<div class="post-metadata">

**Author:** ![iangilan](https://avatars.discourse-cdn.com/v4/letter/i/b38774/32.png) [@iangilan](https://discourse.julialang.org/u/iangilan)\
**Post date:** [August 18, 2022, 3:11am UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803/9 "2022-08-18T03:11:58Z")

</div>

I mean, was it actually about 4 times faster than without multi-threading?

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [August 18, 2022, 3:14am UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803/10 "2022-08-18T03:14:51Z")

</div>

Well, if you actually have several independent workers, thn the work of each processor can be run on multiple threads. Double parallelization…

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [August 18, 2022, 3:16am UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803/11 "2022-08-18T03:16:11Z")

</div>

I don’t know. I didn’t benchmark because your question was just about running in parallel, not performance. There is overhead for spawning a new task, so it usually isn’t worth it for small operations. Almost certainly not for my 10x100 array. You will have to benchmark your specific use case to find out if it is worth it.

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [August 18, 2022, 3:16am UTC](https://discourse.julialang.org/t/parallelization-of-several-independent-loops/85803/12 "2022-08-18T03:16:43Z")

</div>

Thanks!