# Synchronize the distributed inner loop

**URL:** <https://discourse.julialang.org/t/synchronize-the-distributed-inner-loop/49159>\
**Category:** General Usage\
**Tags:** parallel, distributed\
**Created:** [October 28, 2020, 6:04am UTC](https://discourse.julialang.org/t/synchronize-the-distributed-inner-loop/49159 "2020-10-28T06:04:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![libensvivit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/libensvivit/32/19006_2.png) [@libensvivit](https://discourse.julialang.org/u/libensvivit)\
**Post date:** [October 28, 2020, 6:04am UTC](https://discourse.julialang.org/t/synchronize-the-distributed-inner-loop/49159/1 "2020-10-28T06:04:35Z")

</div>

Hi, I have the following loop nest where I wait for the inner loop to finish before the outer loop proceeds.

```julia
@time begin
for var in 1:10
     @sync @distributed for i in 1:100
         work(i)
     end
end
end

```

If I have 4 workers, @distributed parallelizes the loop so that every worker gets an even amount of jobs, worker 1 has the interval 1-25, worker 2 → 26-50 etc.  
But if one work() finishes early, it has to wait for other workers to finish as well to proceed. I would prefer it to start a new job. I also want to measure the executed time. I tried `Distributed.@spawn work()` and it works perfectly but I can’t synchronize them back to move together in the outer loop.

Is there a working example you could share?

---

<div class="post-metadata">

**Author:** ![mikkoku](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikkoku/32/16274_2.png) [@mikkoku](https://discourse.julialang.org/u/mikkoku)\
**Post date:** [October 28, 2020, 5:50pm UTC](https://discourse.julialang.org/t/synchronize-the-distributed-inner-loop/49159/2 "2020-10-28T17:50:20Z")

</div>

The help for `@sync` states that

```julia
  @sync

  Wait until all lexically-enclosed uses of @async, @spawn, @spawnat and @distributed are complete. All exceptions thrown by enclosed async operations are collected and thrown as a CompositeException.

```

So what do you mean by saying that you can’t synchronize them?

---

<div class="post-metadata">

**Author:** ![libensvivit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/libensvivit/32/19006_2.png) [@libensvivit](https://discourse.julialang.org/u/libensvivit)\
**Post date:** [October 28, 2020, 7:20pm UTC](https://discourse.julialang.org/t/synchronize-the-distributed-inner-loop/49159/3 "2020-10-28T19:20:17Z")

</div>

Actually yes this works,

```julia
for var in 1:10
     @sync for i in 1:100
         @spawn work(i)
     end
end

```

I thought this wasn’t working because `work()` function has `print()` inside and it doesn’t print until all the inner loops are finished (and they all print at the same time). By observing this behavior I wouldn’t say it is any different from the previous one, but checking the results it actually runs faster. Still, I don’t understand why it runs the code but `print()` function remains hanging.

---

<div class="post-metadata">

**Author:** ![josePereiro](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josepereiro/32/17322_2.png) [@josePereiro](https://discourse.julialang.org/u/josePereiro)\
**Post date:** [November 2, 2020, 3:01am UTC](https://discourse.julialang.org/t/synchronize-the-distributed-inner-loop/49159/4 "2020-11-02T03:01:29Z")

</div>

Probably you should `flush` the standard output: `flush(stdout)`, place it after each print call you wants to force to print. This could slow down stuff a lot, so use it wisely.