Synchronize the distributed inner loop

General Usage
,
1

Hi, I have the following loop nest where I wait for the inner loop to finish before the outer loop proceeds.

@time begin
for var in 1:10
     @sync @distributed for i in 1:100
         work(i)
     end
end
end

If I have 4 workers, @distributed parallelizes the loop so that every worker gets an even amount of jobs, worker 1 has the interval 1-25, worker 2 → 26-50 etc.
But if one work() finishes early, it has to wait for other workers to finish as well to proceed. I would prefer it to start a new job. I also want to measure the executed time. I tried Distributed.@spawn work() and it works perfectly but I can’t synchronize them back to move together in the outer loop.

Is there a working example you could share?

2

The help for @sync states that

  @sync

  Wait until all lexically-enclosed uses of @async, @spawn, @spawnat and @distributed are complete. All exceptions thrown by enclosed async operations are collected and thrown as a CompositeException.

So what do you mean by saying that you can’t synchronize them?

1 Like
3

Actually yes this works,

for var in 1:10
     @sync for i in 1:100
         @spawn work(i)
     end
end

I thought this wasn’t working because work() function has print() inside and it doesn’t print until all the inner loops are finished (and they all print at the same time). By observing this behavior I wouldn’t say it is any different from the previous one, but checking the results it actually runs faster. Still, I don’t understand why it runs the code but print() function remains hanging.

4

Probably you should flush the standard output: flush(stdout), place it after each print call you wants to force to print. This could slow down stuff a lot, so use it wisely.

2 Likes