What is the best practice for parallel dictionary comprehensions?

Is there a particular reason in your application why you need dictionaries already during the parallel stage?

Not here :sweat_smile:. Indeed for this project I’m using pmap and rolling everything up at the end.

But I wanted to understand spawn and sync better so I can balance the load better in deeper loops, and work out how to write this reduction “properly”. All I can rely on right now is pmap and I’m looking to up my || game, since it’s leaving a lot of CPU on the table because of imbalances.

Update: re 4.) I’m using vs code parameter to set-threads for running a script block by block. Im trying to check if concurrency of pmap is limited by nthreads or not.
I found the pattern from this post helpful

np = addprocs(4)
…
rmprocs(np)

for controlling and freeing ram at the end of a distributed thing.

1 Like