How should I implement parallel maximum likelihood?

[Michael_Eastwood]

I’m sorry I don’t quite have time to type out a full answer, but if you don’t know all the thetas in advance you can replace the pmap with something like

pool = CachingPool(worker)
while not done
    result = remotecall_fetch(pool, f, theta)
    ...
end

If you do this, the master process will be the one deciding which theta to do next. If you’d rather the worker processes don’t have to communicate with the master, then you should modify f to loop over theta until you’ve found the right values.

You might also find the rest of my posts in the thread you linked in the op helpful:

How to avoid repeated data movement between processes?

Do you mean that nm is different on each iteration? For CachingPool to be useful, I believe you need to be re-using a particular value of nm across all workers. You capture it in a closure and then repeatedly use that closure. If you can construct nm on the worker processes themselves, that’s usually a good call (ie. maybe only one field of nm changes on each iteration so you only have to transfer that field?) Finally, you have a couple of other options as well that are more explicit about th…

How to avoid repeated data movement between processes?

Being more explicit about the data movement is generally my preference as well, but it sounds like you might be able to get away with the CachingPool here. Something like this might work for you: foo(k) = bar(nm, k) # capture nm in a closure pool = CachingPool(workers()) @sync for worker in workers() @async while length(queue) > 0 idx = pop!(queue) results[idx] = remotecall_fetch(foo, pool, ks[idx]) end end