I have several large objects that need to be passed to worker processes. Currently, I am just using a simple
@parallel for loop without any reducer. Since the objects are quite large, it takes 5 seconds before a worker process actually starts doing the work.
However, it appears that each worker process is taking this hit sequentially. So, the first worker takes 5 seconds, then the second worker takes another 5 seconds, so on and so forth. While I have many worker processes (24), the later one progressively take a longer time to even start doing work.
Is there any way to avoid that? Perhaps using something other than
@parallel? I’ll work out a MWE if needed. Thanks