Hey, I have a small issue that I haven’t been able to solve for hours. I have a simple pmap,
arr2 = pmap(f, arr).
Later on, after a bunch of computation on arr2, I get an array arr3 of length equal to arr. I need to do another pmap on it,
arr4 = pmap(g, arr3).
However, the key point is that one of the intermediate values computed during the function f(arr[i]) needs to be used by the function g(arr3[i]). Moreover, this intermediate value is very expensive to communicate between processors. Since both of these pmaps operate on the same worker pool, what I would like to do is save this intermediate value locally on the processor when computing f, and have g access this variable.
If it makes things simpler, my worker pool is larger than the size of arr, so each processor does at most one task. Also, I understand that a solution may not be possible using pmap’s, as if a processor is assigned to arr[i] in the first pmap, I need the same processor to be assigned to arr3[i] in the second pmap; I don’t think pmaps offer this level of control. However, at this point, I would be overjoyed with just any solution at all. I’m still getting used to distributed computing in Julia, and would really appreciate help on how to do this 