The function mapreduce(f, op, X; dims = dim)
can be used, but it seems that the parallel analog pmapreduce
from the package ParallelUtilities
does not support the dims
argument.
How would one use simple parallel functions to perform the same operation and still get a big speed up?
Especially when X
is of dims (very very big, big, very big)
(say (10^5, 10^2, 10^3)
) and the reduction happens on the first (largest) dimension. f
are about 400ns
and op
are typically +
.
Example of what I am doing
N = 10^5
n = 100
M = 1000
X = fill(rand(), N, M, n)
x = mapreduce(r -> sin(r, a, b), +, X, dims = 1)
# * What I want * #
using ParallelUtilities
x = pmapreduce(r -> sin(r, a, b), +, X, dims = 1)