I have N datasets (\text{data}^{}_1,..., \text{data}^{}_{10}). For each dataset I need to solve \hat{\theta}^{}_{d}=\arg\min\limits_\theta h(\theta;\text{data}^{}_{d}). I am using pmap
to run the optimizations in parallel for each dataset in datas
which is a struct which stores the ten datasets. The code looks something like this:
function compute_h(d,θ)
Threads.@threads for i in eachindex(d)
#do something to construct h(d,θ)
end
#return h(d,θ)
end
function find_argmin(d)
# argmin = optimize(compute_h(d,θ), θ_guess)
# return argmin
end
pool1 = [i for i=1:10]
pmap(d -> find_argmin(h(d)), WorkerPool(pool1), datas)
The above code runs without any problems. However, I get the error ERROR: No active worker available in pool
when I try to parallelize compute_h
over multiple processors instead of threading it (see below).
function compute_h(d,θ)
@distributed for i in eachindex(d)
#do something to construct h(d,θ)
end
#return h(d,θ)
end
My question: Is there anyway to nest a @distributed for
loop within a pmap
? My guess is that at a minimum I will need to specify that the @distributed for
loop uses workers not already used for the pmap
. However I have no idea how to specify a WorkerPool
for the @distributed for
loop.