I have a utility in a package to run a batch of things. I’d like for that batch function to be able to take in an argument like
n_workers, then stand those workers up, distribute the work, and then remove those new workers. Here’s an example of how things work today, without remote workers:
module MyUtility # Pretend this is a stand-alone package. using Random using Distributed # The expensive function we want to run many times: function foo(n) return randn(n) end # A convenient function to run foo for us a whole bunch of times, reducing the result with # whatever the user provides. function batch(g, reduce) return pmap(g) do n reduce(foo(n)) end end end # Here's my script to use the above package. import .MyUtility # We want to run MyUtility.foo a bunch of times, reducing the results of each run with this: function bar(draws) return sqrt(sum(draws.^2)/length(draws)) end results = MyUtility.batch(1:100, bar) @show results
This clearly works fine. Further, I’m able to add boilerplate to this to make it work with distributed workers. However, I’d like to reduce the boilerplate that users of the package will need in order for
batch to complete its job. I would like for something like this to work:
function batch(g, reduce, n_workers) # Get help. workers = nothing if n_workers > 0 workers = addprocs(n_workers) end # Now do the thing. results = pmap(g) do n reduce(foo(n)) end # Thank our help and release them. if n_workers > 0 rmprocs(workers) end return results end
However, this clearly won’t work. Those workers don’t know anything about
bar or anything used by
bar. Further, if
batch is to stand up the workers, then we can’t wrap
@everywhere (or rather, we can, but it won’t make it to the right workers). So I’m confused about the right way to do this.
Here’s the “lots of boilerplate example” way to do it, where all of the parallel stuff is pushed on the end user rather than being a convenient part of the utility:
using Distributed addprocs(5, exeflags="--project=$(Base.active_project())") # Make sure workers inherit our project! # Bring in packages. @everywhere begin import .MyUtility end # This needs to be a separate @everywhere block for some reason. @everywhere begin function bar(draws) return sqrt(sum(draws.^2)/length(draws)) end end results = MyUtility.batch(1:100, bar)
Clearly, moving from the “regular” version to a “parallel” version implies a very big reorganization of the user’s code, when all they really intend is to “distribute what I’m doing across more cores”. In fact, the one place where they intend to have a change (“run batch on all my cores”) is the only line that doesn’t change.
Any tips for how to do this? I’ve tried to read all of the related threads but haven’t found a good way.