You can give the “elastic” parallel map in Schedulers.jl a try. I usually use it with AzManagers.jl. But, it should work with any cluster manager. There are some caveats with respect to what type of code is auto-loaded onto the new workers that you need to be careful of.
3 Likes