I have an EnsembleProblem defined that calculates outputs for a range of different start times. I need to repeat the the whole EnsembleProblem many times for different parameter values and save the output of each EnsembleProblem to file.
Is it okay to combine pmap with an EnsembleProblem to do all of this in parallel?
Something like
function solve_ep(p)
prob = create_model(p)
probs = EnsembleProblem(prob, prob_func=prob_func)
solve(probs, EnsembleThreads(), trajectories=N)
end
out = pmap(i -> solve_ip(p[:, i]), 1:1000)
Will this kind of “hierarchical” parallelism work as expected / work well? E.g. if I have multiple nodes and cores, will it distribute each EnsembleProblem across the nodes and cores and then distribute the components of EnsembleProblem across different threads?
https://docs.sciml.ai/latest/features/ensemble/#EnsembleAlgorithms-1
EnsembleSplitThreads() - This uses threading on each process, splitting the problem into nprocs() even parts. This is for solving many quick trajectories on a multi-node machine. It’s recommended you have one process on each node.
That might be what you’re looking for.
1 Like
Thank you. Would it be better to try to implement the whole thing as one EnsembleProblem? If I do that but the output I generate is too large to store in memory, is there a way to offload some of the data to disk?
I’ve tried this but using EnsembleSplitThreads appears to do N trajectories on each process instead of N trajectories in total. I am accessing an array using the trajectory index so this is causing an out of bounds error and it’s also doing nprocs() times more work than needed.
Oh, that’s not intended, and I see that happening here:
https://github.com/SciML/DiffEqBase.jl/blob/master/src/ensemble/basic_ensemble_solve.jl#L187
That I needs to split up by the process. Would you mind opening us up an issue with an example? It would be good to get this tested. Our Travis only was testing with one process 