Parallel workflow

General Usage Performance
1

I’m running a bunch of DiffEq solvers in parallel, summarizing the results, and then saving them. Can anyone recommend the best workflow to do this given the current state of Julia’s parallel code, either v1.4 or v1.5? My current solution is (in pseudocode):

const max_batch_size, max_results_in_memory
max_held_batches = floor(Int, max_results_in_memory / max_batch_size)

results_channel = RemoteChannel(() -> Channel{typeof(sample_results)}(max_held_batches))
task = @distributed for batch in batches
    results = [solve(task) for task in batch]
    put!(results_channel, results)
end

n_remaining_batches = length(batches)
while n_remaining_batches > 0
    results = take!(results_channel)
    save(results)
    n_remaining_batches -= 1
end
fetch(task)

I think this works in theory, though I have in the past run into this problem. At the moment, though, I just want to make sure that I’m approaching this the right way. Any input apprecated.

2

Usually for this kind of thing the ensemble interface (Parallel Ensemble Simulations · DifferentialEquations.jl) works quite well, which is essentially doing the boiler plate that you have here.

3 Likes
3

Thank you! I’d been intending to migrate my code to ensembles for a while now. Finally did it yesterday and it works great.