# Parallel workflow

**URL:** https://discourse.julialang.org/t/parallel-workflow/42259
**Category:** Performance
**Tags:** parallel
**Created:** [June 29, 2020, 4:16pm UTC](https://discourse.julialang.org/t/parallel-workflow/42259 "2020-06-29T16:16:58Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![grahamas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/grahamas/32/3401_2.png) [@grahamas](https://discourse.julialang.org/u/grahamas)
#### Post date: [June 29, 2020, 4:16pm UTC](https://discourse.julialang.org/t/parallel-workflow/42259/1 "2020-06-29T16:16:58Z")

</div>

I’m running a bunch of DiffEq solvers in parallel, summarizing the results, and then saving them. Can anyone recommend the best workflow to do this given the current state of Julia’s parallel code, either v1.4 or v1.5? My current solution is (in pseudocode):

```julia
const max_batch_size, max_results_in_memory
max_held_batches = floor(Int, max_results_in_memory / max_batch_size)

results_channel = RemoteChannel(() -> Channel{typeof(sample_results)}(max_held_batches))
task = @distributed for batch in batches
    results = [solve(task) for task in batch]
    put!(results_channel, results)
end

n_remaining_batches = length(batches)
while n_remaining_batches > 0
    results = take!(results_channel)
    save(results)
    n_remaining_batches -= 1
end
fetch(task)

```

I think this works in theory, though I have in the past run into [this problem](https://discourse.julialang.org/t/debugging-remotechannel-memory-error-during-distributed-computation/32062). At the moment, though, I just want to make sure that I’m approaching this the right way. Any input apprecated.

---

<div class="post-metadata">

### Author: ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [June 29, 2020, 5:01pm UTC](https://discourse.julialang.org/t/parallel-workflow/42259/2 "2020-06-29T17:01:11Z")

</div>

Usually for this kind of thing the ensemble interface ([Parallel Ensemble Simulations · DifferentialEquations.jl](https://diffeq.sciml.ai/latest/features/ensemble/)) works quite well, which is essentially doing the boiler plate that you have here.

---

<div class="post-metadata">

### Author: ![grahamas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/grahamas/32/3401_2.png) [@grahamas](https://discourse.julialang.org/u/grahamas)
#### Post date: [July 1, 2020, 3:05pm UTC](https://discourse.julialang.org/t/parallel-workflow/42259/3 "2020-07-01T15:05:38Z")

</div>

Thank you! I’d been intending to migrate my code to ensembles for a while now. Finally did it yesterday and it works great.
