Parallel simulation : coroutine vs multi-threading

New to Julia
, , ,
3

Yes, I agree with Chris that you’d need to think about limiting parallelism if you want to solve OOM.

There are also probably some ways to reduce memory usage, depending on exactly what you are doing. (So MWE or some sketch of your program would be useful.)

As a toy example, if your program computes the trajectories of all the instances of the dynamical systems and then dump them to files like this

solutions = tcollect(solve(prob) for prob in systems)
for sol in solutions
    save_to_file(sol)
end

it would be much more memory efficient if you just dump the solution to the file right away (so that unused sol can be GC’ed away):

Threads.@threads for prob in systems
    sol = solve(prob)
    save_to_file(sol)
end

As another example, if you have some kind of analysis pipeline after solving the system:

solutions = tcollect(solve(prob) for prob in systems)
statistics = tcollect(compute_some_statistics(sol) for sol in solutions)

it’d be much better (both in terms of the run-time and memory usage) to do the analysis right away:

solutions = tcollect(compute_some_statistics(solve(prob)) for prob in systems)

If you implemented tricks like the above already and still need to optimize your program for memory efficiency, you might want to look at some examples in Concurrency patterns for controlled parallelisms to limit the memory usage.

6 Likes