I have a simulation, which needs around 50MB of RAM and 1 second to finish (lets call it
My problem is that I have to run these simulations around 500 times, which suddenly leads to a slowdown (maybe) due to memory issues.
The simulation is not completely allocation free, since I have to do some complicated parsing in between and even call a bit of
eval( Meta.parse(...) ) at unpredictable times. I can clearly see that there is a lot of work for the garbage collector since the memory often drops at regular times.
I use currently a loop like this
# pseudo-code res = Dict( p => ResultType() for p in 1:500 ) Threads.@threads for p in 1:500 res[p] = simulate(p) end
(I will try to create a representative minimal example, but it’s a bit tricky to factor out the relevant parts. If you had a similar situation and know tips, that would already help me.)
- Given that the function
simulateallocates a lot, what is the best strategy for parallelization?