Reducing RAM usage when solving large set of differential equations

General Usage Performance
1

I am solving a large system of differential equations (O(500 000) equations). Unsurprisingly, I run into RAM issues. I already tried to only save a subset of O(13 000) variables using a SavingCallback

saved_values = SavedValues(Float64, Vector{ComplexF64})
function extract(u, t, integrator)
      return u[save_subset]
end
cb = SavingCallback(extract, saved_values, saveat = 0.1)
tspan = (0.0, 70.0)
prob = ODEProblem(eoms!, σ₀, tspan, p)
sol = solve(
     prob,
     Tsit5(),
     callback = cb,
     dtmax = 0.1,
     maxiters = 100000,
     saveat = 70.0, # only save full solution at the end
)

My last job failed with a MaxRSS of 180 GB. Is this something one would expect? Are there more any ways to reduce the RAM usage further?

1 Like
2

There are some things you can try:

  • make sure you use the inplace, i.e. computing du! does not allocate. If you need scratch space to compute du then you can preallocate it and put it in the parameters.
  • try starting Julia with --heap-size-hint 120G. Experiment a bit the value you put there. It should be smaller than the max available to give Julia the opportunity to run a GC before going OOM.
  • you don’t need to set dt_max to force saving at specific locations. Try leaving it unspecified
3

I will try using —heap-size-hint and not setting dtmax, thanks!

You were right, I do use some allocation when calculating du, but I struggle not to. The variables that I allocate are much smaller than 180GB, and they should be dealt with by the garbage collector, right?

Two more pieces to the puzzle:

  • I use multiple threads when calculating du. Might this have an impact?
  • The computation only fails after several hours. The RAM must thus be growing due to some reason.
4

Standard ODE solvers can have many cache variables in order to achieve the performance. There’s specialized low-storage RK methods for this kind of situation:

1 Like
5

Which one would you recommend for instead of Tsit5? Thanks!

6

Hard to tell without knowing your PDE. But maybe RDPK3Sp35.