DAE solve asking for an unreasonable amount of memory

Specific Domains Modelling & Simulations
1

I have a system of differential equations (DAE) I am solving using DifferentialEquations.jl . It works well when I have larger discretization (fewer points), but when I use smaller discretization, it asks for a lot of memory from the system (over 18 GB), before crashing because of running out of memory.

My problem size itself is not huge, my input vector is just 0.47 MB and Jacobian size is 9 MB.

length(u) = 61468
Base.summarysize(u) = 491784 # 0.47 MB
Base.summarysize(jac) = 9513672 # 9 MB
Base.summarysize(system_params) = 31934272 # 30MB

Function evaluation and analytical jacobian update are mostly non-allocating.

BenchmarkTools.Trial: 7364 samples with 1 evaluation per sample.
 Range (min … max):  639.375 μs …  1.002 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     674.792 μs              ┊ GC (median):    0.00%
 Time  (mean ± σ):   677.508 μs ± 18.712 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%

           ▂▅▅▄▂▁▃▇█▇▅▃▂▃▄▄▂▂▁                                 ▂
  ▂▄▆▄▆▇█▇▇████████████████████▇▇▇▇▇▆▇▆▅▆▅▆▅▆▆▆▅▅▅▆▃▅▅▄▄▄▄▄▅▃▅ █
  639 μs        Histogram: log(frequency) by time       760 μs <

 Memory estimate: 18.28 KiB, allocs estimate: 108.

julia>  run(bm_jacupdate)
BenchmarkTools.Trial: 1143 samples with 1 evaluation per sample.
 Range (min … max):  3.670 ms …   8.004 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     4.235 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   4.365 ms ± 536.345 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%

     █▇▃▂▃▂▃▂▁▁ ▁▃                                             
  ▂▁▄██████████████▇██▆▃▆▅▅▄▅▅▄▄▃▄▃▃▃▂▃▃▃▂▃▃▂▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂ ▄
  3.67 ms         Histogram: frequency by time        6.29 ms <

 Memory estimate: 22.31 KiB, allocs estimate: 78.

I solve it using FBDF or Rodas5 method. I tried using matrix free krylov method to see if my jacobian was causing memory issue, although with such a small jacobian it shouldn’t be a problem, unless there is some factorization of Jacobian that is being done and stored. But anyways Krylov method didn’t help.

	sol = DE.solve(nlprob,
		ODEBDF.FBDF(linsolve = KrylovJL_GMRES()),
		save_everystep = false,
		saveat = Float64[],        
		; progress = true,
	)

For larger discretization when I’m able to solve it I see that there are about 600 points stored in the solution. I thought maybe saving the solution is causing the memory issue, even thought it should require about 300 MB only. But to eliminate that possiblity I disabled saving solution. I solve it for (0.0, 1e7). Initial dt is pretty small around 1e-6, then nothing happens for most of the time so large time steps there.

nlfunc = DE.ODEFunction(ode!; jac = ode_jac_update!, sparsity = copy(jac), mass_matrix = M)
nlprob = DE.ODEProblem(nlfunc, u, (0.0, 1e7), system_params).

So, I’m trying to find why this ODE solve is asking for so much memory. I think I have eliminated the usual suspects:-

  1. Non allocating function and jacobian update
  2. Krylov, matrix-free method for inverting Jacobian to avoid jacobian factorization.
  3. Not saving solution at any step.
  4. Tried changing ODE solver Rodas5 → FBDF, as it is better for larger systems (1000+ equations).

Any help in figuring out how to fix this memory issue is greatly appreciated. Thanks in advance.

2

Can you open an issue with a runnable example?

3

Sometimes it helps to increase the size of the swap file.

Also, more aggressive garbage collection might help:
For example:

julia --heap-size-hint=16GB

depending on the amount of RAM you have.

4

Will do

1 Like
5

Thanks. I will definitely try it. But I think it should not require so much memory for such a small problem in the first place, especially as no memory is been allocated during the solve as all the allocations are done in the setup and in the cache of DifferentialEquations.jl.

1 Like
6

Thanks will continue there