# Estimate memory requirements of ODE solver
**URL:** https://discourse.julialang.org/t/estimate-memory-requirements-of-ode-solver/104567
**Category:** General Usage
**Tags:** memory, garbage-collection, ordinarydiffeq
**Created:** [October 4, 2023, 10:25am UTC](https://discourse.julialang.org/t/estimate-memory-requirements-of-ode-solver/104567 "2023-10-04T10:25:54Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Salmon](https://discourse.julialang.org/u/Salmon)
#### Post date: [October 4, 2023, 10:25am UTC](https://discourse.julialang.org/t/estimate-memory-requirements-of-ode-solver/104567/1 "2023-10-04T10:25:54Z")
Hi everyone,
I am solving a **very large** (~10^8) system of ODE’s. I would like to use the `VCABM` solver from ordinaryDiffEq since that seems to be fastest, but I have memory issues that I would like to understand better.
I am aware that the ODE solvers will allocate their cache dependent on the order.
It looks like VCABM needs quite a few caches about 60, looking at the implementation in OrdinaryDiffEq:
> **alg\_cache**
>
> ```julia
> function alg_cache(alg::VCABM, u, rate_prototype, ::Type{uEltypeNoUnits},
> ::Type{uBottomEltypeNoUnits}, ::Type{tTypeNoUnits}, uprev, uprev2, f, t,
> dt, reltol, p, calck,
> ::Val{true}) where {uEltypeNoUnits, uBottomEltypeNoUnits, tTypeNoUnits}
> fsalfirst = zero(rate_prototype)
> k4 = zero(rate_prototype)
> dts = fill(zero(dt), 13)
> c = fill(zero(t), 13, 13)
> g = fill(zero(t), 13)
> ϕ_n = Vector{typeof(rate_prototype)}(undef, 13)
> ϕstar_nm1 = Vector{typeof(rate_prototype)}(undef, 13)
> ϕstar_n = Vector{typeof(rate_prototype)}(undef, 13)
> ϕ_np1 = Vector{typeof(rate_prototype)}(undef, 14)
> for i in 1:13
> ϕ_n[i] = zero(rate_prototype)
> ϕstar_nm1[i] = zero(rate_prototype)
> ϕstar_n[i] = zero(rate_prototype)
> end
> for i in 1:14
> ϕ_np1[i] = zero(rate_prototype)
> end
> β = fill(zero(t), 13)
> order = 1
> max_order = 12
> atmp = similar(u, uEltypeNoUnits)
> recursivefill!(atmp, false)
> tmp = zero(u)
> ξ = zero(dt)
> ξ0 = zero(dt)
> utilde = zero(u)
> utildem2 = zero(u)
> utildem1 = zero(u)
> utildep1 = zero(u)
> atmp = similar(u, uEltypeNoUnits)
> recursivefill!(atmp, false)
> atmpm1 = similar(u, uEltypeNoUnits)
> recursivefill!(atmpm1, false)
> atmpm2 = similar(u, uEltypeNoUnits)
> recursivefill!(atmpm2, false)
> atmpp1 = similar(u, uEltypeNoUnits)
> recursivefill!(atmpp1, false)
> VCABMCache(u, uprev, fsalfirst, k4, ϕstar_nm1, dts, c, g, ϕ_n, ϕ_np1, ϕstar_n, β, order,
> max_order, atmp, tmp, ξ, ξ0, utilde, utildem1, utildem2, utildep1, atmpm1,
> atmpm2, atmpp1, 1)
> end
>
> ```
I confirmed this by printing `sizeofcache = Base.format_bytes(Base.summarysize(cache))` (gives 64 times the size of my state array).
This is quite a lot, but should still fit in my memory (for 100 caches the size of my input array, I should be only at 56 GB memory).
Nonetheless, I find my calculation to consume 2-3 times that amount (~133 GB).
Can anyone give me a pointer if this is expected? How do I find out why so much memory is allocated? A few more points:
- I use `save_everystep = false`. More generally, the memory allocation does not increase after the simulation is started, so it is no memory leak
- I realize that arrays may allocate more memory than they need, but I believe this should not be an issue here, since I allocate my state with zeros(dims…)
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [October 6, 2023, 9:15am UTC](https://discourse.julialang.org/t/estimate-memory-requirements-of-ode-solver/104567/2 "2023-10-06T09:15:02Z")
> [@Salmon](#):
>
> Can anyone give me a pointer if this is expected? How do I find out why so much memory is allocated?
You can look at the allocation profiler.
Someone could take a dive into VCABM though and see if the total number of arrays could be reduced.
---
### Author:  [@Salmon](https://discourse.julialang.org/u/Salmon)
#### Post date: [October 6, 2023, 11:57am UTC](https://discourse.julialang.org/t/estimate-memory-requirements-of-ode-solver/104567/3 "2023-10-06T11:57:50Z")
Thanks, that is a good idea!
I have had a look, though unfortunately it reports the majority of allocated memory to `UnknownType` [which apparently is a known issue](https://github.com/JuliaLang/julia/issues/43688).
Perhaps I can still figure out what the problem is. Am I safe In assuming that the ODE solvers will never allocate memory depending on the parameters, I am passing? My params contain a few more buffer arrays, but i think they should not contribute too much unless there are also ~60 copies of them.
Of course it would be awesome if the overall memory requirement can be reduced, especially given that `VCABM` is great for very big ODE’s.
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [October 6, 2023, 11:59am UTC](https://discourse.julialang.org/t/estimate-memory-requirements-of-ode-solver/104567/4 "2023-10-06T11:59:16Z")
> [@Salmon](#):
>
> Perhaps I can still figure out what the problem is. Am I safe In assuming that the ODE solvers will never allocate memory depending on the parameters, I am passing? My params contain a few more buffer arrays, but i think they should not contribute too much unless there are also ~60 copies of them.
Yes, they don’t copy it.
> [@Salmon](#):
>
> Of course it would be awesome if the overall memory requirement can be reduced, especially given that `VCABM` is great for very big ODE’s.
For very big ODEs, you should use a low-memory RK method. That’s their speciality.
---
### Author:  [@Salmon](https://discourse.julialang.org/u/Salmon)
#### Post date: [October 6, 2023, 12:20pm UTC](https://discourse.julialang.org/t/estimate-memory-requirements-of-ode-solver/104567/5 "2023-10-06T12:20:25Z")
> [@ChrisRackauckas](#):
>
> For very big ODEs, you should use a low-memory RK method. That’s their speciality.
I assume that this should be the choice only if other standard RK solvers (i.e. `Tsit5` or `Vern7` ) consistently do not fit in memory but as long as there is memory left isn’t it better to chose a method that is geared towards speed instead of memory?
So in my case, given that I can use most other RK solvers, I don’t gain any benefit from using the low-memory methods. So for me the choice is more between `Tsit5` (comfortably fits in memory) and `VCABM` (faster, but can require too much memory).
Given the fact that VCABM should on paper only allocate `60~80` times the memory of my ODE state, VCABM is likely the best option if I can figure out where the other ~70GB of memory have gone.
---
### Author:  [@Salmon](https://discourse.julialang.org/u/Salmon)
#### Post date: [October 6, 2023, 1:03pm UTC](https://discourse.julialang.org/t/estimate-memory-requirements-of-ode-solver/104567/6 "2023-10-06T13:03:19Z")
As a debug help, i have added the following to a `FunctionCallingCallback`:
```julia
totalmem = printMem((integrator,State,setup,saved_values))
@info "Memory usage: " totalmem integrator = printMem(integrator) State = printMem(State) parameters = printMem(parameters) savedvalues = printMem(saved_values) integOverState = Base.summarysize(integrator) / Base.summarysize(State)
```
which should give me an overview of all the allocated objects.
```julia
┌ Info: Memory usage:
│ totalmem = "4.598 GiB"
│ integrator = "4.598 GiB"
│ State = "65.552 MiB"
│ parameters = "248.153 MiB"
│ savedvalues = "2.312 MiB"
└ integOverState = 71.821170284851291
```
Nearly all of the memory is taken by the integrator, and it is about what it should take, i.e. ~72x.
So I take that the ODE should fit comfortably in my memory (even if the state is larger, about 600MB for my original problem). Probably my issue is not within `OrdinaryDiffEq` I will investigate whether [its an issue with the GC not being aggressive enough](https://discourse.julialang.org/t/garbage-collection-not-aggressive-enough-on-slurm-cluster/61649/2)
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [October 6, 2023, 2:02pm UTC](https://discourse.julialang.org/t/estimate-memory-requirements-of-ode-solver/104567/7 "2023-10-06T14:02:12Z")
> [@Salmon](#):
>
> Given the fact that VCABM should on paper only allocate `60~80` times the memory of my ODE state, VCABM is likely the best option if I can figure out where the other ~70GB of memory have gone.
Yeah use the best method that fits into memory. I was just mentioning that if you do have very large ODEs, some methods are designed to fit into less memory, with the trade-off of being a generally less efficient method otherwise.
---
### Author:  [@Salmon](https://discourse.julialang.org/u/Salmon)
#### Post date: [October 6, 2023, 2:57pm UTC](https://discourse.julialang.org/t/estimate-memory-requirements-of-ode-solver/104567/8 "2023-10-06T14:57:02Z")
Ok turns out I was being stupid. I forgot that my ODE state is a partition array which has three times the size I was mentioning. So all the memory is precisely accounted for.
Sorry for the bait!