# Unexpected memory allocation inside ODE function with EnsembleProblem

**URL:** <https://discourse.julialang.org/t/unexpected-memory-allocation-inside-ode-function-with-ensembleproblem/38492>\
**Category:** Modelling & Simulations\
**Created:** [April 30, 2020, 4:45pm UTC](https://discourse.julialang.org/t/unexpected-memory-allocation-inside-ode-function-with-ensembleproblem/38492 "2020-04-30T16:45:47Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![MFairley](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mfairley/32/8599_2.png) [@MFairley](https://discourse.julialang.org/u/MFairley)\
**Post date:** [April 30, 2020, 4:45pm UTC](https://discourse.julialang.org/t/unexpected-memory-allocation-inside-ode-function-with-ensembleproblem/38492/1 "2020-04-30T16:45:47Z")

</div>

I am unsure if this is an issue or I just don’t understand how memory allocation works with ensemble problems.

I start Julia with `--track-allocation=user` and run the following small script

```julia
using DifferentialEquations
using Profile

function lorenz!(du,u,p,t)
    du[1] = 10.0*(u[2]-u[1])
    du[2] = u[1]*(28.0-u[3]) - u[2]
    du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0, 100.0)
prob = ODEProblem(lorenz!, u0, tspan)

function prob_func(prob,i,repeat)
    remake(prob, tspan=(rand(), 100.0))
end

ensemble_prob = EnsembleProblem(prob,prob_func=prob_func)

sim = solve(ensemble_prob, Tsit5(), EnsembleThreads(), trajectories=100)
Profile.clear_malloc_data()
sim = solve(ensemble_prob, Tsit5(), EnsembleThreads(), trajectories=100)

```

The function `Lorenz!(du, u, p, t)` should not allocate any memory.

However, looking at the output of the .mem file, I get the following

```julia
        - using DifferentialEquations
        - using Profile
        - 
        - function lorenz!(du,u,p,t)
  1284544 du[1] = 10.0*(u[2]-u[1])
  1220016 du[2] = u[1]*(28.0-u[3]) - u[2]
  1456704 du[3] = u[1]*u[2] - (8/3)*u[3]
        - end
        - 
        - u0 = [1.0;0.0;0.0]
        - tspan = (0.0, 100.0)
        - prob = ODEProblem(lorenz!, u0, tspan)
        - 
        - function prob_func(prob,i,repeat)
        - remake(prob, tspan=(rand(), 100.0))
        - end
        - 
        - ensemble_prob = EnsembleProblem(prob,prob_func=prob_func)
        - 
        - sim = solve(ensemble_prob, Tsit5(), EnsembleThreads(), trajectories=100)
        - Profile.clear_malloc_data()
        - sim = solve(ensemble_prob, Tsit5(), EnsembleThreads(), trajectories=100)

```

There are memory allocations inside the lorenz! function. If I change the `ensemblealg` to `EnsembleSerial` then I don’t observe any memory allocation in the .mem file.

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [April 30, 2020, 5:16pm UTC](https://discourse.julialang.org/t/unexpected-memory-allocation-inside-ode-function-with-ensembleproblem/38492/2 "2020-04-30T17:16:40Z")

</div>

Are you using Julia 1.4.1 or something else? Because when i run your script with track-allocation=user I get:

```julia
        - using DifferentialEquations
        - using Profile
        - 
        - function lorenz!(du,u,p,t)
        0 du[1] = 10.0*(u[2]-u[1])
        0 du[2] = u[1]*(28.0-u[3]) - u[2]
        0 du[3] = u[1]*u[2] - (8/3)*u[3]
        - end
        - 
        - u0 = [1.0;0.0;0.0]
        - tspan = (0.0, 100.0)
        - prob = ODEProblem(lorenz!, u0, tspan)
        - 
        - function prob_func(prob,i,repeat)
        - remake(prob, tspan=(rand(), 100.0))
        - end
        - 
        - ensemble_prob = EnsembleProblem(prob,prob_func=prob_func)
        - 
        - sim = solve(ensemble_prob, Tsit5(), EnsembleThreads(), trajectories=100)
        - Profile.clear_malloc_data()
        - sim = solve(ensemble_prob, Tsit5(), EnsembleThreads(), trajectories=100)
        - 

```

---

<div class="post-metadata">

**Author:** ![MFairley](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mfairley/32/8599_2.png) [@MFairley](https://discourse.julialang.org/u/MFairley)\
**Post date:** [April 30, 2020, 5:31pm UTC](https://discourse.julialang.org/t/unexpected-memory-allocation-inside-ode-function-with-ensembleproblem/38492/3 "2020-04-30T17:31:48Z")

</div>

Yes, I’m using Julia 1.4.1 on macOS

---

<div class="post-metadata">

**Author:** ![MFairley](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mfairley/32/8599_2.png) [@MFairley](https://discourse.julialang.org/u/MFairley)\
**Post date:** [April 30, 2020, 5:33pm UTC](https://discourse.julialang.org/t/unexpected-memory-allocation-inside-ode-function-with-ensembleproblem/38492/4 "2020-04-30T17:33:33Z")

</div>

Also, @pixel27 how many threads do you have? `Threads.nthreads()`

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [April 30, 2020, 5:37pm UTC](https://discourse.julialang.org/t/unexpected-memory-allocation-inside-ode-function-with-ensembleproblem/38492/5 "2020-04-30T17:37:43Z")

</div>

Ahh, that did it. I was only running with 1 thread. When I ran it with 4 i get the memory allocation count.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 30, 2020, 6:25pm UTC](https://discourse.julialang.org/t/unexpected-memory-allocation-inside-ode-function-with-ensembleproblem/38492/6 "2020-04-30T18:25:12Z")

</div>

That’s definitely not good. I might need to call in the big guns here because that’s somewhat mysterious.

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [April 30, 2020, 7:07pm UTC](https://discourse.julialang.org/t/unexpected-memory-allocation-inside-ode-function-with-ensembleproblem/38492/7 "2020-04-30T19:07:06Z")

</div>

I’m at a loss…even looking at the generated code doesn’t really help. My fall back position would be that the memory tracker doesn’t do well with multi-threaded and that is causing some sort of false positive.