# Reduce memory usage for ensemble problems

**URL:** <https://discourse.julialang.org/t/reduce-memory-usage-for-ensemble-problems/90591>\
**Category:** Modelling & Simulations\
**Tags:** diffeq\
**Created:** [November 21, 2022, 2:52pm UTC](https://discourse.julialang.org/t/reduce-memory-usage-for-ensemble-problems/90591 "2022-11-21T14:52:22Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![henry2004y](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henry2004y/32/9284_2.png) [@henry2004y](https://discourse.julialang.org/u/henry2004y)\
**Post date:** [November 21, 2022, 2:52pm UTC](https://discourse.julialang.org/t/reduce-memory-usage-for-ensemble-problems/90591/1 "2022-11-21T14:52:22Z")

</div>

Hi,

I have a test particle tracer which is built on top of OrdinaryDiffEq.jl. This model can have 3D numerical fields as input parameters and solve ODEs for each set of initial conditions.

However, when testing on the multiple particles setup, i.e. an [ensemble problem](https://diffeq.sciml.ai/stable/features/ensemble/), I encountered a memory bottleneck where the field arrays (wrapped by Interpolations.jl [here](https://github.com/henry2004y/TestParticle.jl/blob/77b05b28763d9b30195483bd434da01a81a197e7/src/TestParticle.jl#L58-L69)) are replicated for each solution in the ensemble problem. For example, given a test EM field of ~ 80MB, each solution is also ~ 80MB in memory. More details can be found [here](https://github.com/henry2004y/TestParticle.jl/issues/55).

In my case, the field parameter would be the same for all particles, but each test particle may have different initial condition. Is there a way to reuse the same interpolated field information for the ensemble problem to save memory?

---

<div class="post-metadata">

**Author:** ![henry2004y](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henry2004y/32/9284_2.png) [@henry2004y](https://discourse.julialang.org/u/henry2004y)\
**Post date:** [October 29, 2023, 2:05am UTC](https://discourse.julialang.org/t/reduce-memory-usage-for-ensemble-problems/90591/2 "2023-10-29T02:05:18Z")

</div>

So to put the original question shorter and more precise: is it possible to share parameters for an ensemble problem?

For example,

```julia
using DifferentialEquations

function lorenz!(du,u,p,t)
    σ,ρ,β = p
    du[1] = σ*(u[2]-u[1])
    du[2] = u[1]*(ρ-u[3]) - u[2]
    du[3] = u[1]*u[2] - β*u[3]
end

"Initial state perturbation for EnsembleProblem."
function prob_func(prob, i, repeat)
   prob.u0[1] += i
   prob
end

u0 = [1.0,0.0,0.0]

p_tuple = (10,28,8/3)
#p_vector = Any[10,28, 8/3]

tspan = (0.0,100.0)
prob = ODEProblem(lorenz!,u0,tspan,p_tuple)
#sol = solve(prob, Tsit5())
ensemble_prob = EnsembleProblem(prob; prob_func)

trajectories = 1

@time sols = solve(ensemble_prob, Tsit5(), EnsembleThreads(); trajectories,
   save_on=false, save_start=false, save_end=false,);

```

In this case, the parameter `p = p_tuple = (10,28,8/3)` is not shared: each trajectory will generate a copy. This may not be a problem for a parameter with 3 scalars, but in my original problem, two of the parameters are arrays, and I cannot afford to copy field arrays for each trajectory.

I guess if I use a vector type for the parameter, `p = Any[10, 28, 8/3]`, then probably the parameters are referring to the same memory? Even this is true, there is another warning from DifferentialEquations

```julia
┌ Warning: Using arrays or dicts to store parameters of different types can hurt performance.
│ Consider using tuples instead.
└ @ SciMLBase C:\Users\hyzho\.julia\packages\SciMLBase\ZUwcW\src\performance_warnings.jl:32

```

I added `Any` on purpose, because in my original problem, param is of type

```julia
Tuple{Float64, Float64, TestParticle.Field{false, TestParticle.var"#get_field#3"{Interpolations.ScaledInterpolation{Float64, 3, Interpolations.FilledExtrapolation{Float64, 3, Interpolations.BSplineInterpolation{Float64, 3, Array{Float64, 3}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Float64}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}}, Interpolations.ScaledInterpolation{Float64, 3, Interpolations.FilledExtrapolation{Float64, 3, Interpolations.BSplineInterpolation{Float64, 3, Array{Float64, 3}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Float64}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}}, Interpolations.ScaledInterpolation{Float64, 3, Interpolations.FilledExtrapolation{Float64, 3, Interpolations.BSplineInterpolation{Float64, 3, Array{Float64, 3}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Float64}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}}}}, TestParticle.Field{false, TestParticle.var"#get_field#3"{Interpolations.ScaledInterpolation{Float64, 3, Interpolations.FilledExtrapolation{Float64, 3, Interpolations.BSplineInterpolation{Float64, 3, Array{Float64, 3}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Float64}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}}, Interpolations.ScaledInterpolation{Float64, 3, Interpolations.FilledExtrapolation{Float64, 3, Interpolations.BSplineInterpolation{Float64, 3, Array{Float64, 3}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Float64}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}}, Interpolations.ScaledInterpolation{Float64, 3, Interpolations.FilledExtrapolation{Float64, 3, Interpolations.BSplineInterpolation{Float64, 3, Array{Float64, 3}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Float64}, Interpolations.BSpline{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}}}}}

```

and when I tried to convert that into an vector via `collect(param)`, Julia gave up in type inference.

I got stuck here.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [October 29, 2023, 5:39pm UTC](https://discourse.julialang.org/t/reduce-memory-usage-for-ensemble-problems/90591/3 "2023-10-29T17:39:27Z")

</div>

> [@henry2004y](#):
>
> ```julia
> ensemble_prob = EnsembleProblem(prob; prob_func)
> 
> ```

`ensemble_prob = EnsembleProblem(prob; prob_func, safetycopy = false)`

> [@henry2004y](#):
>
> I guess if I use a vector type for the parameter, `p = Any[10, 28, 8/3]`, then probably the parameters are referring to the same memory?

Use `p = [10, 28, 8/3]` instead.

---

<div class="post-metadata">

**Author:** ![henry2004y](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henry2004y/32/9284_2.png) [@henry2004y](https://discourse.julialang.org/u/henry2004y)\
**Post date:** [February 21, 2024, 11:34pm UTC](https://discourse.julialang.org/t/reduce-memory-usage-for-ensemble-problems/90591/4 "2024-02-21T23:34:25Z")

</div>

I now have an additional question on the `safetycopy` keyword argument. The current version of the code let `safetycopy` control whether or not `prob` is deep copied or not, including `u0` and `param`. Is it possible to have a fine-grain control such that `u0` is deep copied whereas `param` is referenced? The reason I wish to have this is that in my specific problem, `param` is memory consuming but `u0` is cheap. To avoid race conditions when running ensemble problems, I need different `u0` for different threads but shared `param`.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [February 22, 2024, 4:24am UTC](https://discourse.julialang.org/t/reduce-memory-usage-for-ensemble-problems/90591/5 "2024-02-22T04:24:32Z")

</div>

You can always just set `safetycopy=false` and do the copying yourself.