# SavingCallback when using Ensemble Simulations
**URL:** https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483
**Category:** General Usage
**Tags:** differentialequation
**Created:** [October 9, 2022, 3:00pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483 "2022-10-09T15:00:54Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@albertomercurio](https://discourse.julialang.org/u/albertomercurio)
#### Post date: [October 9, 2022, 3:00pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/1 "2022-10-09T15:00:54Z")
Hello,
I’m able to use SavingCallback with single simulations, for example
```julia
using LinearAlgebra
using SparseArrays
using DifferentialEquations
H0 = sprand(ComplexF64, 10, 10, 0.5)
H0 *= H0'
ψ0 = normalize(rand(ComplexF64, 10))
t_l = LinRange(0, 1, 100)
tspan = (t_l[1], t_l[end])
saved_values = SavedValues(Float64, Float64)
save_func = (u, t, integrator) -> norm(u)
cb1 = SavingCallback(save_func, saved_values, saveat = t_l)
dudt!(du,u,p,t) = mul!(du, -1im * H0, u)
prob = ODEProblem(dudt!, ψ0, tspan, callback = cb1)
sol = solve(prob, Tsit5())
saved_values.saveval
100-element Vector{Float64}
```
But now I need to use the SavingCallback with Ensemble Simulation, for example
```julia
function prob_func(prob,i,repeat)
remake(prob,u0=normalize(rand(ComplexF64, 10)))
end
prob = ODEProblem(dudt!, ψ0, tspan, callback = cb1)
ensemble_prob = EnsembleProblem(prob, prob_func=prob_func)
sol = solve(ensemble_prob, Tsit5(), EnsembleSerial(), trajectories=10)
saved_values.saveval
```
But `saved_values.saveval` remains empty. I noticed that if I move the argument `callback = cb1` inside the `solve` function, `saved_values.saveval` have the values of the last (I think) simulation.
How can I store the datas of all the simulations in an array of size 100x10 (for this specific case)?
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [October 9, 2022, 3:26pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/2 "2022-10-09T15:26:34Z")
> [@albertomercurio](#):
>
> But `saved_values.saveval` remains empty. I noticed that if I move the argument `callback = cb1` inside the `solve` function, `saved_values.saveval` have the values of the last (I think) simulation.
>
> How can I store the datas of all the simulations in an array of size 100x10 (for this specific case)?
You need to control the deepcopy yourself. I would recommend directly setting the cache in the prob\_func.
---
### Author:  [@albertomercurio](https://discourse.julialang.org/u/albertomercurio)
#### Post date: [October 9, 2022, 3:32pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/3 "2022-10-09T15:32:05Z")
Ok, so SavingCallback does not support Ensembles yes, and I need to do it by myself, right?
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [October 9, 2022, 5:52pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/4 "2022-10-09T17:52:18Z")
It’s moreso that what you define as `saveval` is just used as a reference, so you should have one reference per solve if you want it to have different saves. It’s a very general tool of that form.
---
### Author:  [@homocomputeris](https://discourse.julialang.org/u/homocomputeris)
#### Post date: [August 19, 2024, 6:19pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/5 "2024-08-19T18:19:34Z")
Do you have an example how to do this?
E.g. for trajectory `i` save the solution squared `u.^2` for each `t` to `saved_values[i]`.
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [August 19, 2024, 10:12pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/6 "2024-08-19T22:12:11Z")
```julia
saved_values = [...]
function prob_func(prob,i, ...)
cb = SavingCallback(..., saved_values[i], ...)
prob = remake(prob, callback = cb)
end
```
---
### Author:  [@homocomputeris](https://discourse.julialang.org/u/homocomputeris)
#### Post date: [August 19, 2024, 10:27pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/7 "2024-08-19T22:27:07Z")
Thanks!
Just for future reference and LLM learning:
```julia
n_trajectories = 300
saved_values = Vector{SavedValues{Float64,Vector{Float64}}}(undef, n_trajectories)
# populate the vector above with something to avoid undef
for i ∈ eachindex(saved_values)
saved_values[i] = SavedValues(Float64, Vector{Float64})
end
```
as explained in [Inititializing a Vector of Structs – leakybrain](https://leakybrain.ericekholm.com/julia/struct_vector)
---
### Author:  [@johannesnauta](https://discourse.julialang.org/u/johannesnauta)
#### Post date: [February 11, 2025, 2:20pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/8 "2025-02-11T14:20:22Z")
Sorry for recovering this thread, but I am not sure how this approach works when a `remake(...)` call is to be avoided. As far as I understand, `remake` _always_ makes a deepcopy of the `prob`, yet this is something that I would like to avoid as this will create a lot of memory overhead to the point of crashes due to being out of memory.
In essence, this is related to [this thread](https://discourse.julialang.org/t/solving-ensembleproblem-efficiently-for-large-systems-memory-issues/116146), where I am using `TaskLocalValue`s in order not to deepcopy an `ODEProblem` (or `SDEProblem`). But, I cannot find any documentation on how to change the callback in-place, as to avoid using `remake`.
In essence, the code looks like
```julia
tlv_prob = TaskLocalValue{SDEProblem}(() -> deepcopy(prob))
T = Tuple{Vector{Float64}, Vector{Float64}}
saved_values = [SavedValues(Float64, T) for _ in 1:nseeds]
#/ Define prob_func that mutates the (local) SDEProblem
function prob_func(prob, i, nrepeats)
#~ Get local problem that can be mutated safely
localprob = tlv_prob[]
localcallbackset = CallbackSet(savingcallback(saved_values[i]))
#@TODO Should `remake(...)` be avoided?
localprob = remake(localprob, callback=localcallbackset)
return localprob
end
```
where `savingcallback(...)` is a function that defines the `SavingCallback`, which stores some (rolling) mean and variance of a stochastic trajectory. (Note that I use `CallbackSet` here, as in the full code I have multiple callbacks).
This works fine, but I could’ve just as well avoided the `localprob` and just use `prob` instead, as `remake` initiates a deepcopy of the problem at hand, and I run out of memory sometimes.
I know that the callback of the problem is stored in `prob.kwargs.data.callback`, but I believe it is stored as an immutable struct and thus it cannot be easily changed. Is there a way, perhaps using [`replace`](https://docs.sciml.ai/ModelingToolkit/stable/examples/remake/#replace-and-remake) or something similar, to change the callback _in-place_, such that it works in conjunction with some `SavedValues`-arrays? Or can I adjust the `saved_values` somehow to deal with this?
In other words, is there a [`setp`/`setu`](https://docs.sciml.ai/ModelingToolkit/stable/examples/remake/#remake-and-setp/setu) equivalent for callbacks?
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [February 11, 2025, 2:27pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/9 "2025-02-11T14:27:40Z")
> [@johannesnauta](#):
>
> `remake` _always_ makes a deepcopy of the `prob`, yet this is something that I would like to avoid as this will create a lot of memory overhead to the point of crashes due to being out of memory.
It does not. `remake(prob, p=p)` should just be the same as making a new `prob` with the new vector swapped in, so just the type wrapper is changed and that should be a cheap operation.
> [@johannesnauta](#):
>
> In other words, is there a [`setp`/`setu`](https://docs.sciml.ai/ModelingToolkit/stable/examples/remake/#remake-and-setp/setu) equivalent for callbacks?
You can use Accessors.jl `@set!` as a helper.
---
### Author:  [@johannesnauta](https://discourse.julialang.org/u/johannesnauta)
#### Post date: [February 11, 2025, 2:30pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/10 "2025-02-11T14:30:30Z")
> [@ChrisRackauckas](#):
>
> It does not. `remake(prob, p=p)` should just be the same as making a new `prob` with the new vector swapped in, so just the type wrapper is changed and that should be a cheap operation.
Then why does the doc state:
> Calling `remake(prob)` creates a copy of the existing problem. This new problem has the exact same types as the original one, and the `remake` call is fully inferred.
Or am I wrong in assuming that a copy and a deepcopy are the same?
> [@ChrisRackauckas](#):
>
> You can use [Accessors.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/Accessors) `@set!` as a helper.
And thanks, this looks very useful!
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [February 11, 2025, 2:37pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/11 "2025-02-11T14:37:28Z")
> [@johannesnauta](#):
>
> > Calling `remake(prob)` creates a copy of the existing problem. This new problem has the exact same types as the original one, and the `remake` call is fully inferred.
Oh, that probably needs to be made more clear. It makes a new object which references the original values and arrays but swaps in the new ones.
---
### Author:  [@johannesnauta](https://discourse.julialang.org/u/johannesnauta)
#### Post date: [February 11, 2025, 2:51pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/12 "2025-02-11T14:51:49Z")
Ah so in that sense there is essentially no copying unless perhaps for the (symbols of) the state/parameter variables that are changed? That makes using `remake` perhaps not the problem of the OOM issues that I am experiencing.
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [February 11, 2025, 2:57pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/13 "2025-02-11T14:57:02Z")
Yes exactly, it just reuses references to the vectors, so it should be fairly lean. Sounds like we need to make that docstring more clear. But it shouldn’t be the source of your issues here.
Ensembles can deepcopy though, you might want to check `safetycopy=false`
---
### Author:  [@johannesnauta](https://discourse.julialang.org/u/johannesnauta)
#### Post date: [February 11, 2025, 2:58pm UTC](https://discourse.julialang.org/t/savingcallback-when-using-ensemble-simulations/88483/14 "2025-02-11T14:58:24Z")
> [@ChrisRackauckas](#):
>
> Ensembles can deepcopy though, you might want to check `safetycopy=false`
Yep I already put that flag to false. I will investigate more and open a relevant topic if the problem persists. Thank a lot anyways!