Significant allocations with Callbacks (Tsit5)

I have a reasonably optimized ODE problem where I used callback for terminating the integrator when some values get too large. Without callback I have (43 allocations: 16.23 KiB), and with callback I have (42,065,928 allocations: 3.13 GiB). The callback takes more than 40% of the time (see below).

Here’s a minimal working example using the lorenz problem:

function lorenz!(du,u,p,t)
 du[1] = 10.0*(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz!,u0,tspan)

function condition(out, u, t, integrator)
    out[1] = u[1] - 25000;
    out[2] = u[2] - 0.2;
    out[3] = u[3] - 1.2;
end

function affect!(integrator, index)
    if (index == 1)
        # ...
        # terminate!(integrator);
    elseif (index == 2)
        # ...
        # terminate!(integrator);
    elseif (index == 3)
        # ...
        # terminate!(integrator);
    end        
end
cellFate = VectorContinuousCallback(condition, affect!, nothing, 3, save_positions=(false, false));

@btime solve(prob,Tsit5()) # no callback
@btime solve(prob,Tsit5(), callback=cellFate) # with callback

In this example, without callback it takes 689.839 μs (11748 allocations: 1.37 MiB), and with callback it takes 2.161 ms (24628 allocations: 2.34 MiB). I don’t understand why the callback function could have so many allocations.

2 Likes

Yeah, that looks like a performance bug, and it turned out to be a journey.

https://github.com/SciML/SciMLBase.jl/pull/99
https://github.com/SciML/DiffEqBase.jl/pull/705
https://github.com/SciML/OrdinaryDiffEq.jl/pull/1473

See the caveats of the last PR though… still working through that.

2 Likes

Maybe you can hack it by using isoutofdomain: let that function return true when the solution goes beyond where you want to terminate. The solver should then (after reaching the minimum step) stop.

Although, Chris is usually super fast with fixing those issues… so you got to be quick!

This issue is weird though, did you see the gif?

bizarre_allocations

The way to get rid of the allocations requires (currently) that you use Revise and reevaluate one of the functions in the package (without changing it). I gotta say, this is a first for me :sweat_smile:

1 Like

I observed the same behavior before, see `prolong2mortars!` with `P4estMesh` allocates sometimes · Issue #628 · trixi-framework/Trixi.jl · GitHub for an analysis (and a dirty workaround in the PR closing that issue). The basic reason there is an inference failure involving recursive calls (and splatting of tuples in this case).

2 Likes