Monitoring the progress of DiffEq solve

The motivation is that, sometimes, solve(prob) takes long time to run. While running, DiffEq doesn’t print any output (even with debug logging on). There is no easy way to tell what’s going on, e.g., if it’s stuck at a certain step, taking long to reinit, or slowly but steadily progressing (although this can be told by using the progressbar).

Is there a recommended way to print status information from DiffEq while the solver is running? I currently insert debug statements in f!, jac!, and all the callback functions, but that’s probably not enough to tell what’s happening in the solver.

You can use the integrator interface to do this. The call

solve(prob, alg; kwargs...)

is essentially equivalent to

integrator = init(prob, alg; kwargs)
for i in integrator
end

So you could simply write that out and put some logging code in the loop to track integration status. The page I linked shows multiple variants of that loop that e.g. has information about the time and current solution or more.

1 Like

you can do solve(prob, alg ; progress = true), see doc

4 Likes

Thank you! This advances my understanding of the integrator interface.

But the same issue remains. When it freezes at a particular step, I don’t know the exact internal subroutine it’s currently running. For example, it could be due to many causes: reinitializing the DAE upon modified u, rejecting a step due to large errors, or simply taking too long to solve nonlinear equations.

I was thinking that maybe having some @debug information in DiffEq can help. I don’t see it throughout the OrdinaryDiffEq codebase, probably there are more important considerations. I would be curious to learn.

Thank you! I’m aware of the progress bar integration and have used it. It is useful when the simulation progresses consistently and slowly.

My issue is that my model has switching. The solver may freeze at particular steps with some combination of tolerance and solution methods. Sometimes, it freezes indefinitely (20 minutes with no progress in percentage) for a problem that might just take several seconds for another algorithm/tolerance. I want to find out what the solver is doing internally.

I guess I can profile the solve function. But if, in the best-case scenario, the function takes 20 minutes to run, then I’ll have to wait that long before seeing the results. If any internal debug information can be readily printed while it’s running, it will be helpful. I would like to know if there’s a way to do so without going into solve! and manually edit the functions that get called.

You can trigger the profiler interactively to get a 1s profile printed out: https://docs.julialang.org/en/v1/stdlib/Profile/#Triggered-During-Execution

3 Likes