# Differential equation becomes unstable when training mixed neural ODE

**URL:** <https://discourse.julialang.org/t/differential-equation-becomes-unstable-when-training-mixed-neural-ode/61461>\
**Category:** Modelling & Simulations\
**Created:** [May 19, 2021, 6:55pm UTC](https://discourse.julialang.org/t/differential-equation-becomes-unstable-when-training-mixed-neural-ode/61461 "2021-05-19T18:55:31Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![bkuwahara](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkuwahara/32/25239_2.png) [@bkuwahara](https://discourse.julialang.org/u/bkuwahara)\
**Post date:** [May 19, 2021, 6:55pm UTC](https://discourse.julialang.org/t/differential-equation-becomes-unstable-when-training-mixed-neural-ode/61461/1 "2021-05-19T18:55:31Z")

</div>

I’m trying to use a neural network to simulate the equation for the predator population in the Lotka-Volterra model:

```julia
function lotka_volterra(du,u,p,t)
  x, y = u
  α, β = (1.5,1.0)
  du[1] = dx = α*x - β*x*y
  du[2] = NN(u, p)
end 

```

The neural network is `NN = FastChain(FastDense(2, 50, tanh), FastDense(50, 1))`. I’m training it against a time series using the real Lotka-Volterra equations using a squared-difference loss function with `sciml_train` and the BFGS optimizer with `initial_stepnorm=0.01f0`.

It runs fine for about 120 iterations, reaching a loss of about 0.03. Then I get the message “Warning: Instability detected. Aborting”. Any idea why this is happening?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 20, 2021, 11:33am UTC](https://discourse.julialang.org/t/differential-equation-becomes-unstable-when-training-mixed-neural-ode/61461/2 "2021-05-20T11:33:23Z")

</div>

For an ODE, there are always some parameters that give a divergent ODE. For example, `u' = a * u`, if `a` is positive and goes large, you can bet the answer will go to infinity very fast and you’ll get `“Warning: Instability detected. Aborting”`. So it’s a fact of life with these kinds of nonlinear models. Luckily there’s an FAQ page on good ways to handle this topic, such as setting divergent trajectories to have infinite loss. See:

[https://diffeqflux.sciml.ai/dev/examples/divergence/](https://diffeqflux.sciml.ai/dev/examples/divergence/)

---

<div class="post-metadata">

**Author:** ![bkuwahara](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkuwahara/32/25239_2.png) [@bkuwahara](https://discourse.julialang.org/u/bkuwahara)\
**Post date:** [May 20, 2021, 3:37pm UTC](https://discourse.julialang.org/t/differential-equation-becomes-unstable-when-training-mixed-neural-ode/61461/3 "2021-05-20T15:37:23Z")

</div>

Thanks for the clarification! I tried the example on the FAQ page and it worked for me, but when I implement the same infinite cost system in my own cost function, I get

```julia
MethodError: Cannot `convert` an object of type Nothing to an object of type Float32

```

and the code stops running rather than continuing to train. Any idea why this might be happening?

Also, I have a callback function plotting the solution at each iteration and it doesn’t look like it’s becoming unstable. Is it normal to have instability arise so suddenly?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 20, 2021, 4:08pm UTC](https://discourse.julialang.org/t/differential-equation-becomes-unstable-when-training-mixed-neural-ode/61461/4 "2021-05-20T16:08:56Z")

</div>

With BFGS? I think @Vaibhavdixit02 was mentioning something.

---

<div class="post-metadata">

**Author:** ![Vaibhavdixit02](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vaibhavdixit02/32/2916_2.png) [@Vaibhavdixit02](https://discourse.julialang.org/u/Vaibhavdixit02)\
**Post date:** [May 20, 2021, 4:36pm UTC](https://discourse.julialang.org/t/differential-equation-becomes-unstable-when-training-mixed-neural-ode/61461/5 "2021-05-20T16:36:39Z")

</div>

This is exactly what I was seeing, this would be with `AutoZygote` (which is default in `sciml_train`) and Optim optimizer.

@bkuwahara pass in the `AutoForwardDiff` argument like `DiffEqFlux.sciml_train(loss,pinit,Newton(), GalacticOptim.AutoForwardDiff())` and try or else use `ADAM` both those should avoid this issue.

Or the best alternative right now would be to use the `size` comparison like in [https://diffeqflux.sciml.ai/dev/examples/divergence/](https://diffeqflux.sciml.ai/dev/examples/divergence/) in your loss function instead of checking the `retcode`

---

<div class="post-metadata">

**Author:** ![bkuwahara](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkuwahara/32/25239_2.png) [@bkuwahara](https://discourse.julialang.org/u/bkuwahara)\
**Post date:** [May 20, 2021, 5:44pm UTC](https://discourse.julialang.org/t/differential-equation-becomes-unstable-when-training-mixed-neural-ode/61461/6 "2021-05-20T17:44:03Z")

</div>

Thanks @Vaibhavdixit02! Using AutoForwardDiff seems to solve the problem. For some reason, the program still stops after only 20 or so iterations with a loss of about 0.3 (maxiters is set to 100) but at least it doesn’t return an error.

I’ve tried ADAM with this particular problem before but I switched to BFGS because ADAM doesn’t seem to want to converge. Even training with BFGS to a loss on the order of 0.7 and then switching to ADAM, ADAM(0.01) immediately jumps to a loss of ~300 and stays there.

I was using the size `size` comparison method to handle the error, and the program was still giving me the `MethodError` message, so I’m still not quite sure what’s causing that.

In any case, I’ll probably stick to using AutoForwardDiff and maybe try out some different network structures to see if they’re more consistently stable. I’m very new to machine learning and Julia as a whole, so I really appreciate the assistance from both of you.

---

<div class="post-metadata">

**Author:** ![Vaibhavdixit02](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vaibhavdixit02/32/2916_2.png) [@Vaibhavdixit02](https://discourse.julialang.org/u/Vaibhavdixit02)\
**Post date:** [May 21, 2021, 8:30am UTC](https://discourse.julialang.org/t/differential-equation-becomes-unstable-when-training-mixed-neural-ode/61461/7 "2021-05-21T08:30:10Z")

</div>

Yeah if you are using the same example I found best result with `Newton` [Update divergence.md by Vaibhavdixit02 · Pull Request #551 · SciML/DiffEqFlux.jl · GitHub](https://github.com/SciML/DiffEqFlux.jl/pull/551) you can take a look here for my conclusions

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 5, 2023, 6:11am UTC](https://discourse.julialang.org/t/differential-equation-becomes-unstable-when-training-mixed-neural-ode/61461/8 "2023-03-05T06:11:46Z")

</div>

The link is now:

[https://docs.sciml.ai/SciMLSensitivity/stable/tutorials/training\_tips/divergence/](https://docs.sciml.ai/SciMLSensitivity/stable/tutorials/training_tips/divergence/)