# DiffEqFlux: neural\_ode stops prematurely

**URL:** <https://discourse.julialang.org/t/diffeqflux-neural-ode-stops-prematurely/19995>\
**Category:** General Usage\
**Tags:** diffeq\
**Created:** [January 23, 2019, 5:03pm UTC](https://discourse.julialang.org/t/diffeqflux-neural-ode-stops-prematurely/19995 "2019-01-23T17:03:22Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![fastwave](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fastwave/32/14725_2.png) [@fastwave](https://discourse.julialang.org/u/fastwave)\
**Post date:** [January 23, 2019, 5:03pm UTC](https://discourse.julialang.org/t/diffeqflux-neural-ode-stops-prematurely/19995/1 "2019-01-23T17:03:22Z")

</div>

I am trying to replicate the example in the README of DiffEqFlux [https://github.com/JuliaDiffEq/DiffEqFlux.jl](https://github.com/JuliaDiffEq/DiffEqFlux.jl). Calling the neural\_ode generated function makes Julia exit before training could begin.

The code is

> using DifferentialEquations  
> using Flux, DiffEqFlux
> 
> function lotka\_volterra(du,u,p,t)  
> x, y = u  
> α, β, δ, γ = p  
> du[1] = dx = α_x - β_x_y  
> du[2] = dy = -δ_y + γ_x_y  
> end  
> u0 = [1.0,1.0]  
> tspan = (0.0,10.0)  
> p = [1.5,1.0,3.0,1.0]  
> prob = ODEProblem(lotka\_volterra,u0,tspan,p)  
> ode\_data = Array(solve(prob,Tsit5(),saveat=0.1))
> 
> dudt = Chain(Dense(2,50,tanh),Dense(50,2))  
> tspan = (0.0f0,10.0f0)  
> n\_ode = x-\>neural\_ode(x,dudt,tspan,Tsit5(),saveat=0.1)
> 
> function predict\_n\_ode()  
> n\_ode(u0)  
> end  
> loss\_n\_ode() = sum(abs2,ode\_data .- predict\_n\_ode())
> 
> data = Iterators.repeated((), 100)  
> opt = ADAM(0.1)
> 
> cb = function () #callback function to observe training  
> display(loss\_n\_ode())  
> end
> 
> println(“Before crashing”)  
> n\_ode(u0)  
> println(“After crashing”)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [January 23, 2019, 7:04pm UTC](https://discourse.julialang.org/t/diffeqflux-neural-ode-stops-prematurely/19995/2 "2019-01-23T19:04:13Z")

</div>

> [@fastwave](#):
>
> n\_ode = x-\>neural\_ode(x,dudt,tspan,Tsit5(),saveat=0.1)

That’s the old (yesterday night before we released) syntax. Basically, swap `x` and `dudt`:

```julia
n_ode = x->neural_ode(dudt,x,tspan,Tsit5(),saveat=0.1)

```

Where in the docs do we have this? It would be good to fix that.

Edit: Fixed the docs. Thanks for the report!

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [January 23, 2019, 7:08pm UTC](https://discourse.julialang.org/t/diffeqflux-neural-ode-stops-prematurely/19995/3 "2019-01-23T19:08:31Z")

</div>

BTW, I’ll like to see what neural network you come up with to fit Lotka-Volterra. I was running the animations and recording them live on a core i5 laptop, so I kept it to the simple case 🙂 . But when I did try to train LV with one hidden layer the NN didn’t seem big enough to capture the function. But on my laptop I couldn’t use the GPUs, so I’m interested to see what kind of NN can be used here 🙂 .

(Also, there’s a much better way to train this, but that’s the topic for another publication)

---

<div class="post-metadata">

**Author:** ![fastwave](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fastwave/32/14725_2.png) [@fastwave](https://discourse.julialang.org/u/fastwave)\
**Post date:** [January 24, 2019, 3:40pm UTC](https://discourse.julialang.org/t/diffeqflux-neural-ode-stops-prematurely/19995/4 "2019-01-24T15:40:51Z")

</div>

Hi, thanks for the help, it works now. I cannot use a GPU either, simply because I don’t have one. So far my experience is that these networks are difficult to train. I don’t think it is the size or depth of the network. I think it is because of the nature of ODEs. Perturbations are amplified exponentially in time and that is hard to handle with any optimisation. Anyway, I will do some more experimentation before making a judgement.

My strategy would be to train with many short trajectories first and then improve on that with smaller number of longer trajectories. At the moment I have no clue how to do multiple trajectories, my modification of the loss function does not work. If you can give an example with two trajectories, that would be great. Thanks

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [January 24, 2019, 3:56pm UTC](https://discourse.julialang.org/t/diffeqflux-neural-ode-stops-prematurely/19995/5 "2019-01-24T15:56:45Z")

</div>

> [@fastwave](#):
>
> I think it is because of the nature of ODEs. Perturbations are amplified exponentially in time and that is hard to handle with any optimisation. Anyway, I will do some more experimentation before making a judgement.

Yup that’s definitely the case.

> [@fastwave](#):
>
> My strategy would be to train with many short trajectories first and then improve on that with smaller number of longer trajectories. At the moment I have no clue how to do multiple trajectories, my modification of the loss function does not work. If you can give an example with two trajectories, that would be great. Thanks

That’s multiple shooting. We actually do that in DiffEq-proper: [http://docs.juliadiffeq.org/latest/analysis/parameter\_estimation.html](http://docs.juliadiffeq.org/latest/analysis/parameter_estimation.html). We will be putting a paper out on how to loss functions that improve the fitting. What the blog post shows is the training using single shooting which is what the paper shows, but we know that there are better ways 🙂.