# To understand neural ODE

**URL:** <https://discourse.julialang.org/t/to-understand-neural-ode/58236>\
**Category:** Machine Learning\
**Tags:** question\
**Created:** [March 30, 2021, 2:48pm UTC](https://discourse.julialang.org/t/to-understand-neural-ode/58236 "2021-03-30T14:48:51Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![vavrines](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vavrines/32/8024_2.png) [@vavrines](https://discourse.julialang.org/u/vavrines)\
**Post date:** [March 30, 2021, 2:48pm UTC](https://discourse.julialang.org/t/to-understand-neural-ode/58236/1 "2021-03-30T14:48:51Z")

</div>

I find I still don’t fully understand neural ordinary differential equations.

As shown in literature (Lu et al., 2017; Haber and Ruthotto, 2017; Chen el al., 2018), a sequence of  
transformations  
`h(t+1) = h(t) + f(h,theta)`  
can be turned into an ODE in the limiting case  
`dh/dt = f(h,theta)`

This formulation holds only under the condition that `theta` is a function of time.  
This way we can say we’ve got an equivalent block of infinite depth.  
However, in applications we just define a network with a fix number of parameters.  
For example, in the following codes,

```julia
using DiffEqFlux, OrdinaryDiffEq
dudt = FastChain(FastDense(2, 50, tanh), FastDense(50, 2))
u0 = Float32[2.0; 0.0]
tspan = (0.f0, 1.f0)
nn = NeuralODE(dudt, tspan, Tsit5())

```

`nn.p` is a fix-sized vector with 252 entries.  
Although the output of NN is feed into next iteration as input in the Tsit5 solver, the old parameters are still there.  
Does the neural ODE hold better learning capacity than a single layer, or they are just the same in this case?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 30, 2021, 4:59pm UTC](https://discourse.julialang.org/t/to-understand-neural-ode/58236/2 "2021-03-30T16:59:08Z")

</div>

> [@vavrines](#):
>
> Does the neural ODE hold better learning capacity than a single layer, or they are just the same in this case?

It’s `n` layers, where `n` is determined adaptively by the neural network.

---

<div class="post-metadata">

**Author:** ![vavrines](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vavrines/32/8024_2.png) [@vavrines](https://discourse.julialang.org/u/vavrines)\
**Post date:** [March 30, 2021, 5:09pm UTC](https://discourse.julialang.org/t/to-understand-neural-ode/58236/3 "2021-03-30T17:09:22Z")

</div>

Could you elaborate a bit on the adaptivity?  
To me it seems everything is fixed after `nn` is defined in my demo code.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 30, 2021, 5:40pm UTC](https://discourse.julialang.org/t/to-understand-neural-ode/58236/4 "2021-03-30T17:40:33Z")

</div>

It defines and ODE which is solved by an adaptive ODE solver. Depending on what steps the ODE solver decides to take, that’s how many layers you effectively have.

---

<div class="post-metadata">

**Author:** ![vavrines](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vavrines/32/8024_2.png) [@vavrines](https://discourse.julialang.org/u/vavrines)\
**Post date:** [March 30, 2021, 6:31pm UTC](https://discourse.julialang.org/t/to-understand-neural-ode/58236/5 "2021-03-30T18:31:12Z")

</div>

Yeah now I got what you mean. In fact this is exactly what I’m asking.  
Sure with different ODE solvers we can have different number of steps, or say “n layers”, but all these layers rely on the parameters of `dudt = FastChain(FastDense(2, 50, tanh), FastDense(50, 2))`.  
Are they really as good as a neural network that has n real layers (with of course the geometrically increasing number of parameters), e.g. `FastChain(dudt, dudt, ...)`?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 30, 2021, 6:37pm UTC](https://discourse.julialang.org/t/to-understand-neural-ode/58236/6 "2021-03-30T18:37:42Z")

</div>

It’s then similar to recurrent models.