# Lipschitz continuity -- Neural ODEs

**URL:** <https://discourse.julialang.org/t/lipschitz-continuity-neural-odes/108463>\
**Category:** Machine Learning\
**Tags:** discussion\
**Created:** [January 7, 2024, 12:29pm UTC](https://discourse.julialang.org/t/lipschitz-continuity-neural-odes/108463 "2024-01-07T12:29:26Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![RahulManavalan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rahulmanavalan/32/31945_2.png) [@RahulManavalan](https://discourse.julialang.org/u/RahulManavalan)\
**Post date:** [January 7, 2024, 12:29pm UTC](https://discourse.julialang.org/t/lipschitz-continuity-neural-odes/108463/1 "2024-01-07T12:29:26Z")

</div>

Here’s a open ended topic that I haven’t seen people mention, explicitly in literature.

In order to integrate neural ODEs, there needs to be guarentees regarding the existence of its solutions. And any arbitrary NN approximation, be it from a CNN, GNN or simply a SHLNN (single hidden layer, fully connected NN); is not necessarily Lipschitz continuous. While there is work on Lipschitz continuous neural networks; there is little reference to such works in the neural ode literature.

@ChrisRackauckas @avikpal Have you thought about this? If I did miss any papers from you; could you please point me to them?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 7, 2024, 2:18pm UTC](https://discourse.julialang.org/t/lipschitz-continuity-neural-odes/108463/2 "2024-01-07T14:18:27Z")

</div>

> [@RahulManavalan](#):
>
> any arbitrary NN approximation, be it from a CNN, GNN or simply a SHLNN (single hidden layer, fully connected NN); is not necessarily Lipschitz continuous.

Why wouldn’t they be Lipshitz? NNs are usually finite compositions of Lipshitz functions (typically linear operations like matrix multiplies and convolutions & Lipshitz activation functions like ReLUs), no?

There are even [algorithms](https://proceedings.neurips.cc/paper_files/paper/2019/file/95e1533eb1b20a97777749fb94fdb944-Paper.pdf) to [automatically estimate the Lipshitz constant](https://proceedings.neurips.cc/paper_files/paper/2018/file/d54e99a6c03704e95e6965532dec148b-Paper.pdf) of DNNs and to [bound the Lipshitz constant during training](https://arxiv.org/abs/1804.04368).

Or do you mean you are worried that the Lipshitz constant may be large? But even if it is large, the ODE solution will still exist, no?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 7, 2024, 12:13pm UTC](https://discourse.julialang.org/t/lipschitz-continuity-neural-odes/108463/3 "2024-04-07T12:13:28Z")

</div>

They are Lipschitz, so I don’t understand the fuss.

And then if the Lipshitz constant is large, this paper goes into detail about how the adjoint is changed to handle that case:

> **[Stiff neural ordinary differential equations](https://pubs.aip.org/aip/cha/article/31/9/093122/1077547/Stiff-neural-ordinary-differential-equations)**
>
> Neural Ordinary Differential Equations (ODEs) are a promising approach to learn dynamical models from time-series data in science and engineering applicati