# Training universal differential equations

**URL:** <https://discourse.julialang.org/t/training-universal-differential-equations/100787>\
**Category:** Modelling & Simulations\
**Created:** [June 24, 2023, 4:51pm UTC](https://discourse.julialang.org/t/training-universal-differential-equations/100787 "2023-06-24T16:51:31Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![T\_J](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/t_j/32/51019_2.png) [@T\_J](https://discourse.julialang.org/u/T_J)\
**Post date:** [June 24, 2023, 4:51pm UTC](https://discourse.julialang.org/t/training-universal-differential-equations/100787/1 "2023-06-24T16:51:31Z")

</div>

Hello,

In a universal differential equation where the ODE + NN is not able to finish the integration routine, Is there a procedure (best practices) to execute in order to train the ODE?

If the integration procedure is not finished the evaluation of the `loss` will throw an error due to differences in the number of points to evaluate. This will most likely be because the estimation of the NN is not yielding good results and the system will possible become unstable.

Is there a procedure to adopt in this sort of situations? Any criteria to check if the ODE will be stable? Can the output of the `predict` function be in `try-except` block. If the integration works, do something, if not do something else? Will `AD` be able to account for this?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 25, 2023, 2:06am UTC](https://discourse.julialang.org/t/training-universal-differential-equations/100787/2 "2023-06-25T02:06:37Z")

</div>

> [@T\_J](#):
>
> In a universal differential equation where the ODE + NN is not able to finish the integration routine, Is there a procedure (best practices) to execute in order to train the ODE?

Iteratively growing the system is one good procedure:

> **[Strategies to Avoid Local Minima · SciMLSensitivity.jl](https://docs.sciml.ai/SciMLSensitivity/dev/tutorials/training_tips/local_minima/)**
>
> Documentation for SciMLSensitivity.jl.

Or doing something like dividing the new term by a big constant (1000) so that it starts as approximately zero, presuming your model is stable when the NN is zero.

---

<div class="post-metadata">

**Author:** ![patrick-kidger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrick-kidger/32/20378_2.png) [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)\
**Post date:** [June 25, 2023, 6:31am UTC](https://discourse.julialang.org/t/training-universal-differential-equations/100787/3 "2023-06-25T06:31:10Z")

</div>

Also: train just your mechanistic component first. Then freeze its parameters and train just the neural network part.

This means that the mechanistic part of your neural ODE will learn as much of the physics as it can, and have the neural network part only learn the residual.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [June 25, 2023, 10:19am UTC](https://discourse.julialang.org/t/training-universal-differential-equations/100787/4 "2023-06-25T10:19:08Z")

</div>

If you wish to learn anything about the values of the parameters of the mechanistic mode, or “learn the physics”, this approach will likely lead to biased parameter estimates, unless by chance the residual happens to be zero mean and close to normally distributed (in case of MSE loss). Having said that, it’s still probably worth doing if you’re experiencing issues with instability, just keep in mind that your “mechanistic model” is unlikely to represent some kind of “truth” or “physics”.

---

<div class="post-metadata">

**Author:** ![T\_J](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/t_j/32/51019_2.png) [@T\_J](https://discourse.julialang.org/u/T_J)\
**Post date:** [June 25, 2023, 10:41am UTC](https://discourse.julialang.org/t/training-universal-differential-equations/100787/5 "2023-06-25T10:41:51Z")

</div>

Hi,  
Would you elaborate a bit on this? How can I freeze its parameters and train just the neural network part? Any documentation with this example?

---

<div class="post-metadata">

**Author:** ![T\_J](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/t_j/32/51019_2.png) [@T\_J](https://discourse.julialang.org/u/T_J)\
**Post date:** [June 25, 2023, 10:46am UTC](https://discourse.julialang.org/t/training-universal-differential-equations/100787/6 "2023-06-25T10:46:27Z")

</div>

> this approach will likely lead to biased parameter estimates, unless by chance the residual happens to be zero mean and close to normally distributed (in case of MSE loss)

the `try-except` block? Or to approach are you referring to?

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [June 25, 2023, 10:53am UTC](https://discourse.julialang.org/t/training-universal-differential-equations/100787/7 "2023-06-25T10:53:43Z")

</div>

The approach to learn part of the model first, and then “complete the model” with a neural network later. The problem is that the parameters learned in the initial stage where the model is incomplete are likely to be wrong (a biased estimate).

---

<div class="post-metadata">

**Author:** ![T\_J](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/t_j/32/51019_2.png) [@T\_J](https://discourse.julialang.org/u/T_J)\
**Post date:** [July 6, 2023, 3:09pm UTC](https://discourse.julialang.org/t/training-universal-differential-equations/100787/8 "2023-07-06T15:09:22Z")

</div>

Can you elaborate a bit on how to achieve that?