# Concept of continuous depths for neural networks in UDEs

**URL:** <https://discourse.julialang.org/t/concept-of-continuous-depths-for-neural-networks-in-udes/72090>\
**Category:** Modelling & Simulations\
**Tags:** sciml\
**Created:** [November 26, 2021, 2:30am UTC](https://discourse.julialang.org/t/concept-of-continuous-depths-for-neural-networks-in-udes/72090 "2021-11-26T02:30:21Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![saadfaizan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/saadfaizan/32/29036_2.png) [@saadfaizan](https://discourse.julialang.org/u/saadfaizan)\
**Post date:** [November 26, 2021, 2:30am UTC](https://discourse.julialang.org/t/concept-of-continuous-depths-for-neural-networks-in-udes/72090/1 "2021-11-26T02:30:21Z")

</div>

I recently came across the concept of continuous depth in Neural ODEs. I was wondering if the neural networks in the UDE framework have a similar concept of continuous depths. @ChrisRackauckas

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 26, 2021, 3:10am UTC](https://discourse.julialang.org/t/concept-of-continuous-depths-for-neural-networks-in-udes/72090/2 "2021-11-26T03:10:54Z")

</div>

Yes, ODEs are continuous in their independent variable.

---

<div class="post-metadata">

**Author:** ![saadfaizan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/saadfaizan/32/29036_2.png) [@saadfaizan](https://discourse.julialang.org/u/saadfaizan)\
**Post date:** [November 26, 2021, 3:17am UTC](https://discourse.julialang.org/t/concept-of-continuous-depths-for-neural-networks-in-udes/72090/3 "2021-11-26T03:17:15Z")

</div>

@ChrisRackauckas Sorry. I think I did not frame my question properly. Is the depth of the neural network considered a ‘parameter’ in the UDE framework? Is it similar to continuous depth of neural ODEs?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 26, 2021, 11:54am UTC](https://discourse.julialang.org/t/concept-of-continuous-depths-for-neural-networks-in-udes/72090/4 "2021-11-26T11:54:39Z")

</div>

I mean, you can think about it like that but I don’t think it’s a very useful abstraction if you’re thinking about real models.

---

<div class="post-metadata">

**Author:** ![mschauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mschauer/32/13946_2.png) [@mschauer](https://discourse.julialang.org/u/mschauer)\
**Post date:** [November 26, 2021, 1:06pm UTC](https://discourse.julialang.org/t/concept-of-continuous-depths-for-neural-networks-in-udes/72090/5 "2021-11-26T13:06:00Z")

</div>

Yeah, you have to be careful that with continuous depth you can reparametrize depth to make a network of depth 10 into a network of depth 3.1415 without changing it…

---

<div class="post-metadata">

**Author:** ![saadfaizan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/saadfaizan/32/29036_2.png) [@saadfaizan](https://discourse.julialang.org/u/saadfaizan)\
**Post date:** [November 28, 2021, 11:16pm UTC](https://discourse.julialang.org/t/concept-of-continuous-depths-for-neural-networks-in-udes/72090/6 "2021-11-28T23:16:26Z")

</div>

Thanks

---

<div class="post-metadata">

**Author:** ![saadfaizan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/saadfaizan/32/29036_2.png) [@saadfaizan](https://discourse.julialang.org/u/saadfaizan)\
**Post date:** [November 28, 2021, 11:19pm UTC](https://discourse.julialang.org/t/concept-of-continuous-depths-for-neural-networks-in-udes/72090/7 "2021-11-28T23:19:17Z")

</div>

@ChrisRackauckas @mschauer Is there any benefit in terms of training accuracy by parameterizing the depth compared to a case where it is not done so?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 30, 2021, 5:52am UTC](https://discourse.julialang.org/t/concept-of-continuous-depths-for-neural-networks-in-udes/72090/8 "2021-11-30T05:52:33Z")

</div>

I don’t think that framework of thinking even makes sense with UDEs. Take a look at this recent talk:

[![](https://global.discourse-cdn.com/julialang/original/3X/9/c/9c0cf1bb5c4ab5b1311abe6a965fa2b4e75563ff.jpeg "The Use and Practice of Scientific Machine Learning (Chris Rackauckas) - nextgen\_ai Freiburg 2021") ](https://www.youtube.com/watch?v=FihLyzdjN_8)

It’s all about mechanistic models. If you’re not talking mechanistic models, then you’re not talking UDEs.