# Nested and different AD methods altogether: How to add AD calculations inside my loss function when using neural differential equations?

**URL:** <https://discourse.julialang.org/t/nested-and-different-ad-methods-altogether-how-to-add-ad-calculations-inside-my-loss-function-when-using-neural-differential-equations/108985>\
**Category:** Machine Learning\
**Tags:** sciml, ad, neural-network, differentialequation\
**Created:** [January 18, 2024, 8:43pm UTC](https://discourse.julialang.org/t/nested-and-different-ad-methods-altogether-how-to-add-ad-calculations-inside-my-loss-function-when-using-neural-differential-equations/108985 "2024-01-18T20:43:51Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![facusapienza](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/facusapienza/32/24317_2.png) [@facusapienza](https://discourse.julialang.org/u/facusapienza)\
**Post date:** [January 21, 2024, 4:16pm UTC](https://discourse.julialang.org/t/nested-and-different-ad-methods-altogether-how-to-add-ad-calculations-inside-my-loss-function-when-using-neural-differential-equations/108985/2 "2024-01-21T16:16:13Z")

</div>

Forward differentiation here does not break but ignores the gradients with respect to the derivative term,

```julia
dUdx = map(x -> ForwardDiff.jacobian(x -> U([x[1], 1.0], θ, st)[1], [x]), steps_reg)

```

gives

```julia
Warning: `ForwardDiff.jacobian(f, x)` within Zygote cannot track gradients with respect to `f`,
│ and `f` appears to be a closure, or a struct with fields (according to `issingletontype(typeof(f))`).

```

@ChrisRackauckas I noticed this is pretty much the same problem reported in the post [Gradient of Gradient in Zygote](https://discourse.julialang.org/t/gradient-of-gradient-in-zygote/52685/3) but here I am interested in reverse-over-forward differentiation. Also a similar thread in [Issue with Zygote ober ForwardDiff-derivative](https://discourse.julialang.org/t/issue-with-zygote-over-forwardDiff.derivative/70824). However, it is not clear for me what is the recommended solution for this cases, if there is any yet. I noticed the posts are a little bit old, so maybe some of their contents may be outdated. Do I need to define a new `rrule()` for this problem in order to make this work?

---

_[View the full topic](https://discourse.julialang.org/t/nested-and-different-ad-methods-altogether-how-to-add-ad-calculations-inside-my-loss-function-when-using-neural-differential-equations/108985)._