# Second-order autodiff: which combinations should work?

**URL:** <https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892>\
**Category:** Optimization (Mathematical)\
**Tags:** zygote, forwarddiff, autodiff, enzyme, diffractor\
**Created:** [May 29, 2024, 6:19am UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892 "2024-05-29T06:19:53Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 29, 2024, 6:19am UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/1 "2024-05-29T06:19:53Z")

</div>

Hey there!

My package DifferentiationInterface.jl provides the necessary infrastructure for second-order automatic differentiation (e.g. Hessians). It allows you to combine arbitrary pairs of AD backends, but of course many of the combinations will fail. My question is therefore: **which pairs of backends should I aim to support and test?**

This was prompted by [issue #278](https://github.com/gdalle/DifferentiationInterface.jl/issues/278) of DI, which asks for second-order Enzyme support, but we can try to cover more ground! Judging by the [list in the README](https://github.com/gdalle/DifferentiationInterface.jl?tab=readme-ov-file#compatibility), there are 13 available AD packages, which translates to roughly 14 different backends (counting forward and reverse Enzyme). I don’t want to test \binom{14}{2} combinations, so I thought of some ways to reduce the list:

- symbolic backends (Symbolics.jl and FastDifferentiation.jl) will rarely be paired with something else, we can discard them
- finite differences backends (FiniteDiff.jl and FiniteDifferences.jl) _should_ pair well with most other backends so I’m less curious about the results of testing
- experimental backends (Diffractor.jl and Tapir.jl) are low-priority
- there are some near-duplicates in the list (I anticipate bikeshedding on this):
  - PolyesterForwardDiff.jl is well-tested if ForwardDiff.jl is
  - ChainRulesCore.jl is well-tested if Zygote.jl is

Here’s the table of combinations that are currently part of the test suite for Hessians, I will add your suggestions to it as the discussion progresses (if they make sense). The table should be read as “outer (row) over inner (column)”.

| outer \ inner | **Enz [F]** | **Enz [R]** | **ForDiff** | **RevDiff** | **Tracker** | **Zygote** |
| --- | --- | --- | --- | --- | --- | --- |
| **Enz [F]** | wanted | tested | wanted | | | |
| **Enz [R]** | wanted | tested | | | | |
| **ForDiff** | tested | wanted | tested | wanted | | tested |
| **RevDiff** | | | | tested | | |
| **Tracker** | | | | | | |
| **Zygote** | | | | | | tested |

What do you think?  
Pinging @oxinabox @wsmoses @ChrisRackauckas @Vaibhavdixit02 @avikpal

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 29, 2024, 7:00am UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/2 "2024-05-29T07:00:53Z")

</div>

Enzyme over enzyme works just fine. You just need to use the “delayed” form for the interior.

Forward over Tracker has issues, I’d just ignore that one.

The rest looks right.

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [May 29, 2024, 9:46am UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/3 "2024-05-29T09:46:01Z")

</div>

For an example of Enzyme over Enzyme, see  
[Automatic differentiation of user-defined operators · JuMP](https://jump.dev/JuMP.jl/stable/tutorials/nonlinear/operator_ad/#Hessian-3)

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 29, 2024, 11:33am UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/4 "2024-05-29T11:33:56Z")

</div>

> [@ChrisRackauckas](#):
>
> Enzyme over enzyme works just fine. You just need to use the “delayed” form for the interior.

I added this trick in the following PR, but I wonder if it is possible to learn a lesson from it.

> <https://github.com/gdalle/DifferentiationInterface.jl/pull/280>

The underlying issue is that for Enzyme, you have to do something differently in order to enable higher-order differentiation (use `autodiff_deferred` instead of `autodiff`).

- Do we lose something if we use `autodiff_deferred` everywhere? Maybe @vchuravy can help.
- Is this dichotomy also true for other backends, like Zygote.jl or Tapir.jl (@willtebbutt)? In that case we might be able to define two versions of important operators like `DI.gradient` and `DI.derivative`: an optimized one and a higher-order friendly one.

---

<div class="post-metadata">

**Author:** ![wsmoses](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsmoses/32/26497_2.png) [@wsmoses](https://discourse.julialang.org/u/wsmoses)\
**Post date:** [May 29, 2024, 11:40am UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/5 "2024-05-29T11:40:39Z")

</div>

The need for deferred is specific to GPU-compiler related packages (including CUDA.jl, etc).

It’s been on our todo list to make our abstract interpreter automatically upgrade internal autodiffs to deferred, but I don’t know enough about the Julia abstract interpreter to do so, and we’ve so far not found someone who does (open issue here: [Automate use of deferred in Higher order derivatives · Issue #1005 · EnzymeAD/Enzyme.jl · GitHub](https://github.com/EnzymeAD/Enzyme.jl/issues/1005) )

Answering your earlier question, generally speaking Enzyme on the outside of all of those AD libraries should work in practice (including itself with deferred on the inside). In practice, not sure but thats why its worth testing.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 29, 2024, 11:44am UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/6 "2024-05-29T11:44:28Z")

</div>

Thanks for the answer!

> [@wsmoses](#):
>
> The need for deferred is specific to GPU-compiler related packages (including [CUDA.jl](https://juliahub.com/ui/Packages/General/CUDA), etc).

My later question was rather: “is it suboptimal in terms of performance if I replace every `autodiff` with `autodiff_deferred` in DI (even for standard first order stuff)”?  
It would make my life _a lot_ simpler not having to handle two versions of each operator, a direct one and a deferred one.

---

<div class="post-metadata">

**Author:** ![willtebbutt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/willtebbutt/32/6790_2.png) [@willtebbutt](https://discourse.julialang.org/u/willtebbutt)\
**Post date:** [May 29, 2024, 11:57am UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/7 "2024-05-29T11:57:08Z")

</div>

> [@gdalle](#):
>
> Is this dichotomy also true for other backends, like [Zygote.jl](https://juliahub.com/ui/Packages/General/Zygote) or [Tapir.jl](https://juliahub.com/ui/Packages/General/Tapir) (@willtebbutt)? In that case we might be able to define two versions of important operators like `DI.gradient` and `DI.derivative`: an optimized one and a higher-order friendly one.

@oxinabox and I are sorting out Diffractor.jl-over-Tapir.jl (forwards-over-reverse) [at the minute](https://github.com/compintell/Tapir.jl/issues/157). I don’t _believe_ there will be any special requirements to defer stuff, or anything like that. All this being said, this is work-in-progress, so things might change.

---

<div class="post-metadata">

**Author:** ![wsmoses](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsmoses/32/26497_2.png) [@wsmoses](https://discourse.julialang.org/u/wsmoses)\
**Post date:** [May 29, 2024, 12:10pm UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/8 "2024-05-29T12:10:58Z")

</div>

There are complications that come if you use deferred, which is why autodiff itself doesn’t just use it by default (though this remains a debate between myself and @vchuravy).

If you know anyone with abstract interpreter experience to help us get the autodiff to autodiff deferred over the finish line, and that is easier than writing the wrapper code and DI, go for it!

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 29, 2024, 1:58pm UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/9 "2024-05-29T13:58:21Z")

</div>

> [@wsmoses](#):
>
> If you know anyone with abstract interpreter experience to help us get the autodiff to autodiff deferred over the finish line

I’m sorry that’s not gonna be me or anyone I know well ^^

> [@wsmoses](#):
>
> writing the wrapper code and DI

I think I’m gonna go for an additional set of operators that are not exposed in the API but that will basically amount to `gradient_higher_order_friendly`. It’s ugly but I should be able to do it with minimal code

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [May 29, 2024, 2:05pm UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/10 "2024-05-29T14:05:53Z")

</div>

If you want the gradient of a scalar-valued function that depends on the gradient of another scalar-valued function, you can use forward-over-reverse combining ForwardDiff with e.g. Zygote or Enzyme or ReverseDiff. See:

> [@Nested AD with Lux etc](https://discourse.julialang.org/t/nested-ad-with-lux-etc/113573/27):
>
> For future reference, here is a manual implementation of forward-over-reverse calculation with parameter gradients (mixed second derivatives). In particular, if you have a scalar-valued h(x,p) = g(\nabla\_x f) for some scalar-valued f(x,p), then one can similarly derive: \left. \nabla\_p h \right|\_{x,p} = \left. \frac{\partial}{\partial\alpha} \left. \nabla\_p f \right|\_{x + \alpha \left. \nabla g \right|\_{z},p} \right|\_{\alpha = 0} \, , where z = \left. \nabla\_x f \right|\_{x,p}. Here is an e…

(You can also use this approach for general Hessians, but it was less obvious to me that it is efficient for scalar-valued functions.)

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 29, 2024, 2:25pm UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/11 "2024-05-29T14:25:25Z")

</div>

At the moment I’m only interested in plain boring second-order autodiff of a function `f` that is presumably defined without autodiff inside of it

---

<div class="post-metadata">

**Author:** ![cgeoga](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cgeoga/32/216186_2.png) [@cgeoga](https://discourse.julialang.org/u/cgeoga)\
**Post date:** [May 29, 2024, 2:34pm UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/12 "2024-05-29T14:34:45Z")

</div>

I use `ForwardDiff.jl` over `ReverseDiff.jl` for Hessians regularly, which has great performance with compiled tapes. Here is my little wrapper struct for doing this efficiently, although maybe the AD people here will wince: [~cgeoga/StandaloneKNITRO.jl (master): src/forwrapper.jl - sourcehut git](https://git.sr.ht/~cgeoga/StandaloneKNITRO.jl/tree/master/item/src/forwrapper.jl).

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 29, 2024, 2:46pm UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/13 "2024-05-29T14:46:13Z")

</div>

That’s interesting, thanks for sharing!

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 30, 2024, 10:13am UTC](https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892/14 "2024-05-30T10:13:09Z")

</div>

I ended up wrapping the backend object, so that `AutoDeferredEnzyme` uses `autodiff_deferred` and `AutoEnzyme` uses `AutoDiff`.  
**Second order with forward Enzyme over reverse Enzyme now works in DI** (as of [v0.5.1](https://github.com/JuliaRegistries/General/pull/107945)), and we can start testing more package combinations too!