# Mixed-mode automatic differentiation using ForwardDiff and ReverseDiff

**URL:** <https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440>\
**Category:** General Usage\
**Tags:** forwarddiff, reversediff, autodiff\
**Created:** [January 11, 2022, 10:10pm UTC](https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440 "2022-01-11T22:10:32Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![StevenWhitaker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevenwhitaker/32/9749_2.png) [@StevenWhitaker](https://discourse.julialang.org/u/StevenWhitaker)\
**Post date:** [January 11, 2022, 10:10pm UTC](https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440/1 "2022-01-11T22:10:32Z")

</div>

I need to take the gradient of a function. I can use ForwardDiff without any issues, but for part of my code I have found ReverseDiff to run much faster. The issue is that the other part of my code errors with ReverseDiff. The basic structure of my code looks something like

```julia
function take_my_gradient(x)

    tmp = errors_with_reversediff(x)
    faster_with_reversediff(tmp)

end

```

Is it possible to differentiate my function using ForwardDiff on the part that doesn’t work with ReverseDiff and using ReverseDiff on the other part? (ReverseDiff.jl’s [README](https://github.com/JuliaDiff/ReverseDiff.jl#why-use-reversediff) seems to indicate it is possible.) If so, can someone please explain how?

---

<div class="post-metadata">

**Author:** ![StevenWhitaker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevenwhitaker/32/9749_2.png) [@StevenWhitaker](https://discourse.julialang.org/u/StevenWhitaker)\
**Post date:** [January 27, 2022, 3:25pm UTC](https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440/2 "2022-01-27T15:25:30Z")

</div>

If anyone has any ideas about this, I would greatly appreciate the help.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 27, 2022, 3:31pm UTC](https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440/3 "2022-01-27T15:31:39Z")

</div>

> [@StevenWhitaker](#):
>
> The issue is that the other part of my code errors with ReverseDiff.

Generally, if ReverseDiff (or Zygote) doesn’t handle a portion of a calculation (separated into some function), you should just use [ChainRulesCore.jl](https://github.com/JuliaDiff/ChainRulesCore.jl) to define a custom “pullback” (vector–Jacobian product) rule for that function, either with manual differentiation (typically by an [adjoint method](https://math.mit.edu/~stevenj/18.336/adjoint.pdf)) or by using some other AD package (though forward-mode AD is not that efficient for pullbacks).

> [@StevenWhitaker](#):
>
> but for part of my code I have found ReverseDiff to run much faster

In general, reverse-mode differentiation (a.k.a. backpropagation or adjoint methods) is much faster than forward-mode when you are computing gradients (i.e. the derivative of one ~~input~~ output with respect to many ~~outputs~~ inputs). See also our [matrix-calculus course notes](https://github.com/mitmath/matrixcalc).

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [January 27, 2022, 4:36pm UTC](https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440/4 "2022-01-27T16:36:29Z")

</div>

> [@stevengj](#):
>
> (i.e. the derivative of one input with respect to many outputs)

Reversed?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 27, 2022, 6:11pm UTC](https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440/5 "2022-01-27T18:11:36Z")

</div>

Whoops, fixed.

---

<div class="post-metadata">

**Author:** ![StevenWhitaker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevenwhitaker/32/9749_2.png) [@StevenWhitaker](https://discourse.julialang.org/u/StevenWhitaker)\
**Post date:** [January 28, 2022, 8:14pm UTC](https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440/6 "2022-01-28T20:14:24Z")

</div>

Thanks for your response!

> [@stevengj](#):
>
> you should just use [ChainRulesCore.jl](https://github.com/JuliaDiff/ChainRulesCore.jl) to define a custom “pullback” (vector–Jacobian product) rule for that function

This was the direction I needed. It hadn’t crossed my mind to enable mixed-mode AD by defining a custom `rrule` that used ForwardDiff. I was able to get things working by doing so (though I had to switch to Zygote; I also tried ReverseDiff and Yota but they didn’t work with my code).

* * *

As a side note for those interested, using this mixed-mode AD with Zygote and ForwardDiff is actually slower and more memory intensive for my problem (i.e., I get `OutOfMemoryError`s for larger problem sizes using mixed-mode AD, even when using a `ForwardDiff.Chunk` size of 1). So I’ll probably stick with just ForwardDiff for now.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 28, 2022, 9:09pm UTC](https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440/7 "2022-01-28T21:09:29Z")

</div>

> [@StevenWhitaker](#):
>
> As a side note for those interested, using this mixed-mode AD with Zygote and ForwardDiff is actually slower and more memory intensive for my problem (i.e., I get `OutOfMemoryError` s for larger problem sizes using mixed-mode AD,

Reverse-mode AD has to store results of all of the intermediate steps of your algorithm in order to “backpropagate” the derivatives, so it is notorious be memory intensive if you are trying to differentiate some kind of iterative calculation.

However, there are often workarounds. For example, if you are using an iterative method to solve some system of equations (an iterative linear or nonlinear solver), you can instead define an “adjoint” pullback rule directly on the solution that completely avoids backpropagating through the iteration. People sometimes also apply techniques like “checkpointing” to trade off computation and memory.

In general, it is extremely helpful to know something about how forward and reverse-mode AD algorithms work in order to use them effectively, and sometimes to know when you should judiciously replace AD with manual derivatives (vector–Jacobian or Jacobian–vector products) for a portion of your code.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [January 28, 2022, 10:20pm UTC](https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440/8 "2022-01-28T22:20:56Z")

</div>

I would use Zygote to orchestrate the reverse-mode AD, then define an rrule for the function that I want to use ForwardDiff with and define another rrule for the function that I want to use ReverseDiff with. The time you will spend inside Zygote should be small that way. Just be careful that when you use ForwardDiff for an rrule, you are essentially evaluating the whole Jacobian to define the rrule which depending on the structure of the Jacobian may not be the most efficient thing to do.

A bit of shameless self promotion: I made this video [Understanding automatic differentiation (in Julia) - YouTube](https://www.youtube.com/watch?v=UqymrMG-Qi4) explaining some of these concepts which you may find useful.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 28, 2022, 11:03pm UTC](https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440/9 "2022-01-28T23:03:33Z")

</div>

> [@mohamed82008](#):
>
> A bit of shameless self promotion: I made this video [Understanding automatic differentiation (in Julia) - YouTube](https://www.youtube.com/watch?v=UqymrMG-Qi4) explaining some of these concepts which you may find useful.

Thanks, I’ve linked to that from our Matrix Calculus course ([GitHub - mitmath/matrixcalc: MIT IAP short course: Matrix Calculus for Machine Learning and Beyond](https://github.com/mitmath/matrixcalc)).

---

<div class="post-metadata">

**Author:** ![StevenWhitaker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevenwhitaker/32/9749_2.png) [@StevenWhitaker](https://discourse.julialang.org/u/StevenWhitaker)\
**Post date:** [February 1, 2022, 5:22pm UTC](https://discourse.julialang.org/t/mixed-mode-automatic-differentiation-using-forwarddiff-and-reversediff/74440/10 "2022-02-01T17:22:19Z")

</div>

Thanks for the link; I’ll be sure to check it out.