# Taking gradients in Julia

**URL:** <https://discourse.julialang.org/t/taking-gradients-in-julia/68731>\
**Category:** General Usage\
**Tags:** question, zygote, forwarddiff, reversediff\
**Created:** [September 24, 2021, 11:45pm UTC](https://discourse.julialang.org/t/taking-gradients-in-julia/68731 "2021-09-24T23:45:03Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ZHEN\_LIN](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhen_lin/32/27672_2.png) [@ZHEN\_LIN](https://discourse.julialang.org/u/ZHEN_LIN)\
**Post date:** [September 24, 2021, 11:45pm UTC](https://discourse.julialang.org/t/taking-gradients-in-julia/68731/1 "2021-09-24T23:45:03Z")

</div>

I’m looking for a fast package that can take gradients of functions I write in Julia. Similar to the `autograd` package for Python.

It seems that there are a few different options such as `Zygote`, `ForwardDiff.jl`, `ReverseDiff.jl` and many others.

What would be a fast (or maybe the fastest) and accurate option to take gradients in Julia?

Thanks.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [September 24, 2021, 11:58pm UTC](https://discourse.julialang.org/t/taking-gradients-in-julia/68731/2 "2021-09-24T23:58:53Z")

</div>

> [@ZHEN\_LIN](#):
>
> What would be a fast (or maybe the fastest) and accurate option to take gradients in Julia?

For a summary of options, see: [https://juliadiff.org/](https://juliadiff.org/)

If you have a few parameters and a lot of functions to differentiate, probably use ForwardDiff (forward-mode differentiation); if you have a lot of parameters and a few functions, probably use Zygote or maybe ReverseDiff (reverse-mode differentiation).

---

<div class="post-metadata">

**Author:** ![ZHEN\_LIN](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhen_lin/32/27672_2.png) [@ZHEN\_LIN](https://discourse.julialang.org/u/ZHEN_LIN)\
**Post date:** [September 25, 2021, 1:50am UTC](https://discourse.julialang.org/t/taking-gradients-in-julia/68731/3 "2021-09-25T01:50:33Z")

</div>

> [@stevengj](#):
>
> If you have a few parameters and a lot of functions to differentiate, probably use ForwardDiff (forward-mode differentiation); if you have a lot of parameters and a few functions, probably use Zygote or maybe ReverseDiff (reverse-mode differentiation).

Thanks! What would be the reason or intuition behind the difference of the packages’ performance in these different cases you mentioned?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 25, 2021, 3:46am UTC](https://discourse.julialang.org/t/taking-gradients-in-julia/68731/4 "2021-09-25T03:46:08Z")

</div>

We’re putting out a paper in a few days that will go through quite a few examples of AD package performance and how it differs, but one discussion of this can be found in the following paper:

> **[1812.01892.pdf](https://arxiv.org/pdf/1812.01892.pdf)**
>
> 588.27 KB

You might want to watch the talk that explains the results:

[![](https://global.discourse-cdn.com/julialang/original/3X/1/8/181eab0d3ecc8b6bb414fd210ff1716711dedef4.jpeg "A Comparison of Automatic Differentiation and Adjoints for Derivatives of Differential Equations") ](https://www.youtube.com/watch?v=6hhF6Llv4sI)

but one slide that’s really relevant:

 ![Capture](https://global.discourse-cdn.com/julialang/original/3X/3/6/36cf87e1e811ce91e2e8c68dfb14633c14cb0b9b.png)

Essentially forward mode methods scale like the number of inputs while reverse mode scales like the number of outputs, but in many applications this can look like `O(states * parameters)` for forward vs `O(states + parameters)` for reverse. So obviously reverse is better right? Wrong: there are many natural reasons why reverse-mode AD will have a higher baseline overhead.

So if forward-mode AD is faster when problems are small and reverse-mode AD is faster when problems are large, where’s the cutoff? That’s very problem-dependent, and the other paper to be posted soon will show that the given problem can change what AD packages are going to be fast as well. But one thing to look at is the following:

 ![E_v2wmUXIAEssaZ](https://global.discourse-cdn.com/julialang/original/3X/c/4/c43d385138a3a0886fe9a46de438c66bb4393046.png)

We found that when you had like a size 50 system you’d get the some reverse-mode methods (“based on” Enzyme.jl) would be faster than ForwardDiff.jl, and around 100-150 or so you could get versions of ReverseDiff.jl then at the cutoff. So “roughly 100” is a decent general idea for switching from forward to reverse, depending on the properties of the package.

More on details about taking gradients in a tweet thread [https://twitter.com/ChrisRackauckas/status/1440018868269985796](https://twitter.com/ChrisRackauckas/status/1440018868269985796) .

When will Zygote be faster vs Enzyme vs etc? Will update on that in about a day or two.

---

<div class="post-metadata">

**Author:** ![ZHEN\_LIN](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhen_lin/32/27672_2.png) [@ZHEN\_LIN](https://discourse.julialang.org/u/ZHEN_LIN)\
**Post date:** [September 25, 2021, 10:47pm UTC](https://discourse.julialang.org/t/taking-gradients-in-julia/68731/5 "2021-09-25T22:47:17Z")

</div>

> [@ChrisRackauckas](#):
>
> When will Zygote be faster vs Enzyme vs etc? Will update on that in about a day or two.

Thanks a lot! Look forward to this update!

Also I’m curious about if you observe any “significant” difference in speed among these different packages. Or they are roughly comparable even if one is slightly faster in the different situations as you described?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 25, 2021, 10:56pm UTC](https://discourse.julialang.org/t/taking-gradients-in-julia/68731/6 "2021-09-25T22:56:00Z")

</div>

It can be an order of magnitude difference.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [September 25, 2021, 10:58pm UTC](https://discourse.julialang.org/t/taking-gradients-in-julia/68731/7 "2021-09-25T22:58:39Z")

</div>

> [@ZHEN\_LIN](#):
>
> Also I’m curious about if you observe any “significant” difference in speed among these different packages.

It’s not just a difference in software. There is a fundamental difference in algorithms and computational scaling between forward and reverse mode AD, as I said, and which one is better depends on the number of inputs vs the number of outputs; google it.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 28, 2021, 1:26am UTC](https://discourse.julialang.org/t/taking-gradients-in-julia/68731/8 "2021-09-28T01:26:55Z")

</div>

Here’s the paper I mentioned where Appendix B describes how on the same application 4 or 5 different AD mechanisms can be the optimal choice depending on the user inputs.

> **[2109.12449.pdf](https://arxiv.org/pdf/2109.12449.pdf)**
>
> 758.21 KB

This paper also conveniently describes AbstractDifferentiation.jl which is a higher level API for using any AD system, which I would recommend for handling this complexity.

[https://github.com/JuliaDiff/AbstractDifferentiation.jl](https://github.com/JuliaDiff/AbstractDifferentiation.jl)