# Learning automatic differentiation

**URL:** <https://discourse.julialang.org/t/learning-automatic-differentiation/56158>\
**Category:** Offtopic\
**Created:** [February 27, 2021, 9:31pm UTC](https://discourse.julialang.org/t/learning-automatic-differentiation/56158 "2021-02-27T21:31:22Z")\
**Posts on this page:** 1\
**Showing post:** 7

<div class="post-metadata">

**Author:** ![kimlaberinto](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kimlaberinto/32/21534_2.png) [@kimlaberinto](https://discourse.julialang.org/u/kimlaberinto)\
**Post date:** [February 28, 2021, 5:52am UTC](https://discourse.julialang.org/t/learning-automatic-differentiation/56158/7 "2021-02-28T05:52:55Z")

</div>

> [@anon74562486](#):
>
> In my study I have only encountered a few examples of backprop but I realized (by reading the discussions here and watching videos on youtube such as Mike Innes on julia language channel) that automatic differentiation is an extremely broad subject, and I know nothing about it.

That’s a very similar experience I had as well. I’m still learning Automatic Differentiation (AD), but here are some resources and links that helped my understanding.

When I starting going through the text/lectures/documents describing AD, at first I had a hard time understanding what the difference between Forward Mode AD and Reverse Mode AD was and the context between different techniques.

So here’s a basic/overview video on YouTube that I liked and helped me to get a broad overview of the topic.

**What is Automatic Differentiation? by Ari Seff**

[![](https://global.discourse-cdn.com/julialang/original/3X/8/3/8361843e81ba6f96ef0f1aaba079a9912f69bff5.jpeg "What is Automatic Differentiation?") ](https://www.youtube.com/watch?v=wG_nF1awSSY)

- I found this video helped clarify what the difference was between Forward Mode and Reverse Mode and how their use cases can differ.
- Its a fairly broad overview but helped me better understand the context of everything.
- The video also has a introduction on how AD is different from symbolic and numerical differentiation.

After I got a gist of how forward-mode and reverse-mode are different, I got confused on what the difference was between Backpropagation and Reverse Mode AD was. From what I learned, Backpropagation is a specific case of Reverse Mode AD. From my understanding reverse mode AD is more general and can handle more outputs, where backpropagation is for only 1 scalar outputs.

> <https://stackoverflow.com/questions/23485636/how-is-backpropagation-the-same-or-not-as-reverse-automatic-differentiation>

Then, I also realized that I’ve already seen Reverse Mode AD in Python before with backprop in machine learning algorithms. So that got me thinking about what the difference is with Python and Julia AD ecosystems. There was a great conversation on Slack that that. Feel free to check here:

> [@Automatic Differentiation (AD) in Python compared to Julia and AD Basics](https://discourse.julialang.org/t/automatic-differentiation-ad-in-python-compared-to-julia-and-ad-basics/56031):
>
> Hello everyone! The quotes below is a reconstruction of a wonderful conversation I had with Lyndon White (@oxinabox) on the Julia Slack on February 24, 2021 (Thank you to Lyndon as well to let me quote and repost this on Discourse)! I thought it might be useful to record it in Discourse in case anyone had similar questions about some of the differences in Automatic Differentiation in Python compared to Julia. And we can also continue the discussion on this if anyone wants to add anything else. …

Now that I have a better contextual understanding I’m starting to dive deeper into that reading list that was mentioned by Lyndon in the ChainRules docs. I’m trying to deepen my understanding of how Automatic Differentiation (AD) works.

Let me know if there are any questions or any corrections to what I said. I’m still learning, but I hope this helps!

---

_[View the full topic](https://discourse.julialang.org/t/learning-automatic-differentiation/56158)._