I am a beginner in programming in julia and am studying the basics of deep learning with Flux and Zygote. In my study I have only encountered a few examples of backprop but I realized (by reading the discussions here and watching videos on youtube such as Mike Innes on julia language channel) that a…

Maybe worth starting with GitHub - MikeInnes/diff-zoo: Differentiation for Hackers There are a variety of good academic reviews of the subject, tutorials and textbooks if you want to learn more.

Maybe start with https://www.jmlr.org/papers/volume18/17-468/17-468.pdf

My recommend reading/watching list is here in the ChainRules docs, at the bottom of the page I have been told that the ChainRules docs themselves are quite educational. We definitely tried hard to be, though I think they could be better.

Are there any books that also have examples in julia? There are no good books on Automatic Differentiation. Let alone ones that have examples in Julia. Griewank and Walther 2008 “Evaluating Derivatives” is fairly comprehensive (for reverse mode), but not all that readable. The alternatives that…

[image] anon74562486: In my study I have only encountered a few examples of backprop but I realized (by reading the discussions here and watching videos on youtube such as Mike Innes on julia language channel) that automatic differentiation is an extremely broad subject, and I know nothing about…

Thanks to everyone, every one of your responses has been helpful.

From my understanding reverse mode AD is more general and can handle more outputs, where backpropagation is for only 1 scalar outputs. Could be, but remember the field is absolutely terrible at consistency of naming things. Probably because it’s good at reinventing things and giving them new nam…

Learning automatic differentiation

Offtopic

anon74562486 February 27, 2021, 9:50pm 3

Thank you.
I’m a beginner, which textbook/tutorial can you recommend me?

1 Like

Topic		Replies	Views
Understanding automatic differentiation (long-form YouTube video) Teaching & Outreach autodiff	0	779	December 11, 2021
Automatic differentiation - Julia implementation advantages Machine Learning	8	4188	June 7, 2019
Automatic differentiation in Julia for gradient computation General Usage autodiff	1	814	September 8, 2023
What is the difference between Zygote vs ForwardDiff and ReverseDiff Machine Learning	4	6772	February 23, 2021
Which autodiff to currently use for a neural network backend? General Usage package , statistics , machinevision	10	2252	October 1, 2018

Learning automatic differentiation

Related topics