# Automatic Differentiation

**URL:** <https://discourse.julialang.org/t/automatic-differentiation/17472>\
**Category:** Machine Learning\
**Created:** [November 13, 2018, 2:00pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472 "2018-11-13T14:00:35Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![kkmann](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kkmann/32/1727_2.png) [@kkmann](https://discourse.julialang.org/u/kkmann)\
**Post date:** [November 13, 2018, 2:00pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/1 "2018-11-13T14:00:35Z")

</div>

Hey,

I am still having a bit of a hard time figuring out what flux.jl actually does. One important aspect seems to be making gradients available automatically via tracking. How does that differ from the approach in ForwardDiff.jl, i.e. why is it not possible/feasible/sensible to get the gradient of my loss with respect to all parameters using ForwardDiff?

The current tracking implementation seems to be very similar to what Tensorflow + eager execution does, right?

Thanks,

Kevin

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 13, 2018, 2:07pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/2 "2018-11-13T14:07:37Z")

</div>

Flux is reverse, while ForwardDiff is forward-mode AD (as its name implies). See eg [wikipedia](https://en.wikipedia.org/wiki/Automatic_differentiation#The_chain_rule,_forward_and_reverse_accumulation) about the difference.

> [@kkmann](#):
>
> why is it not possible/feasible/sensible to get the gradient of my loss with respect to all parameters using ForwardDiff?

Reverse mode is ideal for \mathbb{R}^n\to\mathbb{R} functions, while forward mode is the opposite. That said, in practice, for a small n ForwardDiff can be fine.

---

<div class="post-metadata">

**Author:** ![kkmann](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kkmann/32/1727_2.png) [@kkmann](https://discourse.julialang.org/u/kkmann)\
**Post date:** [November 13, 2018, 2:18pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/3 "2018-11-13T14:18:16Z")

</div>

Thanks for the quick answer. Reverse \> Forward due to speed considerations, I guess? But why not mix both - there are still some cases where flux cannot provide a gradient but ForwardDiff can - couldn’t ForwardDiff be uses as a default fallback when the tracker encounters a ‘dead end’?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 13, 2018, 2:49pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/4 "2018-11-13T14:49:13Z")

</div>

All AD libraries have limitations (but these are best reported as an issue with a minimal example so that they can be fixed). That said, I find ForwardDiff to be the most robust in practice, so it is indeed useful as a fallback.

Also, as you said, mixed mode AD can be useful, especially if the problem has a structure that can exploit this.

I would recommend that you consult a good source on AD for a deeper understanding. I find

```nohighlight
@book{griewank2008evaluating,
  title={Evaluating derivatives: principles and techniques of algorithmic differentiation},
  author={Griewank, Andreas and Walther, Andrea},
  volume={105},
  year={2008},
  publisher={Siam}
}

```

especially nice.

---

<div class="post-metadata">

**Author:** ![kkmann](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kkmann/32/1727_2.png) [@kkmann](https://discourse.julialang.org/u/kkmann)\
**Post date:** [November 13, 2018, 3:18pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/5 "2018-11-13T15:18:07Z")

</div>

thanks again!

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [November 13, 2018, 3:27pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/6 "2018-11-13T15:27:41Z")

</div>

I believe both Reversediff.jl and Flux uses mixed mode AD.  
[https://github.com/FluxML/Flux.jl/blob/39dcfd3933fff5cc498e0330de0064f521aad9a7/src/tracker/lib/array.jl](https://github.com/FluxML/Flux.jl/blob/39dcfd3933fff5cc498e0330de0064f521aad9a7/src/tracker/lib/array.jl)

It’s partly explained in this paper if I remember correctly [[1810.08297] Dynamic Automatic Differentiation of GPU Broadcast Kernels](https://arxiv.org/abs/1810.08297)

---

<div class="post-metadata">

**Author:** ![RoyiAvital](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/royiavital/32/571_2.png) [@RoyiAvital](https://discourse.julialang.org/u/RoyiAvital)\
**Post date:** [February 11, 2019, 1:26pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/7 "2019-02-11T13:26:02Z")

</div>

I saw this today - [A Benchmark of Selected Algorithmic Differentiation Tools on Some Problems in Computer Vision and Machine Learning](https://arxiv.org/abs/1807.10129).

The code is given in - [ADBench - `AutoDiff` Benchmarks](https://github.com/awf/autodiff).

What do you think?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 11, 2019, 1:43pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/8 "2019-02-11T13:43:56Z")

</div>

The abstract describes the approach as

> a skilled programmer devotingroughly a week to each tool produced the timings we present.

I would have thought that a skilled programmer would notice that a library named `ForwardDiff` lacking _reverse_ AD may not be an accident, and proceed to try at least

> **[GitHub - JuliaDiff/ReverseDiff.jl: Reverse Mode Automatic Differentiation for...](https://github.com/JuliaDiff/ReverseDiff.jl)**
>
> Reverse Mode Automatic Differentiation for Julia. Contribute to JuliaDiff/ReverseDiff.jl development by creating an account on GitHub.

if nothing else.

I have failed to find the actual Julia code, but I wonder if [at least the low-hanging optimizations](http://www.juliadiff.org/ForwardDiff.jl/latest/user/advanced.html#Configuring-Chunk-Size-1) were applied.

---

<div class="post-metadata">

**Author:** ![RoyiAvital](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/royiavital/32/571_2.png) [@RoyiAvital](https://discourse.julialang.org/u/RoyiAvital)\
**Post date:** [February 11, 2019, 1:53pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/9 "2019-02-11T13:53:52Z")

</div>

I was surprised to see that MATLAB’s libraries were faster.  
Julia should beat MATLAB even without those “Low Hanging Fruits”.

It is really strange…

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 11, 2019, 2:00pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/10 "2019-02-11T14:00:34Z")

</div>

> [@RoyiAvital](#):
>
> Julia should beat MATLAB even without those “Low Hanging Fruits”.

I am not sure — note that if you don’t specify the config, the `ForwardDiff` methods themselves are type unstable as the chunks are calculated dynamically.

**EDIT** the repo is somewhat disorganized, but the Julia code appears to be [here](https://github.com/awf/autodiff/tree/d1bc878a7d04d9e1033f104ca9ec358a9e8c2fca/tools/Julia). It seems it does not to precalculate chunk size or configuration, and also has various readily apparent [type stability problems](https://github.com/awf/autodiff/blob/d1bc878a7d04d9e1033f104ca9ec358a9e8c2fca/usr/awf/Julia/gmm3.jl#L64).

---

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [February 11, 2019, 2:12pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/11 "2019-02-11T14:12:35Z")

</div>

Code appears to be here: [ADBench/usr/awf/Julia at master · microsoft/ADBench · GitHub](https://github.com/awf/autodiff/tree/master/usr/awf/Julia)  
There should be plenty of optimizations left to be done here: Julia’s forward diff is considerably slower than C++'s finite difference. Julia’s ReverseDiff has a number of problems, so I would imagine they tried it, it didn’t work and they didn’t pursue it further (which is fair game). There are references to ReverseDiffSource in the code, though. In any case, writing efficient Julia code is not completely trivial, and using Julia’s autodiff tools efficiently is even harder (at the moment at least), so we can’t really blame the author there.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 11, 2019, 2:19pm UTC](https://discourse.julialang.org/t/automatic-differentiation/17472/12 "2019-02-11T14:19:18Z")

</div>

> [@antoine-levitt](#):
>
> writing efficient Julia code is not completely trivial, and using Julia’s autodiff tools efficiently is even harder (at the moment at least), so we can’t really blame the author there.

Since C++ was included, I think it is reasonable to give the same amount of effort to Julia as it would take to write it in C++. And that would take a lot, so that should cover at least following the advice in the [performance tips](https://docs.julialang.org/en/v1/manual/performance-tips/#man-performance-tips-1). IMO that would be the minimum to make this comparison interesting — nothing more than some applications of `@code_warntype` and benchmarking.