# Why Optim.jl does not allow for backwards mode autodifferentiation?

**URL:** <https://discourse.julialang.org/t/why-optim-jl-does-not-allow-for-backwards-mode-autodifferentiation/100346>\
**Category:** Optimization (Mathematical)\
**Tags:** autodiff\
**Created:** [June 14, 2023, 4:30pm UTC](https://discourse.julialang.org/t/why-optim-jl-does-not-allow-for-backwards-mode-autodifferentiation/100346 "2023-06-14T16:30:18Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Devetak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devetak/32/50611_2.png) [@Devetak](https://discourse.julialang.org/u/Devetak)\
**Post date:** [June 14, 2023, 4:30pm UTC](https://discourse.julialang.org/t/why-optim-jl-does-not-allow-for-backwards-mode-autodifferentiation/100346/1 "2023-06-14T16:30:18Z")

</div>

I apologise if this is a silly question.

Why Optim.jl does not allow for backwards mode autodifferentiation? If I understand things correctly backwards mode is better for the case when the output dimension is smaller than the input dimension. This is the case in optimization problems. In fact, if I understand correctly optimization problems are the best case of this as output is just a scalar.

Is there something I am missing/getting wrong?

Thank you in advance!

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 14, 2023, 4:40pm UTC](https://discourse.julialang.org/t/why-optim-jl-does-not-allow-for-backwards-mode-autodifferentiation/100346/2 "2023-06-14T16:40:44Z")

</div>

It does. See:

> **[OptimizationFunction · Optimization.jl](https://docs.sciml.ai/Optimization/stable/API/optimization_function/#Automatic-Differentiation-Construction-Choice-Recommendations)**
>
> Documentation for Optimization.jl.

Some (but not all) of the choices are reverse mode AD.

> [@Devetak](#):
>
> If I understand things correctly backwards mode is better for the case when the output dimension is smaller than the input dimension. This is the case in optimization problems. In fact, if I understand correctly optimization problems are the best case of this as output is just a scalar.

Indeed. For large optimization problems, the reverse-mode methods are the most efficient. If you check the recommendations I linked above, it only recommends forward mode for small problems. This is because reverse mode can have more overhead and thus for a small enough optimization problem the gradients via forward-mode can still be faster. That cutoff point is problem-dependent and always changing but ~100 is around the point where you want to have definitely made the swap.

---

<div class="post-metadata">

**Author:** ![Devetak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devetak/32/50611_2.png) [@Devetak](https://discourse.julialang.org/u/Devetak)\
**Post date:** [June 15, 2023, 7:10am UTC](https://discourse.julialang.org/t/why-optim-jl-does-not-allow-for-backwards-mode-autodifferentiation/100346/3 "2023-06-15T07:10:37Z")

</div>

Maybe there was some confusion. I meant Optim.jl, not Optimization.jl. See here: [Optim.jl](https://julianlsolvers.github.io/Optim.jl/stable/#user/gradientsandhessians/).

But thank you. The package you linked solves my problems.

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [June 15, 2023, 7:52am UTC](https://discourse.julialang.org/t/why-optim-jl-does-not-allow-for-backwards-mode-autodifferentiation/100346/4 "2023-06-15T07:52:23Z")

</div>

If you want to use reverse mode with Optim directly, you need to pass the gradient function yourself as the second argument to `Optim.optimize`.  
IIUC, Optimization.jl is a front-end that does this automatically.

---

<div class="post-metadata">

**Author:** ![ElOceanografo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eloceanografo/32/624_2.png) [@ElOceanografo](https://discourse.julialang.org/u/ElOceanografo)\
**Post date:** [June 15, 2023, 8:47am UTC](https://discourse.julialang.org/t/why-optim-jl-does-not-allow-for-backwards-mode-autodifferentiation/100346/5 "2023-06-15T08:47:15Z")

</div>

A minimum working example of how to do that (it’s easy!):

```julia
using Optim, ReverseDiff

f(x) = sum(abs2, x) # objective function
g!(G, x) = ReverseDiff.gradient!(G, f, x)

x0 = randn(1000)
optimize(f, g!, x0)

```

---

<div class="post-metadata">

**Author:** ![Devetak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devetak/32/50611_2.png) [@Devetak](https://discourse.julialang.org/u/Devetak)\
**Post date:** [June 15, 2023, 11:12am UTC](https://discourse.julialang.org/t/why-optim-jl-does-not-allow-for-backwards-mode-autodifferentiation/100346/6 "2023-06-15T11:12:51Z")

</div>

Yes of course! I was just confused why forward mode is supported natively and for backwards you need to provide it yourself.

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [June 15, 2023, 12:27pm UTC](https://discourse.julialang.org/t/why-optim-jl-does-not-allow-for-backwards-mode-autodifferentiation/100346/7 "2023-06-15T12:27:39Z")

</div>

Historical reasons – forward mode was correct and stable long before backward mode was.