# Warning secondorder ADtype in optimization.jl

**URL:** <https://discourse.julialang.org/t/warning-secondorder-adtype-in-optimization-jl/122539>\
**Category:** New to Julia\
**Tags:** differentiation, optimization\
**Created:** [November 12, 2024, 10:04am UTC](https://discourse.julialang.org/t/warning-secondorder-adtype-in-optimization-jl/122539 "2024-11-12T10:04:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![JADekker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jadekker/32/210281_2.png) [@JADekker](https://discourse.julialang.org/u/JADekker)\
**Post date:** [November 12, 2024, 10:04am UTC](https://discourse.julialang.org/t/warning-secondorder-adtype-in-optimization-jl/122539/1 "2024-11-12T10:04:19Z")

</div>

Hi, I’m trying to replicate the example from [Optim.jl · Optimization.jl](https://docs.sciml.ai/Optimization/stable/optimization_packages/optim/), i.e. I’m running

```julia
using Optimization, OptimizationOptimJL
function RunTest()
    rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
    cons = (res, x, p) -> res .= [x[1]^2 + x[2]^2]
    x0 = zeros(2)
    p = [1.0, 100.0]
    prob = OptimizationFunction(rosenbrock, Optimization.AutoForwardDiff(); cons = cons)
    prob = Optimization.OptimizationProblem(prob, x0, p, lcons = [-5.0], ucons = [10.0])
    sol = solve(prob, IPNewton())
    display(sol)
end
RunTest()

```

However, I get the following warning. How can I specify a SecondOrder with AutoForwardDiff()? (I’ve read the docs and know that I’ll probably want to use AutoEnzyme instead, but that produces the same problem, so I’ve decided to keep the MWE close to the example on the website).

> The selected optimization algorithm requires second order derivatives, but `SecondOrder` ADtype was not provided.  
> │ So a `SecondOrder` with AutoForwardDiff() for both inner and outer will be created, this can be suboptimal and not work in some cases so  
> │ an explicit `SecondOrder` ADtype is recommended.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [November 12, 2024, 10:14am UTC](https://discourse.julialang.org/t/warning-secondorder-adtype-in-optimization-jl/122539/2 "2024-11-12T10:14:49Z")

</div>

The code that governs AD backend selection seems to be here:

> <https://github.com/SciML/OptimizationBase.jl/blob/2ffab7e93197c1fc8d9ed6a39857e301a71a474e/src/adtypes.jl#L222-L236>

By default, when you provide a backend `adtype`, it will use `soadtype = DifferentiationInterface.SecondOrder(adtype, adtype)` to compute the Hessian. You can make this choice yourself by providing `adtype = SecondOrder(adtype_outer, adtype_inner)`, in which case `adtype_inner` will be used for the gradient.

Note that I deduced this from the code but if it is not documented then it is subject to change. @Vaibhavdixit02 is the right person to ask.

---

<div class="post-metadata">

**Author:** ![JADekker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jadekker/32/210281_2.png) [@JADekker](https://discourse.julialang.org/u/JADekker)\
**Post date:** [November 12, 2024, 10:22am UTC](https://discourse.julialang.org/t/warning-secondorder-adtype-in-optimization-jl/122539/3 "2024-11-12T10:22:45Z")

</div>

Thanks! By importing DifferentiationInterface I could just use

```julia
    optprob = OptimizationFunction(rosenbrock, SecondOrder(AutoForwardDiff(), AutoForwardDiff()), cons = con2_c)

```

which solved the problem for me!

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [November 12, 2024, 10:27am UTC](https://discourse.julialang.org/t/warning-secondorder-adtype-in-optimization-jl/122539/4 "2024-11-12T10:27:52Z")

</div>

It’s a bit weird that the default example throws a warning, I agree. And the current implementation is slightly suboptimal too. I opened an issue to clarify and improve this part

> <https://github.com/SciML/OptimizationBase.jl/issues/129>
>
> This issue is about the machinery for choosing backends and throwing warnings:
> …
> https://github.com/SciML/OptimizationBase.jl/blob/2ffab7e93197c1fc8d9ed6a39857e301a71a474e/src/adtypes.jl#L222-L236
> 
> https://github.com/SciML/OptimizationBase.jl/blob/2ffab7e93197c1fc8d9ed6a39857e301a71a474e/src/cache.jl#L45-L58
> 
> I think that this could be both optimized and simplified due to recent changes in DI.
> 
> Nowadays, \`DI.inner\` and \`DI.outer\` can also be called on backends which are not \`SecondOrder\`, they just act as the identity. Thus, you don't need to explicitly create a \`SecondOrder(adtype, adtype)\`. Passing \`adtype\` alone will be equivalent in most cases, and faster in some because it can leverage custom Hessian implementations within a single backend (e.g. \`SecondOrder(AutoForwardDiff(), AutoForwardDiff())\` cannot call \`ForwardDiff.hessian\` whereas \`AutoForwardDiff()\` can).
> Furthermore, DI's \`hvp\` and \`hessian\` for \`AutoZygote()\` already use ForwardDiff over Zygote.
> 
> Here are my suggestions:
> 
> \- Simplify the \`generate\_adtype\` logic and its variants to avoid creating \`SecondOrder\` objects altogether.
> \- Throw a warning based on the modes \`DI.inner\` and \`DI.outer\`, e.g. when the inner backend is not a reverse mode backend. This can be checked with \`ADTypes.mode(DI.inner(adtype)) isa Union{ADTypes.ReverseMode,ADTypes.ForwardOrReverseMode}\`. Of course you also want to allow ForwardDiff so feel free to refine.
> \- Document this behavior so that users are less confused by the warnings (see \[this Discourse thread\](https://discourse.julialang.org/t/warning-secondorder-adtype-in-optimization-jl/122539/2)).
> 
> What do you think @Vaibhavdixit02?