Odd warning and issues with optimization only when inside Optimization.jl. Zygote.hessian works fine

[gdalle]

@Hareruya

However, whenever I perform optimization I get this warning that I can’t seem to find the specific documentation as to what this means:

Warning: The selected optimization algorithm requires second order derivatives, but `AutoZygote` ADtype was provided. So a `SecondOrder` with `AutoZygote` for inner and `AutoForwardDiff` for outer will be created, for choosing another pair an explicit `SecondOrder` ADType is recommended.

I looked over the Optimization.jl page and didn’t quite understand this. “SecondOrder” returns no results in the page either.

This warning appears because Optimization.jl uses my package DifferentiationInterface.jl (DI) under the hood to perform AD. When you only provide a first-order backend like AutoZygote(), Optimization.jl makes an informed decision on what to use for second-order AD. It requires two nested AD calls, and these can be performed with two different backends combined into a struct called DifferentiationInterface.SecondOrder. While you could make both of these calls with the same backend, it is usually a better idea to combine a forward and a reverse mode backend. Thus, to get good performance by default, Optimization.jl computes hessians with SecondOrder(AutoForwardDiff(), AutoZygote()) even though you didn’t ask for it explicitly.^[1]

@Hareruya

This would not be much of a problem if I could consistently optimize for all cases, but another issue is that sometimes the code will not even resolve because I will get this error:

Error: Cannot determine ordering of Dual tags Nothing and ForwardDiff.Tag{DifferentiationInterfaceForwardDiffExt.ForwardDiffOverSomethingHVPWrapper{typeof(loss)}, Float64}

With the stacktrace being particularly unhelpful due to the line of code giving issue in the latter case does not affect the former.

This is more worrying, and it could be linked to the following DI issue:

Error in ForwardDiff tagging

Tagging is a bit tricky for second-order AD, and a minimum working example would help tremendously here. Can you at least share the stack trace and what your Dual overloads look like, even if it’s not a fully runnable code?

Ping @Vaibhavdixit02

---

1. Note that DI actually uses forward-over-reverse for Hessians with pure AutoZygote() as well, because that is what Zygote.hessian chooses to do. ↩︎