I’m having some trouble getting Zygote over ForwardDiff.derivative to work. I’m going to refer to this prior post . The following code used to fail with ERROR: setindex! not defined for ForwardDiff.Partials{1,Float64}. The fix was to define some additional adjoints, as @ChrisRackauckas showed in Dif…

The example above fails in the way I describe. gs should have the gradients wrt to params(f), but has nothing instead.

My example is more along the lines of this, computing the dot product between the gradient and a vector, which is equivalent to the directional derivative. using Flux using ForwardDiff net = Chain(Dense(2, 128, relu), Dense(128, 128, relu), Dense(128, 1)) p, re = Flux.destructure(net) x = randn(F…

If you add the DiffEqFlux adjoints does it work?

No, that doesn’t make a difference.

Interesting. @mcabbott would you know something about what might’ve changed?

Yes this won’t work, sadly. The warning from Zygote.forwarddiff is: Note that the function `f` will *drop gradients* for any closed-over values. and that’s what’s being used here . That is, it’s forward-over-forward, and takes derivatives only with respect to the explicit parameter, not to anything…

[image] mcabbott: Does DiffEqFlux.jl have (pirate?) code which handles this? Yes. ZygoteRules.@adjoint function ForwardDiff.Dual{T}(x, ẋ::Tuple) where T @assert length(ẋ) == 1 ForwardDiff.Dual{T}(x, ẋ), ḋ -> (ḋ.partials[1], (ḋ.value,)) end ZygoteRules.@adjoint ZygoteRules.literal_getpro…

Is there any possibility of a workaround? This used to work ~1 year ago. The only other approach I have been able to make work is ReverseDiff over Zygote, but for some reason this is super slow (I’ll create another thread about this).

Issue with Zygote over ForwardDiff.derivative

Specific Domains Machine Learning

facusapienza January 21, 2024, 4:18pm 11

Has it been any update or progress in this line? I recently encountered a similar problem that I posted in Nested and different AD methods altogether: How to add AD calculations inside my loss function when using neural differential equations? that I am trying to make work. Thanks!

Topic		Replies	Views
Flux loss: Gradient wrt input leads to empty gradient wrt parameters or to "can't differentiate foreigncall" Machine Learning flux , forwarddiff , diffeqflux	3	605	April 8, 2022
Is it possible perform reverse mode differentiation (Flux.jl with Zygote.jl) of a forward mode differentiation result (e.g. ForwardDiff)? Machine Learning question , flux	3	1481	March 10, 2020
Using Flux: gradient on DifferentialEquations: solve results in an error New to Julia question , diffeq , differentiation	11	1057	May 8, 2023
Gradient error in Flux model inputs Machine Learning question , flux , zygote	5	1367	January 13, 2021
ForwardDiffSensitivity() and TrackerAdjoint() give different answers Machine Learning diffeq , sde	3	823	July 18, 2020

Issue with Zygote over ForwardDiff.derivative

Related topics