Hi, I’m quite new to Julia and trying to understand automatic differentiation. I have a loss function for a model, and I’m trying to get the gradient and hessian w.r.t parameters fitted to the model. In doing so, comparing Zygote, ForwardDiff and ReverseDiff, I noticed that the values for the gradi…

I mentioned last time that there was an issue with uniqueness of times in the way that your model was originally posed, and you made that same issue again. If you fix that it’s fine: using DifferentialEquations, Flux, LinearAlgebra, DiffEqFlux, DiffEqSensitivity using ForwardDiff, FiniteDiff using …

Normally, they should agree, so this can be a bug. An MWE replicating this would be helpful. [image] Please read: make it easier to help you Meta Discussion Welcome to the Julia Discourse! We are enthusiastic about helping Julia programmers, both beginner and experienced.…

To what tolerance are you doing your computation?

Hi, here is a mwe. It seems that zygote and ReverseDiff agree both on gradient and hessian, whereas ForwardDiff is different. using DifferentialEquations, Flux, LinearAlgebra, DiffEqFlux, DiffEqSensitivity using ForwardDiff using Zygote using ReverseDiff function bimolecular!(du,u,p,t,dens,cons) …

So would I need to save at each time point only once, and then in my loss function I would compare each saveat to multiple observables?

That would be best. I opened https://github.com/SciML/DiffEqSensitivity.jl/issues/335 to make this safer but would recommend just dropping via unique before saveat.

So, I have finally implemented the loss function with unique time points, but now the zygote hessian does not work anymore. I have a function that takes in the solution, the experimental time points (with repeats), and the unique time points, and finds the differences at each experimental time poin…

[image] vaglino: The issue appears to be in the statement Δy[i] = Δyᵢ as the Δy is a Float, whereas the Δyᵢ is a ForwardDiff.Dual type, since Σ_sol is a ForwardDiff.Dual type as well. @dhairyagandhi96 is Zygote.Buffer not using similar to match the eltype?

It indeed uses similar but does not enforce the eltype by default. It follows the same API as similar, so you could pass in T as Buffer(xs, T, sz) if needed.

[image] dhairyagandhi96: It follows the same API as similar , so you could pass in T as Buffer(xs, T, sz) if needed. But the API of similar is to use the eltype of the array by default?

Let me clarify, we use similar and splat the arguments after the array. So yes, it will check eltype via similar, but does not default to Buffer(xs, T). You should see the correct eltype, as you would with similar.

Different results between Zygote, ForwardDiff, and ReverseDiff

New to Julia

Tamas_Papp September 13, 2020, 12:48pm 2

Normally, they should agree, so this can be a bug. An MWE replicating this would be helpful.

Topic		Replies	Views
Zero gradients with Zygote vs correct gradients with ReverseDiff using DiffEqFlux Machine Learning zygote , reversediff , diffeqflux	4	1460	January 24, 2022
Zygote dozens* of times slower than manually written function Performance zygote , forwarddiff	17	1923	April 21, 2022
Compute gradient of gradient norm using zygote New to Julia zygote	17	2182	August 26, 2022
Ambiguous method bug in ReverseDiff General Usage	6	228	May 9, 2023
Wrong gradient from Zygote? Numerics question , zygote	13	828	November 14, 2023

Different results between Zygote, ForwardDiff, and ReverseDiff

Related topics