I’m new to Zygote.jl and I’d like to compute the squared sum of a gradient, i.e., g = gradient(f, x), q = gradient(g’*g, x) Sucessfully did this using Forward mode: function grad_g_sqr(f, d, x) function gs(x) a = ForwardDiff.gradient(f, x) return a' * a / 2 end gg = F…

Hi and welcome to the community! Could you post a minimal working example so that it is easier to help you? [image] Please read: make it easier to help you Meta Discussion Welcome to the Julia Discourse! We are enthusiastic about helping Julia programmers, both beginner a…

Nested reverse AD with Zygote is very limited in terms of what it supports and (as I understand it) is almost never what you want. If you want to compute a hessian, check out Utilities · Zygote .

I don’t think th OP wants a hessian? (Correct me if I’m wrong!) I actually have the same situation quite often and fix it by writing wrappers for the chain rules. And then chainrules for the chainrules. Sorry this is a really short and confusing reply. I’ll try to put together a script later to de…

[image] cortner: Sorry this is a really short and confusing reply. I’ll try to put together a script later to demonstrate what I mean. That would be beautiful, because we receive quite a number of complaints regarding AD and it would simplify things to have a common line of defense;)

[image] cortner: I don’t think th OP wants a hessian? (Correct me if I’m wrong!) I was going off of function hessz(...), but mixing forward + reverse modes is just as applicable to any kind of nested AD.

Thanks gdalle, I finally wrapped up an example that is working. What I did: julia> ff(x) = sum((x .- 1).^2) ff (generic function with 1 method) julia> using Zygote julia> function hessz(f, x) gs(x) = sum(Zygote.gradient(f, x)[1] .^ 2 / 2) gg = Zygote.gradient(gs,…

[image] ToucheSir: [image] cortner: I don’t think th OP wants a hessian? (Correct me if I’m wrong!) I was going off of function hessz(...), but mixing forward + reverse modes is just as applicable to any kind of nested AD. Thanks for the comment. I was actually trying to avoid using fu…

[image] cortner: I don’t think th OP wants a hessian? (Correct me if I’m wrong!) I actually have the same situation quite often and fix it by writing wrappers for the chain rules. And then chainrules for the chainrules. Sorry this is a really short and confusing reply. I’ll try to put together…

I really think the whole discussion of gradient of gradient vs Jacobean and using mixed backward forward doesn’t apply here. To compute gradient of f we use of course backward. Then g'*g is again a scalar so we should of course use backward mode. The canonical situation which is why I’m interested …

Compute gradient of gradient norm using zygote

New to Julia

ToucheSir April 20, 2022, 11:48pm 10

Have a gander at Gradient of Gradient in Zygote - #3 by ChrisRackauckas.

Topic		Replies	Views
Newbie: Gradient of a gradient performance in Zygote General Usage zygote	2	536	March 21, 2021
Gradient of Gradient in Zygote General Usage	2	2791	January 1, 2021
Nested AD with Lux etc Machine Learning ad	26	1506	May 1, 2024
Differentiating Jacobian-vector product for sliced score matching? Machine Learning flux , zygote , ad	18	681	June 29, 2023
Mutation error in Zygote hessian Machine Learning question , package	2	219	June 22, 2023

Compute gradient of gradient norm using zygote

Related topics