Flux.jacobian(::Chain, ::AbstractArray) gives the full jacobian of a neural network. Flux.hessian however throws the error “output is not scalar”. Let m = Chain(Dense(5,3,elu), Dense(3,3)) . What I can do is compute the gradient of each value in the jacobian matrix w.r. to each input using Flux.gra…

One way to compute the Hessian that usually works pretty efficiently is to apply the reverse-on-forward trick, as outlined in 1502.05767.pdf 573.54 KB If one replaces the floating point type with a dual number type, such as http://www.juliadiff.org/ForwardD…

Reverse mode is going to give columns, and I don’t think you need that. Using double forward mode will be the fastest here. You’ll need to mapchildren to remove the tracker information (or use the Flux#zygote branch) and then just forward diff (or use a hyperdual) If you do want to Forward-over-Rev…

Thank you both for your answers. And thank you for this nice paper, I understand AD much better now. I wanted to attempt both RoF and double forward methods. But I encounter a stack overflow issue when I use ForwardDiff methods on a neural network. Only the jacobian method takes an AbstractArray as …

That will stack overflow :man_shrugging: an issue in flux (pun intended)

I opened an issue then. By the way, indeed double forward mode is exactly what I need. The first pass will compute [dy_1/dx_1 … dy_m/dx_1] and the derivative with respect to x_1 will give the dx_1^2. So I’ll go with this mode once I figure out how to use that with a ::Chain. Reverse-on-forward woul…

It’s hardcoded in Flux to not work (and stackoverflow… it should throw a better error), but kind of for a good reason. We are working for a solution to this right now for other reasons, so stay tuned. We may start chatting about it in #autodiff

Did you have any luck @HenriDeh ? I’m facing similar issues. [image] [Flux] Issue while computing Hessian MethodError: objects of type Tracker.Grads are not callable Machine Learning I’m new to Flux and I’m trying to compute the hessian. Here’s the code: grad = Flux.gr…

Hi, no unfortunately I haven’t heard of any changes regarding this. I know that Flux is getting a complete revamp with zygote and other stuff so this might not be their priority at the moment. It could also fix this in the process, I don’t know. I have put this aspect of my project on hold in the me…

[image] HenriDeh: Flux.jacobian(::Chain, ::AbstractArray) gives the full jacobian of a neural network. Flux.hessian however throws the error “output is not scalar”. I had this same issue and resolved it using the sum function: using Flux m = Chain(Dense(2, 1)) x = zeros(2) # Flux.hessian(m,…

Is there an efficient way to compute the Hessian of a NN?

Specific Domains Machine Learning

HenriDeh August 5, 2019, 2:26pm 8

Ha okay ! Great, thank you I’ll keep track of the progress then !

Topic		Replies	Views
Hessian matrix of a neural network vetor output Specific Domains physics , zygote , neural-network	3	531	July 16, 2022
Apply hessian of neural network to array Machine Learning	2	466	March 20, 2021
Batched gradients and hessians with Flux Machine Learning question	10	476	January 25, 2025
Errors when trying to compute hessian of flux neural net and jacobian of jacobian with zygote Machine Learning flux , zygote	1	425	July 4, 2022
[Flux] Issue while computing Hessian MethodError: objects of type Tracker.Grads are not callable Machine Learning flux	12	1552	August 28, 2019

Is there an efficient way to compute the Hessian of a NN?

Related topics