Enzyme computing the Hessian(?)

Specific Domains Optimization (Mathematical)
1

Hi there!

It seems like the (fantastic!) package Enzyme supports the computation of the Hessian times a vector but not the Hessian itself. Is this just a matter of time? or is the Hessian too expensive as to implement it?

Thanks!

1 Like
2

You can compute a hessian by computing

[
hessian_vector_product([1, 0, …]),
hessian_vector_product([0, 1, …]),
]

If you think this is useful to add as syntactic sugar for folks, please open a PR/issue!

3 Likes
3

True, awesome, thanks a lot!

1 Like
4

Do you have reason to believe your Hessian is sparse?

5

I don’t have a reason to believe this. Implementing the iteration over unit basis suggested by William I indeed get a non-sparse Hessian.

Why are you asking though?

6

Putting an @compile from Reactant.jl on the outside of the Enzyme hessian code can potentially result in a speedup for sparse code (doing propagation of one hot vectors). It also could automatically reduce allocations and add GPU/TPU support. But also if you have a problem where those aren’t the case, it may not help.

1 Like
7

Because in sparse settings there are faster, non-trivial ways to group the columns of the Hessian. But in dense settings, what Billy suggested is essentially optimal, up to batching (propagating several one-hot vectors at once with BatchDuplicated).

1 Like