Hi all, I’m working on a problem where I require an autodifferentiable eigenvalue solver for real, square matrices (but non-hermitian). Since Linear algebra’s version is based on LAPACK, I can’t use autodiff and I haven’t been able to make any other versions work (e.g. using GenericLinearAlgebra.jl…

[image] GJBoth: I’m working on a problem where I require an autodifferentiable eigenvalue solver for real, square matrices (but non-hermitian). You could always implement the derivative yourself. The adjoint formula for differentiating eigenvalues with respect to changes in the matrix is pre…

I’m pretty new to auto diff; how exactly would this work?

You can see how it’s done for some common deep learning functions here https://github.com/FluxML/Zygote.jl/blob/a50f4cf88b1a9e74a94e5295cb468b94ae3f1d9e/src/lib/nnlib.jl#L4 A guide is provided here https://fluxml.ai/Zygote.jl/dev/adjoints/

Perhaps ArnoldiMethods.jl Iterative Solvers.jl and KrylovKit.jl might work for you. (all implemented in Julia.

Autograd used to have it before the upgrade to 1.0 made descending into tuples for backward mode ad difficult. I’ve implemented some hacks for svd myself, I’d have to look them up (essentially you’d have to unwrap the SVD return type). Anyway the backward mode code for svd is still in the autograd p…

I tried ArnoldiMethods but it seemed there was some sampling in there (not sure, I’m not versed in these numerical methods), making it impossible to do autograd… I’ll check the other things but I think making a custom adjoint might be easiest.

[image] stevengj: You really don’t want to apply automatic differentiation blindly to an iterative solver. In case anyone is interested, I tried doing exactly this with all the iterative solvers @carstenbauer mentioned, on Julia 1.1, 1.2rc and 1.3-alpha with Zygote on its release version and …

I got a very simple iterative eigensolver to AD with Zygote sometime ago, so it can be done. But indeed it doesn’t scale; much better is to write a custom adjoint that solves the perturbative equation iteratively.

Thanks for the detailed response. I’m optimizing over a bandstructure so my eigenvalues can/will be degenerate, so I’ll have to use generalized gradients. I’ll start implementing soon!

[image] antoine-levitt: I got a very simple iterative eigensolver to AD with Zygote sometime ago, so it can be done. But indeed it doesn’t scale; much better is to write a custom adjoint that solves the perturbative equation iteratively. Indeed, a big part of the design of ChainRules that I a…

Native eigenvals for differentiable programming

Specific Domains Machine Learning

GJBoth August 3, 2019, 12:06pm 8

Would be great if you could look them up, then I’ll try and start implementing something for eig. Any idea how hard it would be?

Topic		Replies	Views
Computing matrix derivatives for eigenvaule sensitivity analysis Numerics autodiff , enzyme	6	436	April 12, 2024
Derivative of eigenvalues and eigenvectors of Hermitian Matrix by automatic differentiation General Usage question	19	4719	June 11, 2018
Eigenvalue sensitivites using autodiff Numerics eigenvalues , autodiff	5	931	May 2, 2022
Automatic differentiation of f(x) constructed by decomposition of a matrix M(x) Optimization (Mathematical) differentiation , flux , machine-learning , numerics	7	2957	January 8, 2020
Best practices for optimization problems with ODE solves and eigen calls Performance question , optimization	12	249	November 21, 2025

Native eigenvals for differentiable programming

Related topics