# Native eigenvals for differentiable programming

**URL:** <https://discourse.julialang.org/t/native-eigenvals-for-differentiable-programming/27126>\
**Category:** Machine Learning\
**Created:** [August 3, 2019, 9:06am UTC](https://discourse.julialang.org/t/native-eigenvals-for-differentiable-programming/27126 "2019-08-03T09:06:09Z")\
**Posts on this page:** 1\
**Showing post:** 9

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [August 3, 2019, 12:24pm UTC](https://discourse.julialang.org/t/native-eigenvals-for-differentiable-programming/27126/9 "2019-08-03T12:24:29Z")

</div>

> [@GJBoth](#):
>
> I’m working on a problem where I require an autodifferentiable eigenvalue solver for real, square matrices (but non-hermitian).

You could always implement the derivative yourself. The adjoint formula for differentiating eigenvalues with respect to changes in the matrix is pretty simple; see [https://github.com/mitmath/18335/blob/spring21/notes/adjoint/adjoint.pdf](https://github.com/mitmath/18335/blob/spring21/notes/adjoint/adjoint.pdf) (it’s equivalent to the Hellmann–Feynman theorem); differentiating scalar functions of the eigenvectors is a bit more complicated but not too bad. (My notes show the case of Hermitian matrices; the extension to non-Hermitian problems is similar but involves left and right eigenvectors — IIRC, every xᵀ is replaced with the left eigenvector.)

One snag is that eigenvalues cease to be differentiable when two eigenvalues cross. (And eigenvector methods of non-Hermitian problems can become ill-conditioned near crossings.) Crossings may seem unlikely, but eigenvalue-optimization problems often push eigenvalues towards crossings (especially in Hermitian problems). There is a way around it using generalized gradients, as I review in my notes here: [https://github.com/mitmath/18335/blob/spring21/notes/adjoint/eigenvalue-adjoint.pdf…](https://github.com/mitmath/18335/blob/spring21/notes/adjoint/eigenvalue-adjoint.pdf%E2%80%A6) but usually in my own work I’ve found that’s it’s possible to reformulate ostensibly eigenvalue-optimization problems in ways that avoid explicit eigenvalue computations. For example, [in terms of SDPs](http://arxiv.org/abs/arXiv:1405.4350) or in terms of [resolvent operators](http://doi.org/10.1364/OE.21.030812).

> [@carstenbauer](#):
>
> Perhaps ArnoldiMethods.jl Iterative Solvers.jl and KrylovKit.jl might work for you. (all implemented in Julia.

You _really_ don’t want to apply automatic differentiation blindly to an iterative solver. Differentiating a recurrence relationship like this is _much_ more expensive than differentiating the final result.

For the same reason, a Julia-native eigensolver wouldn’t help here. Applying AD to an eigenvalue algorithm like QR is a non-starter. (All eigenvalue algorithms for matrices bigger than 4×4 are iterative, thanks to the [Abel–Ruffini theorem](https://en.wikipedia.org/wiki/Abel%E2%80%93Ruffini_theorem).)

Automatic differentiation is a great tool, but at some point you should also learn how to compute derivatives yourself, if only to understand how to use these tools effectively. Moreover, once you get to a complicated enough problem that involves external libraries, AD tools quickly hit a wall — I’ve never been able to use AD for a non-trivial problem in PDE-constrained optimization.

---

_[View the full topic](https://discourse.julialang.org/t/native-eigenvals-for-differentiable-programming/27126)._