# Zygote.jl: How to get the gradient of sparse matrix

**URL:** <https://discourse.julialang.org/t/zygote-jl-how-to-get-the-gradient-of-sparse-matrix/59067>\
**Category:** General Usage\
**Tags:** question, package, differentiation, zygote, reversediff\
**Created:** [April 12, 2021, 3:03am UTC](https://discourse.julialang.org/t/zygote-jl-how-to-get-the-gradient-of-sparse-matrix/59067 "2021-04-12T03:03:39Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 13, 2023, 1:25pm UTC](https://discourse.julialang.org/t/zygote-jl-how-to-get-the-gradient-of-sparse-matrix/59067/6 "2023-06-13T13:25:24Z")

</div>

> [@mohamed82008](#):
>
> I suggest writing an rrule for `sparse` and contributing it to ChainRules.jl. It’s doable and pretty straightforward.

Even if you do this, I’m worried that Zygote (or Enzyme) will still try to construct a dense matrix for the primal-tangent input to the `rrule`’s pullback?

For example, consider something as simple as the scalar-valued function f(p) = x^T A(p) y, where A(p) constructs an \ell \times m sparse matrix from some parameters p \in \mathbb{R}^n, while x \in \mathbb{R}^\ell and y \in \mathbb{R}^m are (dense) constant vectors. The partial derivatives are \frac{\partial f}{\partial p\_k} = x^T \frac{\partial A}{\partial p\_k} y = \mathrm{tr}[yx^T \frac{\partial A}{\partial p\_k}] = (xy^T) \cdot \frac{\partial A}{\partial p\_k}, where \cdot is the [Frobenius inner product](https://en.wikipedia.org/wiki/Frobenius_inner_product). In a sparse situation where \frac{\partial A}{\partial p\_k} has only O(1) nonzero entries, then \frac{\partial f}{\partial p\_k} can be computed in O(1) operations and the whole \nabla\_p f can be computed in O(n) operations with O(n) storage (like the calculation of f(p) itself) 😃 — this could easily be implemented in an `rrule` for f(p). However, if you instead define an `rrule` for A(p) (or for the `sparse` constructor), then the input tangent vector to the A(p) pullback is the rank-1 matrix xy^T, and if Zygote stores this as a dense matrix it will require O(\ell m) storage and time ☹.

Can Zygote (or Enzyme) be easily taught to store a low-rank tangent like xy^T implicitly?

---

_[View the full topic](https://discourse.julialang.org/t/zygote-jl-how-to-get-the-gradient-of-sparse-matrix/59067)._