# ForwardDiff and Zygote return wrong jacobian for log(det(L))

**URL:** <https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961>\
**Category:** Numerics\
**Tags:** autodiff\
**Created:** [March 16, 2022, 11:53am UTC](https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961 "2022-03-16T11:53:15Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![spinkney](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@spinkney](https://discourse.julialang.org/u/spinkney)\
**Post date:** [March 16, 2022, 11:53am UTC](https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961/1 "2022-03-16T11:53:15Z")

</div>

I’m no expert in matrix derivatives so I may have made a mistake. The derivative of the log determinant of a Cholesky factor is

\frac{\partial \log|L|}{\partial L} = L^{-T}

using a toy example in Julia

```julia
using Zygote
S = [1.0 0.8; 0.8 1.0]
L = cholesky(S).L

reshape(jacobian(L -> log(det(L)), L)[1], 2, 2)
2×2 Matrix{Float64}:
 1.0 0.0
 0.0 1.66667

inv(L')
julia> inv(L')
2×2 UpperTriangular{Float64, Matrix{Float64}}:
 1.0 -1.33333
  ⋅ 1.66667

```

is this a known issue or is there a work around?

---

<div class="post-metadata">

**Author:** ![YingboMa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yingboma/32/2181_2.png) [@YingboMa](https://discourse.julialang.org/u/YingboMa)\
**Post date:** [March 17, 2022, 1:06am UTC](https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961/2 "2022-03-17T01:06:35Z")

</div>

I think Zygote is correct. The determinant of a lower triangular matrix is simply the product of its diagonal elements.

```julia
julia> ForwardDiff.gradient(L -> log(prod(diag(L))), L)
2×2 LowerTriangular{Float64, Matrix{Float64}}:
 1.0 ⋅
 0.0 1.66667

```

confirms the correctness of Zygote.

---

<div class="post-metadata">

**Author:** ![spinkney](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@spinkney](https://discourse.julialang.org/u/spinkney)\
**Post date:** [March 17, 2022, 1:26am UTC](https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961/3 "2022-03-17T01:26:30Z")

</div>

See [Derivative of log determinant of triangular matrix - Mathematics Stack Exchange](https://math.stackexchange.com/questions/1344046/derivative-of-log-determinant-of-triangular-matrix)  
 ![Screen Shot 2022-03-16 at 9.25.06 PM](https://global.discourse-cdn.com/julialang/original/3X/5/6/56eeca5a5f6be39aef7c3753b5c50cb30f2027c3.png)  
and the site [matrixcalculus.org](http://matrixcalculus.org) has

 ![Screen Shot 2022-03-16 at 9.25.39 PM](https://global.discourse-cdn.com/julialang/original/3X/e/c/ecce74db18cb8f9318634194bb6b1bd1b1c5e103.png)

which is why the result from Zygote is puzzling to me

---

<div class="post-metadata">

**Author:** ![YingboMa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yingboma/32/2181_2.png) [@YingboMa](https://discourse.julialang.org/u/YingboMa)\
**Post date:** [March 17, 2022, 2:48am UTC](https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961/4 "2022-03-17T02:48:41Z")

</div>

That’s because the L factor from `cholesky` is a lower triangular matrix. If you want to perturb the upper triangular elements as well, you can convert the factor to a `Matrix` first.

```julia
julia> reshape(Zygote.jacobian(L -> log(det(L)), Matrix(L))[1], 2, 2)
2×2 Matrix{Float64}:
 1.0 -1.33333
 0.0 1.66667

```

---

<div class="post-metadata">

**Author:** ![spinkney](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@spinkney](https://discourse.julialang.org/u/spinkney)\
**Post date:** [March 17, 2022, 10:02am UTC](https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961/5 "2022-03-17T10:02:26Z")

</div>

Rad! It’s cool that the derivative is constrained on the space of lower triangular matrices and entirely obvious now! Thanks!

---

<div class="post-metadata">

**Author:** ![Bernard\_GODARD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernard_godard/32/4155_2.png) [@Bernard\_GODARD](https://discourse.julialang.org/u/Bernard_GODARD)\
**Post date:** [March 18, 2022, 8:09pm UTC](https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961/6 "2022-03-18T20:09:01Z")

</div>

Does someone know why this does not work in ForwardDiff but only in Zygote?

```julia
julia> ForwardDiff.gradient(L -> log(det(L)), Matrix(L))

2×2 Matrix{Float64}:
 1.0 0.0
 0.0 1.66667

julia> reshape(Zygote.jacobian(L -> log(det(L)), Matrix(L))[1], 2, 2)

2×2 Matrix{Float64}:
 1.0 -1.33333
 0.0 1.66667

```

---

<div class="post-metadata">

**Author:** ![YingboMa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yingboma/32/2181_2.png) [@YingboMa](https://discourse.julialang.org/u/YingboMa)\
**Post date:** [March 18, 2022, 10:48pm UTC](https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961/7 "2022-03-18T22:48:28Z")

</div>

Because ForwardDiff has no special overloads for array valued functions, and LinearAlgebra’s `det` has the optimization of first checking if the matrix is triangular before computing the LU factorization. Hence, the L factor is going to only go through the triangular branch. As a workaround, you can do

```julia
julia> ForwardDiff.gradient(L -> log(det(lu(L))), Matrix(L))
2×2 Matrix{Float64}:
 1.0 -1.33333
 0.0 1.66667

```

or simply

```julia
julia> ForwardDiff.gradient(L -> logdet(L), Matrix(L))
2×2 Matrix{Float64}:
 1.0 -1.33333
 0.0 1.66667

```

because `logdet` doesn’t have the optimization (though, this is an implementation detail that should not be relied on).

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [March 18, 2022, 10:49pm UTC](https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961/8 "2022-03-18T22:49:04Z")

</div>

> [@Bernard\_GODARD](#):
>
> Does someone know why this does not work in ForwardDiff

With [#481](https://github.com/JuliaDiff/ForwardDiff.jl/pull/481) it works:

```julia
julia> ForwardDiff.gradient(L -> log(det(L)), Matrix(L))
2×2 Matrix{Float64}:
 1.0 -1.33333
 0.0 1.66667

```

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 18, 2022, 10:55pm UTC](https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961/9 "2022-03-18T22:55:11Z")

</div>

> [@mcabbott](#):
>
> With [#481](https://github.com/JuliaDiff/ForwardDiff.jl/pull/481) it works:

Hm, seems like quite a significant PR without reviewers?

Edit: the highest ranked [search result](https://samritchie.io/dual-numbers-and-automatic-differentiation/)

```julia
(defn eq
  "For non-differentials, this is identical to [[clojure.core/=]].
  For [[Differential]] instances, equality acts on tangent components too.

  If you want to ignore the tangent components, use [[equiv]]."
  ([_] true)
  ([a b]
   (= (->terms a)
      (->terms b)))
  ([a b & more]
   (reduce eq (eq a b) more)))

```

seems to agree with you. And BTW: the imaginary part in the PR 's title references the tangents/gradients (I first thought the PR would be about complex numbers)?

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [March 22, 2022, 6:51pm UTC](https://discourse.julialang.org/t/forwarddiff-and-zygote-return-wrong-jacobian-for-log-det-l/77961/10 "2022-03-22T18:51:43Z")

</div>

The PR is a bit stuck because every time I try to fix the tests I get lost – there are many levels of nested testsets trying to ensure that nothing ever changes, but some must change for this.

Sorry I guess “imaginary” is misleading. The essay about what to do is [this issue](https://github.com/JuliaDiff/ForwardDiff.jl/issues/480). When I searched I got the impression that everyone flipped a coin to decide which rule to adopt here when implementing dual numbers. But I didn’t find this particular one. (Edit – because it was written after my issue.)