# Forward mode Autodiff for Matrix multiplication

**URL:** <https://discourse.julialang.org/t/forward-mode-autodiff-for-matrix-multiplication/72032>\
**Category:** General Usage\
**Tags:** autodiff\
**Created:** [November 24, 2021, 9:04pm UTC](https://discourse.julialang.org/t/forward-mode-autodiff-for-matrix-multiplication/72032 "2021-11-24T21:04:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tgautam03](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tgautam03/32/20545_2.png) [@tgautam03](https://discourse.julialang.org/u/tgautam03)\
**Post date:** [November 24, 2021, 9:04pm UTC](https://discourse.julialang.org/t/forward-mode-autodiff-for-matrix-multiplication/72032/1 "2021-11-24T21:04:17Z")

</div>

Hi, I’m new to Julia and looking to learn more about Autodiff in julia. I’ve matrix multiplication and want to get the derivative of a matrix wrt to a matrix.

For example in the equations below, I want to compute \frac{\partial X\_4}{\partial X\_1}, where both X\_1 and X\_4 are matrices (W\_i are also matrices and f is some function applied to each element of the matrix) .

X\_1 = X\_0 \cdot W\_1  
X\_2 = f(X\_1)  
X\_3 = X\_2 \cdot W\_2  
X\_4 = f(X\_3)

This is very similar to NN, but here my **output is not a scalar**. I want to use forward mode autodiff for this however I can’t figure out how to do this using `ForwardDiff` package.

Apart from this, it would be great if someone can explain how chain rule would work in this case (I’m familiar with reverse mode autodiff where output is scalar but I can’t figure out the mathematics behind something similar but in forward mode).

I hope I’ve explained my problem clearly.

Thanks

---

<div class="post-metadata">

**Author:** ![Keno](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keno/32/285_2.png) [@Keno](https://discourse.julialang.org/u/Keno)\
**Post date:** [November 24, 2021, 9:45pm UTC](https://discourse.julialang.org/t/forward-mode-autodiff-for-matrix-multiplication/72032/2 "2021-11-24T21:45:17Z")

</div>

```julia
julia> function foo(X₀, W₁, W₂)
           X₁ = X₀ * W₁
           X₂ = sin.(X₁)
           X₃ = X₂ * W₂
           X₄ = sin.(X₃)
       end

julia> let W₁=randn(10, 10), W₂=randn(10, 10), X₀=randn(10, 10)
       ForwardDiff.jacobian(X₀->foo(X₀, W₁, W₂), X₀)
       end

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [November 24, 2021, 10:06pm UTC](https://discourse.julialang.org/t/forward-mode-autodiff-for-matrix-multiplication/72032/3 "2021-11-24T22:06:37Z")

</div>

> [@tgautam03](#):
>
> Apart from this, it would be great if someone can explain how chain rule would work in this case

I find it easiest to imagine adding a small perturbation \delta X\_0 to the input matrix, and carrying it through to first order:

X\_1 + \delta X\_1 = X\_0 W\_1 + \delta X\_0 W\_1 \\ X\_2 + \delta X\_2 = f(X\_1 + \delta X\_1) = f(X\_1) + f'(X\_1) \odot \delta X\_1 + O(\delta X\_0^2)\\ X\_3 + \delta X\_3 = X\_2 W\_2 + (f'(X\_1) \odot \delta X\_1) W\_2 + O(\delta X\_0^2) \\ X\_4 + \delta X\_4 = f(X\_3 + \delta X\_3) = f(X\_3) + f'(X\_3) \odot [(f'(X\_1) \odot (\delta X\_0 W\_1)) W\_2] + O(\delta X\_0^2)

where \odot denotes the elementwise (Hadamard) product.

Now, the question is, when you ask for the derivative (Jacobian) of the output, what do you want? If you want the directional derivative \delta X\_4 in a certain input direction \delta X\_0, you can directly read it off above. If you want the Jacobian “matrix”, in some sense it’s “really” a rank-4 tensor (4d array). ForwardDiff gives you the Jacobian assuming the input and output are “flattened” using [`vec`](https://en.wikipedia.org/wiki/Vectorization_(mathematics)). In either case you need to do a bit of multilinear algebra to work it out by hand.

---

<div class="post-metadata">

**Author:** ![tgautam03](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tgautam03/32/20545_2.png) [@tgautam03](https://discourse.julialang.org/u/tgautam03)\
**Post date:** [November 25, 2021, 12:44am UTC](https://discourse.julialang.org/t/forward-mode-autodiff-for-matrix-multiplication/72032/4 "2021-11-25T00:44:38Z")

</div>

In my problem, X\_0 is n \times 2 and X\_4 is n \times 1, so what I want is \frac{\partial X\_4(i)}{\partial X\_0(i,1)} and \frac{\partial X\_4(i)}{\partial X\_0(i,2)}. I believe both of these would be n \times 1 vectors (here `i=1:n`).