# ForwardDiff with Matrix of Vector as input argument

**URL:** <https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049>\
**Category:** Specific Domains\
**Tags:** question\
**Created:** [October 8, 2024, 6:46am UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049 "2024-10-08T06:46:19Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Domenico\_Lahaye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/domenico_lahaye/32/203728_2.png) [@Domenico\_Lahaye](https://discourse.julialang.org/u/Domenico_Lahaye)\
**Post date:** [October 8, 2024, 6:46am UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/1 "2024-10-08T06:46:19Z")

</div>

Greetings. Consider the small function

```julia
function foo(v)
    return [v; v] 
end 

```

Then the following works

```julia
v0 = [1;1]
out = ForwardDiff.jacobian(v -> foo(v), v0)

```

The following works as well

```julia
A0 = [1 2; 3 4]
out = ForwardDiff.jacobian(A -> foo(A), A0)type or paste code here

```

The following, however, returns in a method error

```julia
T0 = [[1.,1.] for i=1:2, j=1:2]
out = ForwardDiff.jacobian(T -> foo(T), T0)

```

How can I extend the method definitions to treat the latter case?

Thx!

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 8, 2024, 7:57am UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/2 "2024-10-08T07:57:50Z")

</div>

Hi @Domenico_Lahaye! ForwardDiff expects the input and output of your function to be an array of numbers. You could use `stack` to make sure of that, but you have to be careful with the shapes and what they mean.

---

<div class="post-metadata">

**Author:** ![Domenico\_Lahaye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/domenico_lahaye/32/203728_2.png) [@Domenico\_Lahaye](https://discourse.julialang.org/u/Domenico_Lahaye)\
**Post date:** [October 8, 2024, 8:15am UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/3 "2024-10-08T08:15:21Z")

</div>

Clear, thx.

Essentially, i am trying to solve a vector-valued PDE. I wish to form a Jacobian in which the elements are 2-by-2 block matrices.

I will check for far I get by stacking the input for forwardDiff and un-stacking its output.

More here later. Many thanks for now.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 8, 2024, 8:19am UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/4 "2024-10-08T08:19:17Z")

</div>

An alternative is ComponentArrays.jl, but honestly I would just recommend stacking everything into a 3-dimensional tensor. Note that the jacobian returned by ForwardDiff will always be a 2d matrix though, it essentially flattens the input and output into vectors.

---

<div class="post-metadata">

**Author:** ![Domenico\_Lahaye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/domenico_lahaye/32/203728_2.png) [@Domenico\_Lahaye](https://discourse.julialang.org/u/Domenico_Lahaye)\
**Post date:** [October 8, 2024, 9:37am UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/5 "2024-10-08T09:37:46Z")

</div>

Or merely do

```julia
T01 = [1 1; 1 1]
T02 = [1 1; 1 1]
out1 = ForwardDiff.jacobian(T -> foo(T), T01)
out2 = ForwardDiff.jacobian(T -> foo(T), T02)
[out1 out2]

```

Thank you so much!

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 8, 2024, 9:48am UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/6 "2024-10-08T09:48:27Z")

</div>

You can also just call

```julia
ForwardDiff.jacobian(foo, T01)

```

---

<div class="post-metadata">

**Author:** ![Domenico\_Lahaye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/domenico_lahaye/32/203728_2.png) [@Domenico\_Lahaye](https://discourse.julialang.org/u/Domenico_Lahaye)\
**Post date:** [October 8, 2024, 9:55am UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/7 "2024-10-08T09:55:20Z")

</div>

True. This, however, is no longer true in the application that I target as the function foo() has more than one input argument.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 8, 2024, 9:59am UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/8 "2024-10-08T09:59:41Z")

</div>

Final remarks: if performance is a bottleneck for you, you may want to look into:

- StaticArrays.jl if your input is small
- preallocation of [config](https://juliadiff.org/ForwardDiff.jl/stable/user/api/#Preallocating/Configuring-Work-Buffers)

---

<div class="post-metadata">

**Author:** ![Domenico\_Lahaye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/domenico_lahaye/32/203728_2.png) [@Domenico\_Lahaye](https://discourse.julialang.org/u/Domenico_Lahaye)\
**Post date:** [October 8, 2024, 10:20am UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/9 "2024-10-08T10:20:55Z")

</div>

I do hate to disappoint you. Developments are too premature for remarks to final 😀

1/ The splitting T0 into T01 and T02 does away with StaticArrays.jl in the original approach. There might be a bring StaticArray.jl back in;

2/ Yes, definitely, I do need to look into config.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 8, 2024, 10:29am UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/10 "2024-10-08T10:29:57Z")

</div>

Note that when you use `SArrays` you don’t need configuration, `jacobian` as is works best.  
Maybe you don’t need to split and can create a single `SArray` of dimension 3?

---

<div class="post-metadata">

**Author:** ![Domenico\_Lahaye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/domenico_lahaye/32/203728_2.png) [@Domenico\_Lahaye](https://discourse.julialang.org/u/Domenico_Lahaye)\
**Post date:** [October 8, 2024, 4:52pm UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/11 "2024-10-08T16:52:20Z")

</div>

How do you make StaticArrays work with dual numbers?

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 8, 2024, 5:03pm UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/12 "2024-10-08T17:03:01Z")

</div>

They should work out of the box. Do you have a MWE?

---

<div class="post-metadata">

**Author:** ![Domenico\_Lahaye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/domenico_lahaye/32/203728_2.png) [@Domenico\_Lahaye](https://discourse.julialang.org/u/Domenico_Lahaye)\
**Post date:** [October 8, 2024, 5:16pm UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/13 "2024-10-08T17:16:57Z")

</div>

Good to know that it should work out of the box.

No MWE directly available. Will make one in case troubles persist.

Thx!

---

<div class="post-metadata">

**Author:** ![Domenico\_Lahaye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/domenico_lahaye/32/203728_2.png) [@Domenico\_Lahaye](https://discourse.julialang.org/u/Domenico_Lahaye)\
**Post date:** [October 8, 2024, 6:19pm UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/14 "2024-10-08T18:19:03Z")

</div>

Good news: I now succeed in differentiating the entire code!

Now debugging and optimization can start.

---

<div class="post-metadata">

**Author:** ![Domenico\_Lahaye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/domenico_lahaye/32/203728_2.png) [@Domenico\_Lahaye](https://discourse.julialang.org/u/Domenico_Lahaye)\
**Post date:** [October 9, 2024, 4:13pm UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/15 "2024-10-09T16:13:16Z")

</div>

I discovered that

```julia
function foo(x,y)
    SVector{3,Float64}(x,y,0) # observe type annotation 
end 
x0 = 1.
y0 = 2. 
ForwardDiff.derivative(x -> foo(x,y0),x0) 

```

results is a method errror and that

```julia
function foo(x,y)
    SVector{3}(x,y,0) # observe absence of type annotation 
end 
x0 = 1.
y0 = 2. 
ForwardDiff.derivative(x -> foo(x,y0),x0) 

```

works.

Is this known / generally accepted?

Did I misread the documentation?

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 9, 2024, 6:09pm UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/16 "2024-10-09T18:09:15Z")

</div>

> [@Domenico\_Lahaye](#):
>
> Is this known / generally accepted?

Yes it is known because your first function coerces the inputs into a `Float64` type, whereas ForwardDiff needs more generality to function, see [Limitations of ForwardDiff · ForwardDiff](https://juliadiff.org/ForwardDiff.jl/stable/user/limitations/). You need to allow arbitrary real types to flow through your function, including any containers you generate.

---

<div class="post-metadata">

**Author:** ![Domenico\_Lahaye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/domenico_lahaye/32/203728_2.png) [@Domenico\_Lahaye](https://discourse.julialang.org/u/Domenico_Lahaye)\
**Post date:** [October 9, 2024, 6:15pm UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/17 "2024-10-09T18:15:03Z")

</div>

Thx!

Can you please point me to examples using `ForwardDiff.JacobianConfig()` ?

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 9, 2024, 7:17pm UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/18 "2024-10-09T19:17:28Z")

</div>

I don’t have any in mind apart from the [API reference](https://juliadiff.org/ForwardDiff.jl/stable/user/api/#Preallocating/Configuring-Work-Buffers), but if you’re working solely with `SArray`s then configuration should be unnecessary. Do you have a complete MWE that I could maybe help you optimize?

---

<div class="post-metadata">

**Author:** ![Domenico\_Lahaye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/domenico_lahaye/32/203728_2.png) [@Domenico\_Lahaye](https://discourse.julialang.org/u/Domenico_Lahaye)\
**Post date:** [October 9, 2024, 9:12pm UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/19 "2024-10-09T21:12:04Z")

</div>

Thx.

StaticArrays are used in lower levels of the code only.

Here is a MWE.

Given two functions with deliberately dense Jacobians

```julia
function foo(x)
    return sum(x)*ones(length(x))
end 

function gnu(x)
    return sum(x.*x)*ones(length(x))
end

```

and given three input vectors of deliberately larger dimension

```julia
x1 = ones(10)
x2 = ones(100)
x3 = ones(1000)

```

the following two questions arrise:

**Q1: How do avoid increasing number of allocations with increasing problem size?**

Here is what I currently see:

```julia
@btime ForwardDiff.jacobian(x -> foo(x), x1);
840.726 ns (6 allocations: 4.58 KiB)

@btime ForwardDiff.jacobian(x -> foo(x), x2);
26.125 μs (23 allocations: 191.45 KiB)

@btime ForwardDiff.jacobian(x -> foo(x), x3);
  2.445 ms (258 allocations: 16.72 MiB)

```

**Q2: How do reuse buffer crease for Jacobian for the function foo() in order to compute the Jacobian for the function gnu()?**

Is my understand correct that the buffer can be shared between the function foo() and gnu()?

Thx!

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 9, 2024, 9:45pm UTC](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049/20 "2024-10-09T21:45:42Z")

</div>

> [@Domenico\_Lahaye](#):
>
> **Q1: How do avoid increasing number of allocations with increasing problem size?**

You can’t, because your functions both allocate new vectors. ForwardDiff computes Jacobians by running the function once per input dimension\[1\], so the number of allocations will necessarily scale with the input.

Still, you can greatly increase efficiency by:

1. Writing your functions in-place: `f!(y, x)` instead of `y = f(x)`
2. Pre-allocating a Jacobian matrix `J`
3. Pre-allocating a [`JacobianConfig`](https://juliadiff.org/ForwardDiff.jl/stable/user/api/#ForwardDiff.JacobianConfig) and pass it to [`jacobian!`](https://juliadiff.org/ForwardDiff.jl/stable/user/api/#ForwardDiff.jacobian!) along with `J`

The idea is that steps 2 and 3 only need to be done once, and then you can compute many derivatives reusing the same memory.

> [@Domenico\_Lahaye](#):
>
> **Q2: How do reuse buffer crease for Jacobian for the function foo() in order to compute the Jacobian for the function gnu()?**

You can’t, because ForwardDiff uses a custom tag for each function to identify where the `Dual` numbers came from. You can [disable it](https://juliadiff.org/ForwardDiff.jl/stable/user/advanced/#Custom-tags-and-tag-checking) but I would advise against it.

* * *

1. Not entirely true, the chunking mechanism means you can actually divide the input dimension by the [chunk size](https://juliadiff.org/ForwardDiff.jl/stable/user/advanced/#Configuring-Chunk-Size).

[Next page](https://discourse.julialang.org/t/forwarddiff-with-matrix-of-vector-as-input-argument/121049.md?page=2)