# Forwarddiff parametric gradient?

**URL:** <https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302>\
**Category:** New to Julia\
**Tags:** forwarddiff\
**Created:** [November 30, 2021, 2:51pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302 "2021-11-30T14:51:01Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [November 30, 2021, 2:51pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/1 "2021-11-30T14:51:01Z")

</div>

Hey,

Suppose I have a function that takes two vector parameters :

```julia
function f(p,q) # takes two vectors of same size
    return sum(p .* q) # return only one value
end

```

I want to obtain the gradient of f with respect to the first parameter as a function of the second. I did :

```julia
using ForwardDiff
g(p,q) = ForwarDiff.gradient(p -> f(p,q),p)

```

But it seems like the formal differentiation will append each time I call the function `g`. Is there a way to have it append only once, keeping `q` as a formal parameter during the differentiation ?

---

<div class="post-metadata">

**Author:** ![jonniedie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonniedie/32/12842_2.png) [@jonniedie](https://discourse.julialang.org/u/jonniedie)\
**Post date:** [November 30, 2021, 3:21pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/2 "2021-11-30T15:21:47Z")

</div>

Not a very general solution to your problem, but there is `Base.Fix1` and `Base.Fix2` if you only need to fix either the first or second argument.

```julia
g(p,q) = ForwardDiff.gradient(Base.Fix2(f, q), p)

```

---

<div class="post-metadata">

**Author:** ![jonniedie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonniedie/32/12842_2.png) [@jonniedie](https://discourse.julialang.org/u/jonniedie)\
**Post date:** [November 30, 2021, 3:23pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/3 "2021-11-30T15:23:33Z")

</div>

And I suppose it would be pretty trivial to write an implementation of this for fixing a parameter at a different position than the first or second argument.

---

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [November 30, 2021, 3:25pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/4 "2021-11-30T15:25:41Z")

</div>

But that still recomputes the derivation at each call to g ? I was hoping for the forwarddiff AD to append conditionally on a parameter, but maybe it is not possible ?

---

<div class="post-metadata">

**Author:** ![jonniedie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonniedie/32/12842_2.png) [@jonniedie](https://discourse.julialang.org/u/jonniedie)\
**Post date:** [November 30, 2021, 3:32pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/5 "2021-11-30T15:32:42Z")

</div>

It won’t have to recompile, if that’s what you mean. That’s the benefit of using `Fix_`. But I think these days the compiler can deal with inner closures without having to recompile every time as well (maybe someone can correct me if I’m wrong here).

---

<div class="post-metadata">

**Author:** ![jonniedie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonniedie/32/12842_2.png) [@jonniedie](https://discourse.julialang.org/u/jonniedie)\
**Post date:** [November 30, 2021, 3:34pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/6 "2021-11-30T15:34:30Z")

</div>

ForwardDiff doesn’t do any sort of symbolic differentiation, so there’s no work it has to recompute there. It’s just passing values through your function. So the main thing is getting rid of compilation at each call, which `Fix_` will do.

---

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [November 30, 2021, 3:41pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/7 "2021-11-30T15:41:46Z")

</div>

Okay, thanks for the explanation. I’ll try with Symbolics.jl if it can do it.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [November 30, 2021, 5:13pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/8 "2021-11-30T17:13:08Z")

</div>

> [@lrnv](#):
>
> But it seems like the formal differentiation will append each time I call the function `g` . Is there a way to have it append only once, keeping `q` as a formal parameter during the differentiation ?

Nope, it should only get compiled once (as long as you don’t change the _types_ of the values).

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [November 30, 2021, 5:15pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/9 "2021-11-30T17:15:54Z")

</div>

> [@jonniedie](#):
>
> So the main thing is getting rid of compilation at each call, which `Fix_` will do.

It doesn’t recompile on each call even if you use anonymous function.

The _only_ reason for the `Fix2` type is so that specialized algorithms can be employed for certain curried functions. e.g. if you are doing `findfirst(==(UInt8(3)), a)` with a byte array `a`, because `==(UInt8(3))` turns into a `Fix2{==}` data structure the `findfirst` algorithm can specialize and call `memchr`.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [November 30, 2021, 5:20pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/10 "2021-11-30T17:20:06Z")

</div>

> [@jonniedie](#):
>
> ForwardDiff doesn’t do any sort of symbolic differentiation

Technically it _does_ do symbolic differentiation, it’s just that the symbolic differentiation happens deep within the compiler. ForwardDiff works by evaluating your function using a type of [dual number](https://en.wikipedia.org/wiki/Dual_number), for which the standard arithmetic rules essentially turn into the chain rule of differentation. So, when the function is compiled and type-specialized for dual numbers, the compiler is actually forming the symbolic derivative in the compiled code.

But this compilation process happens only once for given argument types—you don’t have to worry that the derivative will be symbolically recomputed each time you pass a different argument. That’s _still_ true even if you form anonymous functions in the body of your function `g`.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [November 30, 2021, 5:27pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/11 "2021-11-30T17:27:17Z")

</div>

For example, consider `f(x,y) = x^2 * y`, and suppose we want to differentiate with respect to `x`. The symbolic answer is `2x*y`, of course. We can do this with ForwardDiff and anonymous functions:

```julia
import ForwardDiff
f(x, y) = x^2 * y
df(x,y) = ForwardDiff.derivative(x -> f(x, y), x)

```

If you inspect the resulting compiled code, you’ll see that `df(x, y)` just computes `2x*y` for any input `y`:

```julia
julia> df(3.0,4.0)
24.0

julia> @code_llvm debuginfo=:none df(3.0,4.0)
define double @julia_df_967(double %0, double %1) {
top:
  %2 = fmul double %0, 2.000000e+00
  %3 = fmul double %2, %1
  ret double %3
}

```

So, the symbolic derivative is right there in the compiled code, which is re-used for any arguments of the same type — no recompilation or re-differentiation!

---

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [November 30, 2021, 5:59pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/12 "2021-11-30T17:59:25Z")

</div>

Which is just perfect. Thanks again for pointing this out, i have a tendency to forget it…

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [November 30, 2021, 6:02pm UTC](https://discourse.julialang.org/t/forwarddiff-parametric-gradient/72302/13 "2021-11-30T18:02:30Z")

</div>

Marked as solution then?