Forwarddiff parametric gradient?

Hey,

Suppose I have a function that takes two vector parameters :

function f(p,q) # takes two vectors of same size
    return sum(p .* q) # return only one value
end

I want to obtain the gradient of f with respect to the first parameter as a function of the second. I did :

using ForwardDiff
g(p,q) = ForwarDiff.gradient(p -> f(p,q),p)

But it seems like the formal differentiation will append each time I call the function g. Is there a way to have it append only once, keeping q as a formal parameter during the differentiation ?

Not a very general solution to your problem, but there is Base.Fix1 and Base.Fix2 if you only need to fix either the first or second argument.

g(p,q) = ForwardDiff.gradient(Base.Fix2(f, q), p)
1 Like

And I suppose it would be pretty trivial to write an implementation of this for fixing a parameter at a different position than the first or second argument.

But that still recomputes the derivation at each call to g ? I was hoping for the forwarddiff AD to append conditionally on a parameter, but maybe it is not possible ?

It won’t have to recompile, if that’s what you mean. That’s the benefit of using Fix_. But I think these days the compiler can deal with inner closures without having to recompile every time as well (maybe someone can correct me if I’m wrong here).

ForwardDiff doesn’t do any sort of symbolic differentiation, so there’s no work it has to recompute there. It’s just passing values through your function. So the main thing is getting rid of compilation at each call, which Fix_ will do.

Okay, thanks for the explanation. I’ll try with Symbolics.jl if it can do it.

Nope, it should only get compiled once (as long as you don’t change the types of the values).

1 Like

It doesn’t recompile on each call even if you use anonymous function.

The only reason for the Fix2 type is so that specialized algorithms can be employed for certain curried functions. e.g. if you are doing findfirst(==(UInt8(3)), a) with a byte array a, because ==(UInt8(3)) turns into a Fix2{==} data structure the findfirst algorithm can specialize and call memchr.

2 Likes

Technically it does do symbolic differentiation, it’s just that the symbolic differentiation happens deep within the compiler. ForwardDiff works by evaluating your function using a type of dual number, for which the standard arithmetic rules essentially turn into the chain rule of differentation. So, when the function is compiled and type-specialized for dual numbers, the compiler is actually forming the symbolic derivative in the compiled code.

But this compilation process happens only once for given argument types—you don’t have to worry that the derivative will be symbolically recomputed each time you pass a different argument. That’s still true even if you form anonymous functions in the body of your function g.

2 Likes

For example, consider f(x,y) = x^2 * y, and suppose we want to differentiate with respect to x. The symbolic answer is 2x*y, of course. We can do this with ForwardDiff and anonymous functions:

import ForwardDiff
f(x, y) = x^2 * y
df(x,y) = ForwardDiff.derivative(x -> f(x, y), x)

If you inspect the resulting compiled code, you’ll see that df(x, y) just computes 2x*y for any input y:

julia> df(3.0,4.0)
24.0

julia> @code_llvm debuginfo=:none df(3.0,4.0)
define double @julia_df_967(double %0, double %1) {
top:
  %2 = fmul double %0, 2.000000e+00
  %3 = fmul double %2, %1
  ret double %3
}

So, the symbolic derivative is right there in the compiled code, which is re-used for any arguments of the same type — no recompilation or re-differentiation!

5 Likes

Which is just perfect. Thanks again for pointing this out, i have a tendency to forget it…

1 Like

Marked as solution then?

1 Like