Gradient and directional second derivative with ForwardDiff

Specific Domains Probabilistic Programming
1

So I have a function f (minus the log-likelihood ℓ retrieved from Turing in my case) and I need two things:

1.) gradient (∇f)(x) in some point x

2.) and how it changes in a direction θ:
∂ₜ(t -> ∇f(x + t*θ)) in t = 0

So I figured out how to compute it in one run

    return function (y, x, θ)
        x_ = x + Dual{:hSrkahPmmC}(0.0, 1.0)*θ
        y_ = ForwardDiff.gradient(x->-ℓ(x), x_)
        y .= value.(y_)
        y, dot(θ, y_).partials[]
    end

but I got warned that this might create tag-order problems if ℓ uses ForwardDiff down the road - perhaps (I am not sure because the outer gradient wraps the inner custom tag. Any pointers for me?

2

I think you want to use:

ForwardDiff.derivative(t -> ForwardDiff.gradient(f, x + t * θ), 0)
1 Like
3

I also need the gradient, derivative will only give me the second derivative. I’d hope using a proper ForwardDiff.Tag(f, typeof(x)) instead of a random symbol may solve it.

4

Use the DiffResults API of ForwardDiff to get the gradient and its derivative simultaneously. Differentiation API · ForwardDiff!

2 Likes