Automatic differentiation for higher-order derivatives

General Usage
1

Hi,
I would like to get some results from 0th to nth derivatives evaluation of a given function.
For example,
my_func(x::Real) returns (f(x), f'(x), f''(x), ..., f^{n}(x)).
Note that DiffResults.jl with ForwardDiff.jl may be what I want but it seems only give f(x), f(x), and H(x).

Any ideas?

3 Likes
2

I do that here

1 Like
3

Thanks :slight_smile:
BTW, is it efficient? I mean, DiffResults.jl claims that the calculation of multiple orders of automatic differentiation would be redundant.

4

BTW, is it efficient

In what terms?

I use it for GPU based applications, it works well.

5

Higher order AD is a complicated subject, but the current best guidance I would have is to use ForwardDiff for scalar functions up to fourth order or so, and TaylorSeries.jl beyond that. For non-scalar code we don’t currently have a great answer, but Diffractor.jl will provide this functionality once released.

7 Likes
6

May I ask what are your time estimates regarding diffractor? I am currently calculating 2nd order gradients with Zygote and the first gradient takes some time, more than 10 minutes.

7

Soon. I’ve sent it to some people for early comments, but it isn’t quite ready yet to try widely and I want to avoid people being disappointed. If your code is public, I’d be happy to give it a try with Diffractor just to see if it works or not (and if not, put it on my list of things to fix).

10 Likes
8 
function ndiff(f,x,n)
    tag = ForwardDiff.Dual{nothing}(x,one(x))
    for _ in 1:n
        tag = ForwardDiff.Dual{nothing}(tag,one(tag))
    end
    rawres = f(tag)
    return rawres
end

basically rawres should have all the derivative information, but im looking on how to extract it. diffresults stores this dual and makes it easier to obtain the proper information

9

Inspired by you, I wrote a code:

    if auto_diff
        _funcs = Function[x_cmd_func]
        for i in 1:d+1
            push!(_funcs, (t) -> ForwardDiff.derivative(_funcs[i], t))
        end
    end
    funcs(t) = [_func(t) for _func in _funcs]
10

Hi,

you may also take a look in TaylorSeries.jl. The n-th coefficient of the Taylor series expansion of f(x+a) is the derivative f^{(n)}(a)/n!, which I think is what you are looking for.

In code, you do the following:

julia> using TaylorSeries

julia> t = Taylor1(5)  # "independent" variable of order (degree) 5
 1.0 t + 𝒪(t⁶)

julia> sin(t)  # Taylor series around 0 of `sin`
 1.0 t - 0.16666666666666666 t³ + 0.008333333333333333 t⁵ + 𝒪(t⁶)

julia> sin(1.0+t)   # Taylor series around `1.0 of `sin`
 0.8414709848078965 + 0.5403023058681398 t - 0.42073549240394825 t² - 0.09005038431135663 t³ + 0.03506129103366235 t⁴ + 0.004502519215567832 t⁵ + 𝒪(t⁶)
2 Likes
11

Hi Keno,

thanks for the offer. Unfortunately, the code is not yet public, as it is part of my research work. I can try to carve out some MWE and send it to you as an example. Would it work for you?

Tomas

12

Sure, if it’s easy to do. Don’t spend to much time on it though.