# Higher derivatives using ForwardDiff

**URL:** <https://discourse.julialang.org/t/higher-derivatives-using-forwarddiff/28814>\
**Category:** Numerics\
**Created:** [September 16, 2019, 3:42pm UTC](https://discourse.julialang.org/t/higher-derivatives-using-forwarddiff/28814 "2019-09-16T15:42:29Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![RJDennis](https://avatars.discourse-cdn.com/v4/letter/r/90db22/32.png) [@RJDennis](https://discourse.julialang.org/u/RJDennis)\
**Post date:** [September 16, 2019, 3:42pm UTC](https://discourse.julialang.org/t/higher-derivatives-using-forwarddiff/28814/1 "2019-09-16T15:42:29Z")

</div>

Hi All,

If I have a function, say,

`test(x) = x[1]^3*log(x[2])*sqrt(x[3])`

then what is the best way to use ForwardDiff to compute higher-order partial derivatives of this function?

I tried the following  
`first_d(x) = ForwardDiff.gradient(test,x)`  
`second_d(x) = ForwardDiff.hessian(test,x)`  
`third_d(x) = ForwardDiff.jacobian(second_d,x)`  
`p = [1.5, 2.0, 3.0]`  
`first_d(p)`  
`second_d(p)`  
`third_d(p)`

but this general approach seems very slow for functions containing a few more variables.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 16, 2019, 4:03pm UTC](https://discourse.julialang.org/t/higher-derivatives-using-forwarddiff/28814/2 "2019-09-16T16:03:34Z")

</div>

Can you quantify “very slow”? Compared to what? Just to give a number:

```julia
julia> using BenchmarkTools

julia> @btime third_d($p)
  2.704 μs (11 allocations: 4.97 KiB)

```

---

<div class="post-metadata">

**Author:** ![dpsanders](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dpsanders/32/3573_2.png) [@dpsanders](https://discourse.julialang.org/u/dpsanders)\
**Post date:** [September 16, 2019, 4:45pm UTC](https://discourse.julialang.org/t/higher-derivatives-using-forwarddiff/28814/3 "2019-09-16T16:45:52Z")

</div>

For higher derivatives you can try TaylorSeries.jl.

---

<div class="post-metadata">

**Author:** ![RJDennis](https://avatars.discourse-cdn.com/v4/letter/r/90db22/32.png) [@RJDennis](https://discourse.julialang.org/u/RJDennis)\
**Post date:** [September 16, 2019, 9:36pm UTC](https://discourse.julialang.org/t/higher-derivatives-using-forwarddiff/28814/4 "2019-09-16T21:36:25Z")

</div>

Hi Kristoffer,

so the problem that I am having with speed relates to the compliation time, actually. Once compiled my function only takes a few thousanths of a second to run, but it takes well-over a minute to compile. Unfortunately, the function is not one that gets run over and over again so the first time it runs is the time that matters. I guess I’m willing to sacrifice some run time in order to get a shorter compilation time.

---

<div class="post-metadata">

**Author:** ![RJDennis](https://avatars.discourse-cdn.com/v4/letter/r/90db22/32.png) [@RJDennis](https://discourse.julialang.org/u/RJDennis)\
**Post date:** [September 16, 2019, 9:39pm UTC](https://discourse.julialang.org/t/higher-derivatives-using-forwarddiff/28814/5 "2019-09-16T21:39:12Z")

</div>

Thanks. I had a look at TaylorSeries earlier today before posting; I’ll take another look.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [September 16, 2019, 9:39pm UTC](https://discourse.julialang.org/t/higher-derivatives-using-forwarddiff/28814/6 "2019-09-16T21:39:39Z")

</div>

> [@RJDennis](#):
>
> I guess I’m willing to sacrifice some run time in order to get a shorter compilation time.

Depending on the amount of runtime performance you’re willing to lose, you might want to try interpreting.

```julia
julia> using JuliaInterpreter

julia> @btime @interpret third_d($p);
  47.894 ms (149692 allocations: 5.71 MiB)

```

The first `@interpret` call in this place is still a bit slow, but not as bad as your compilation time for nested ForwardDiff. Here’s a fresh julia session:

```julia
julia> begin
       using JuliaInterpreter, ForwardDiff
       test(x) = x[1]^3*log(x[2])*sqrt(x[3])
       first_d(x) = ForwardDiff.gradient(test,x)
       second_d(x) = ForwardDiff.hessian(test,x)
       third_d(x) = ForwardDiff.jacobian(second_d,x)
       p = [1.5, 2.0, 3.0]
       end;

julia> @time @interpret third_d(p)
  4.670857 seconds (10.34 M allocations: 509.138 MiB, 3.84% gc time)

julia> @time @interpret third_d(p);
  0.049506 seconds (149.71 k allocations: 5.712 MiB)

```

The other nice thing about this approach is that you re-use the compilation machinery from function call to function call so you pay the compile price the first time you interpret something but no more later on.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 17, 2019, 6:52am UTC](https://discourse.julialang.org/t/higher-derivatives-using-forwarddiff/28814/7 "2019-09-17T06:52:05Z")

</div>

You can play around with the chunk size parameter. A lower chunk size should lower compilation time at some cost to execution time.

[http://www.juliadiff.org/ForwardDiff.jl/stable/user/advanced.html#Configuring-Chunk-Size-1](http://www.juliadiff.org/ForwardDiff.jl/stable/user/advanced.html#Configuring-Chunk-Size-1)

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [September 17, 2019, 7:32am UTC](https://discourse.julialang.org/t/higher-derivatives-using-forwarddiff/28814/8 "2019-09-17T07:32:03Z")

</div>

A chunk size of one seems to work really well:

```julia
julia> using ForwardDiff

julia> test(x) = x[1]^3*log(x[2])*sqrt(x[3])
test (generic function with 1 method)

julia> first_d(x) = ForwardDiff.gradient(test,x,ForwardDiff.GradientConfig(test,x,ForwardDiff.Chunk{1}()))
first_d (generic function with 1 method)

julia> second_d(x) = ForwardDiff.hessian(test,x,ForwardDiff.HessianConfig(test,x,ForwardDiff.Chunk{1}()))
second_d (generic function with 1 method)

julia> third_d(x) = ForwardDiff.jacobian(second_d,x,ForwardDiff.JacobianConfig(second_d,x,ForwardDiff.Chunk{1}()))
third_d (generic function with 1 method)

julia> p = [1.5, 2.0, 3.0]
3-element Array{Float64,1}:
 1.5
 2.0
 3.0

julia> @time third_d(p)
  1.531497 seconds (1.70 M allocations: 85.207 MiB, 7.16% gc time)
9×3 Array{Float64,2}:
  7.2034 7.79423 1.80085  
  7.79423 -2.92284 0.974279 
  1.80085 0.974279 -0.225106 
  7.79423 -2.92284 0.974279 
 -2.92284 1.46142 -0.24357  
  0.974279 -0.24357 -0.0811899
  1.80085 0.974279 -0.225106 
  0.974279 -0.24357 -0.0811899
 -0.225106 -0.0811899 0.0562765

julia> @time third_d(p)
  0.000016 seconds (37 allocations: 4.500 KiB)
9×3 Array{Float64,2}:
  7.2034 7.79423 1.80085  
  7.79423 -2.92284 0.974279 
  1.80085 0.974279 -0.225106 
  7.79423 -2.92284 0.974279 
 -2.92284 1.46142 -0.24357  
  0.974279 -0.24357 -0.0811899
  1.80085 0.974279 -0.225106 
  0.974279 -0.24357 -0.0811899
 -0.225106 -0.0811899 0.0562765

julia> using BenchmarkTools

julia> @btime third_d($p)
  4.362 μs (33 allocations: 4.34 KiB)
9×3 Array{Float64,2}:
  7.2034 7.79423 1.80085  
  7.79423 -2.92284 0.974279 
  1.80085 0.974279 -0.225106 
  7.79423 -2.92284 0.974279 
 -2.92284 1.46142 -0.24357  
  0.974279 -0.24357 -0.0811899
  1.80085 0.974279 -0.225106 
  0.974279 -0.24357 -0.0811899
 -0.225106 -0.0811899 0.0562765

julia> versioninfo()
Julia Version 1.4.0-DEV.106
Commit 6a20ad7eaa* (2019-09-08 06:58 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i3-4010U CPU @ 1.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.0 (ORCJIT, haswell)
Environment:
  JULIA_NUM_THREADS = 4

```

---

<div class="post-metadata">

**Author:** ![RJDennis](https://avatars.discourse-cdn.com/v4/letter/r/90db22/32.png) [@RJDennis](https://discourse.julialang.org/u/RJDennis)\
**Post date:** [September 17, 2019, 10:46am UTC](https://discourse.julialang.org/t/higher-derivatives-using-forwarddiff/28814/9 "2019-09-17T10:46:30Z")

</div>

Thanks for the suggestion (and you too Elrod). Making the chuck size equal to 1 does lower the compliation time significantly.