# Why is my ForwardDiff derivative caluculation 5x as slow as just evaluating the function, when allocations are just 2x?

**URL:** <https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555>\
**Category:** Numerics\
**Tags:** differentiation\
**Created:** [August 25, 2017, 11:12am UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555 "2017-08-25T11:12:39Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![vgdev](https://avatars.discourse-cdn.com/v4/letter/v/47e85d/32.png) [@vgdev](https://discourse.julialang.org/u/vgdev)\
**Post date:** [August 25, 2017, 11:12am UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555/1 "2017-08-25T11:12:39Z")

</div>

So, I am trying to understand what impacts the speed of calculating the derivative of a function using ForwardDiff. I am evaluating the mean of several simulated paths of a stochastic variable. However, evaluating the derivative of the function calculating the mean with respect to some parameter a is 5x as slow as just evaluating the mean function it self, but from @btime I see that memory allocation is just ~2x. Thus, I am wondering what causes is to be 5x as slow? (I have read elsewhere that I should only expect about x2 slowdown).

```julia
a = -0.43;
@btime mean = computing_volterra_mean(a)
g(x) = ForwardDiff.derivative(computing_volterra_mean, x)
@btime deriv_mean = g(a)

```

> 112.625 ms (48002 allocations: 36.90 MiB)

> 552.525 ms (75011 allocations: 64.84 MiB)

Code:

```julia
using ForwardDiff
using DualNumbers
using BenchmarkTools
using DiffBase

> function b(a, k)
> return (k^(a+1.0)-(k-1.0)^(a+1.0))/(a+1.0);
> end
> 
> function get_sigma_Z1(n)
> return 1.0/sqrt(n);
> end
> 
> function get_sigma_Z2(a,n)
> return 1.0/sqrt(((2.0*a+1.0)*n^(2.0*a+1.0)))
> end
> 
> function get_rho(a)
> return sqrt(2.0*a+1.0)/(a+1.0)
> end
> 
> function gen_G(a,n,s)
> G =zeros(eltype(a),s+1);
> for i =2:s+1
> G[i] = b(a,Float64(i))/(n^a)
> end
> return G;
> end
> 
> function direct_convolution(x, y, s, a)
> c = zeros(eltype(a),2*s)
> @inbounds @simd for j=1:s
> @inbounds @simd for k=1:(s+1)
> c[j+k-1] = c[j+k-1]+ x[j]*y[k]
> end
> end
> return c
> end
> 
> function gen_volterra(a,s,n,G,rand_1,rand_2,rho, sigma_Z1,sigma_Z2)
> Z_1 = zeros(eltype(a),s); #1st normal
> Z_2 = zeros(eltype(a),s); # 2nd normal
> 
> @inbounds @simd for i = 1:s
> Z_1[i] = sigma_Z1*rand_1[i] #set correct variance of random variate
> Z_2[i] = sigma_Z2*(rho*rand_1[i]+sqrt(1.0-rho^2.0)*rand_2[i]); #set correct variance of random variate
> end
> return [0;(direct_convolution(Z_1,G,s,a)[1:s]+Z_2)];
> end
> 
> function computing_volterra_mean(a)
> num_paths =1000;
> n =500;
> s =500;
> 
> G = gen_G(a,n,s); #Generate kernel
> rho = get_rho(a); # precomute rho, and stddevs
> sigma_Z1 =get_sigma_Z1(n);
> sigma_Z2 = get_sigma_Z2(a,n)
> result =zero(a);
> 
> @inbounds @simd for i =1:num_paths
> rand1 =randn(s);
> rand2 =randn(s);
> result+=gen_volterra(a,s,n,G,rand1,rand2,rho,sigma_Z1,sigma_Z2)[s+1]
> end
> 
> return result/num_paths;
> end

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 25, 2017, 12:04pm UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555/2 "2017-08-25T12:04:02Z")

</div>

Try profiling, especially with `ProfileView.jl`, paying attention to red segments (type instability).

---

<div class="post-metadata">

**Author:** ![vgdev](https://avatars.discourse-cdn.com/v4/letter/v/47e85d/32.png) [@vgdev](https://discourse.julialang.org/u/vgdev)\
**Post date:** [August 25, 2017, 1:51pm UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555/3 "2017-08-25T13:51:18Z")

</div>

Thanks for the tip @Tamas_Papp. Not aware of the `ProfileView.jl` package. As such I am not experienced in reading these profile charts, but as far as I have understood ‘red’ bars means type instability. Below are the profiles of the function evaluation and the derivative calculation. Its not much difference, but maybe you see something I dont?

Profile of `computing_volterra_mean(a)`:

 ![function_profile](https://global.discourse-cdn.com/julialang/original/3X/7/0/70d3e29242ad5b8c1abf62b5e199d92fbf420332.png)

Profile of `deriv_mean = g(a)`:

 ![derivative_profile](https://global.discourse-cdn.com/julialang/original/3X/4/b/4bd696bcd9af36ecd2a52394cfbeae9ae1b02375.png)

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [August 25, 2017, 1:54pm UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555/4 "2017-08-25T13:54:02Z")

</div>

You won’t get SIMD with dual numbers unless you start Julia with `-O3`. [Advanced Usage Guide · ForwardDiff](http://www.juliadiff.org/ForwardDiff.jl/latest/user/advanced.html#SIMD-Vectorization-1)

Depending on if your bottle neck is actually in the tight loops or not, it might help.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 25, 2017, 2:07pm UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555/5 "2017-08-25T14:07:16Z")

</div>

Now you have to work through your code to make sure it is type stable and optimize bottlenecks. `gen_volterra`, for example, may not be type stable because you are concatenating a `0` (not a `zero(...)`). Type stability is usually necessary, but not sufficient for optimal performance, see the performance tips in the manual.

---

<div class="post-metadata">

**Author:** ![vgdev](https://avatars.discourse-cdn.com/v4/letter/v/47e85d/32.png) [@vgdev](https://discourse.julialang.org/u/vgdev)\
**Post date:** [August 25, 2017, 2:07pm UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555/6 "2017-08-25T14:07:57Z")

</div>

I am using Atom, but I have set the Optimisation level to 3 already. Is this equal to staring Julia with `-O3` ?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [August 25, 2017, 2:14pm UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555/7 "2017-08-25T14:14:24Z")

</div>

Most likely.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [August 25, 2017, 2:17pm UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555/8 "2017-08-25T14:17:15Z")

</div>

Concatenations of numbers and arrays promote to a common element type. The allocations going to dual numbers has only allocated by a factor of 2 which seems reasonable since a dual number occupies twice the size.

Why are you assuming that there is a type instability?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [August 25, 2017, 2:27pm UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555/9 "2017-08-25T14:27:43Z")

</div>

Looks like `direct_convolution` is the bottle neck and is just ~4x slower for Dual numbers.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 25, 2017, 2:29pm UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555/10 "2017-08-25T14:29:21Z")

</div>

Mistake on my part.

---

<div class="post-metadata">

**Author:** ![vgdev](https://avatars.discourse-cdn.com/v4/letter/v/47e85d/32.png) [@vgdev](https://discourse.julialang.org/u/vgdev)\
**Post date:** [August 25, 2017, 3:50pm UTC](https://discourse.julialang.org/t/why-is-my-forwarddiff-derivative-caluculation-5x-as-slow-as-just-evaluating-the-function-when-allocations-are-just-2x/5555/11 "2017-08-25T15:50:16Z")

</div>

Why would it be x4 slower with dual numbers and not x2?