# ForwardDiff and GradientConfig memory usage

**URL:** <https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145>\
**Category:** Performance\
**Created:** [April 3, 2018, 11:55pm UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145 "2018-04-03T23:55:47Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![gideonsimpson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gideonsimpson/32/1928_2.png) [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Post date:** [April 3, 2018, 11:55pm UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145/1 "2018-04-03T23:55:47Z")

</div>

I noticed the following issue when trying to minimize my memory allocation while using `ForwardDiff.gradient!` . My example looks like:

```julia
using ForwardDiff
function run0(x, f, n)
       out = similar(x);
       cfg = ForwardDiff.GradientConfig(f, x);
       for i = 1:n
              ForwardDiff.gradient!(out, f, x, cfg);
       end
end

function V(x)

       result = (dot(x,x)-1)^2;

       return result
end

```

If I now compute:

```julia
julia> x100 = rand(100);
julia> @time run0(x100, V, 10);
  0.586337 seconds (350.17 k allocations: 19.781 MiB, 1.96% gc time)
julia> @time run0(x100, V, 10);
  0.000659 seconds (7 allocations: 10.656 KiB)

julia> @time run0(x100, V, 100);
  0.005968 seconds (7 allocations: 10.656 KiB)

julia> @time run0(x100, V, 1000);
  0.026512 seconds (7 allocations: 10.656 KiB)

```

and you see that the memory usage is insensitive to the number of iterations.

If, however, I use a small vector,

```julia
julia> x5=rand(5);

julia> @time run0(x5, V, 10);
  0.116469 seconds (68.13 k allocations: 3.728 MiB)

julia> @time run0(x5, V, 10);
  0.000157 seconds (17 allocations: 1.453 KiB)

julia> @time run0(x5, V, 20);
  0.000167 seconds (27 allocations: 2.078 KiB)

julia> @time run0(x5, V, 100);
  0.000385 seconds (107 allocations: 7.078 KiB)

julia> @time run0(x5, V, 1000);
  0.002894 seconds (1.01 k allocations: 63.328 KiB)

```

I see growth in memory usage with the number of iterations.

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [April 4, 2018, 7:11pm UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145/2 "2018-04-04T19:11:39Z")

</div>

bumping this - would love to know as well what is going on here.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 4, 2018, 8:04pm UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145/3 "2018-04-04T20:04:46Z")

</div>

Ok, so funnily enough this is exactly the problem we discussed before regarding not parameterizing on function arguments.

[https://github.com/JuliaDiff/ForwardDiff.jl/pull/315](https://github.com/JuliaDiff/ForwardDiff.jl/pull/315) removes the allocations (and makes it a bit faster).

```julia
julia> x5=rand(5);

julia> @time run0(x5, V, 10);
  0.000094 seconds (19 allocations: 1.703 KiB)

julia> @time run0(x5, V, 10);
  0.000047 seconds (7 allocations: 848 bytes)

julia> @time run0(x5, V, 20);
  0.000039 seconds (7 allocations: 848 bytes)

julia> @time run0(x5, V, 100);
  0.000104 seconds (7 allocations: 848 bytes)

```

---

<div class="post-metadata">

**Author:** ![gideonsimpson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gideonsimpson/32/1928_2.png) [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Post date:** [April 4, 2018, 9:41pm UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145/4 "2018-04-04T21:41:35Z")

</div>

So what do I need to do (I’m still a bit of a novice)? I tried `Pkg.checkout("ForwardDiff")` and then running with that, but I’m still having the same issues with my code.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 4, 2018, 9:43pm UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145/5 "2018-04-04T21:43:48Z")

</div>

You need to check out the name of that branch since it isn’t merged. `Pkg.checkout("ForwardDiff", "kc/f_spec")` probably.

---

<div class="post-metadata">

**Author:** ![gideonsimpson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gideonsimpson/32/1928_2.png) [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Post date:** [April 4, 2018, 9:48pm UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145/6 "2018-04-04T21:48:37Z")

</div>

Got it, thanks.

---

<div class="post-metadata">

**Author:** ![gideonsimpson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gideonsimpson/32/1928_2.png) [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Post date:** [April 5, 2018, 1:35am UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145/7 "2018-04-05T01:35:17Z")

</div>

I’m noticing (possibly) related issues with the following function:

```julia
function V(x)
       result = 0.0;
       for i in 1:length(x)
              result+= (x[i]^2-1)^2
       end
       return result
end

```

The setup is otherwise the same, though I am now using your suggested pull. But using this `V(x)`, I get linear growth in allocations. _However_, if I swap `result=0.0` for `result=zero(eltype(x))`, everything is well behaved again. Is this just an example of a need for explicit typing, or is there something else at work here?

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [April 5, 2018, 2:13am UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145/8 "2018-04-05T02:13:39Z")

</div>

Yes, this is just a classic type-instability, because the eltype of `x` is a `ForwardDiff.Dual`, so in your code `result` changes from a `Float64` to a `ForwardDiff.Dual`. It’s the same situation as [https://docs.julialang.org/en/stable/manual/performance-tips/#Avoid-changing-the-type-of-a-variable-1](https://docs.julialang.org/en/stable/manual/performance-tips/#Avoid-changing-the-type-of-a-variable-1) but with a different set of types.

---

<div class="post-metadata">

**Author:** ![gideonsimpson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gideonsimpson/32/1928_2.png) [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Post date:** [April 5, 2018, 4:33pm UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145/9 "2018-04-05T16:33:54Z")

</div>

I am a bit surprised that `0.0` is inadequate for ensuring that it was interpreted as a floating point number. Regardless, is the use of `eltype` in this example the stylistically favored solution?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 5, 2018, 4:36pm UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145/10 "2018-04-05T16:36:04Z")

</div>

0.0 is a `Float64`. That is different from a `Dual` number. Using `eltype` works well or

```julia
function V(x::Vector{T}) where {T}
    result = zero(T)
    ....
end

```

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [April 5, 2018, 4:46pm UTC](https://discourse.julialang.org/t/forwarddiff-and-gradientconfig-memory-usage/10145/11 "2018-04-05T16:46:39Z")

</div>

0.0 _is_ sufficient to ensure that `result` starts out as a Float64. The problem is that when you use `ForwardDiff`, your function is called not with a `Float64` argument but with a special `ForwardDiff.Dual` number. Try printing `eltype(x)` inside your function when you compute its gradient to see that.

That’s why the recommendation is to use `zero(eltype(x))` which will just do the right thing (Float64 for Float64, Dual for Dual, etc.) at no additional cost.