Automatic Differentiation Slow (Slower than Finite-Differences)

Specific Domains Optimization (Mathematical)
14

Yeah, there have been a bunch of changes in reinterpret. This works on both 0.6 and master:

using Compat
using ForwardDiff
const LinearAlgebra = Compat.LinearAlgebra

function MTv(M::Matrix{Float64}, v::Vector{ForwardDiff.Dual{T,V,N}}) where {T, V <: LinearAlgebra.BlasFloat, N}
    num_v = length(v)
    vmat = transpose(reshape(reinterpret(V, v), N + 1, num_v)) # first column: values, other columns: partials
    result = transpose(M * vmat) # first row: values, other rows: partials
    reinterpret(ForwardDiff.Dual{T, V, N}, reshape(result, length(result)))
end

using Compat.Test

@testset "MTv" begin
    m = 3
    n = 4
    p = 5

    M = rand(m, n)
    v = [ForwardDiff.Dual(rand(), ForwardDiff.Partials(tuple(rand(p)...))) for _ = 1 : n]
    ret1 = MTv(M, v)
    ret2 = M * v
    @test ret1 ≈ ret2
end

nparams = 10;
v_len = 100_000;
out_len =30;
v = ForwardDiff.Dual.(randn(v_len),[tuple(rand(nparams)...) for dd=1:v_len]);
M=rand(out_len, v_len);

using BenchmarkTools
@btime MTv($M, $v);
@btime $M * $v;

Results on latest master:

  6.503 ms (6 allocations: 2.88 KiB)
  14.319 ms (1 allocation: 2.75 KiB)

Results on 0.6.2:

  9.500 ms (12 allocations: 8.40 MiB)
  17.520 ms (1 allocation: 2.75 KiB)

Edit: I noticed that the difference is much smaller (but still significant) with -O3.

1 Like