I remember reading somewhere that the cost of automatic differentiation is usually less than 2 times the cost of evaluating the function itself.
I tried to benchmark a small code and the results that I got were a little confusing.
using BenchmarkTools using Zygote function simple(x) return sin(x) end function complicated(x) return sin(x)^2*exp(x)*x^-2 + cos(x)^3 end x = 1:0.01:10 @btime simple.($x); @btime simple'.($x); @btime complicated.($x); @btime complicated'.($x);
14.942 μs (1 allocation: 7.19 KiB) 21.299 μs (1 allocation: 7.19 KiB) 47.380 μs (1 allocation: 7.19 KiB) 672.962 μs (14417 allocations: 1006.73 KiB)
Why is the AD of simple function so much faster than that of the complicated function?
Why are there so many allocations while taking the derivative of the complicated function?