Types and gradients, including Forward.gradient

General Usage
20

If anybody is curious about the ReverseDiff vs. AutoGrad performance (just timing the gradients, not the training):

Set up code:

using BenchmarkTools

# these are constants (following your code)
const n = 500
const p = 1000
const x = randn(n, p)'
const y = sum(x[1:5,:],1) .+ randn(n)'*0.1

# these are the parameters
w = 0.0001*randn(1,p)
b = [0.0]
input = (w, b)

Autograd version:

julia> begin
           using AutoGrad
           # tried to do `sum(abs2.(...))` since `sumabs2` is getting 
           # deprecated, but AutoGrad threw an error so I switched back
           loss(input) = sumabs2(y - ((input[1] * x) .+ input[2][1])) / size(y, 2)
           loss∇ = grad(loss)
           @benchmark loss∇($input)
       end
BenchmarkTools.Trial:
  memory estimate:  43.11 kb
  allocs estimate:  200
  --------------
  minimum time:     1.801 ms (0.00% GC)
  median time:      1.836 ms (0.00% GC)
  mean time:        1.861 ms (0.49% GC)
  maximum time:     7.337 ms (69.84% GC)

ReverseDiff version:

julia> begin
           using ReverseDiff
           output = map(zeros, input)
           loss(w, b) = sum(abs2.(y - ((w * x) .+ b[1]))) / size(y, 2)
           loss∇! = ReverseDiff.compile_gradient(loss, input)
           @benchmark loss∇!($output, $input)
       end
BenchmarkTools.Trial:
  memory estimate:  48.00 bytes
  allocs estimate:  2
  --------------
  minimum time:     1.621 ms (0.00% GC)
  median time:      1.637 ms (0.00% GC)
  mean time:        1.648 ms (0.00% GC)
  maximum time:     2.617 ms (0.00% GC)