The problem is probably that somewhere you are trying to cast something to Float64, e.g. this gives the same error,
ForwardDiff.derivative(x -> Float64(x)^2, 2)
For ForwardDiff to work you need to have your code written so any generic Real can pass through it. This might be as easy as just removing the type annotations / explicit type conversions, or maybe parameterizing your functions with T <: Real where needed.