Technically it does do symbolic differentiation, it’s just that the symbolic differentiation happens deep within the compiler. ForwardDiff works by evaluating your function using a type of dual number, for which the standard arithmetic rules essentially turn into the chain rule of differentation. So, when the function is compiled and type-specialized for dual numbers, the compiler is actually forming the symbolic derivative in the compiled code.
But this compilation process happens only once for given argument types—you don’t have to worry that the derivative will be symbolically recomputed each time you pass a different argument. That’s still true even if you form anonymous functions in the body of your function g.