Hello,
The term v^\top \nabla_x f(x) v can be rewritten as a directional derivative \frac{\mathrm{d}}{\mathrm{d} \alpha}(v^\top f(x+\alpha v))\Big|_{\alpha = 0}: we only need the derivative of the scalar function \alpha \mapsto v^\top f(x + \alpha v) for \alpha = 0.
I don’t know how to implement it properly with Zygote though.