No, because the derivative is a differential equation of a smooth function of p and df/dp, so if f is smooth then so its that derivative.
What is referred to is that adaptive ODE solvers’ solution is essentially noise below the tolerance limit. So if you solve at a tolerance of 1e-8, then the solution can jump around by 1e-8 even for small changes in the parameters. This is because it does not guarantee any accuracy below that, and a change of a parameter can cause a step rejection which then gives a different solving process. However, that only is a problem if you’re using finite difference gradients: with continuous sensitivities the sensitivity equation is itself continuous and its calculation can be done with error controls.