It is hard to say more about this without an MWE (eg do you provide derivatives manually or use AD, what is rho [stepsize correction?], …).