This is in dire need of a MWE. It’s most likely that some code is inside the function being optimized is using ForwardDiff without your knowledge. This may be intentional (on the library author’s part) or due to a misunderstanding of the higher-level libraries.