There was a discussion recently about (I think) a similar problem in the context of stack traces: DifferentialEquations Package Kills Performance Everywhere when TruncatedStacktraces is used
I’m not knowledgeable enough to determine if that applies here, bit maybe you find something useful in the discussion, some solution approaches are discussed near the end of the thread.