While I have no real expertise with stiff solvers, I would argue that what you see is completely normal. Your dynamical system seems to be chaotic, non-autonomous, multistable, and metastable. This type of dynamical system will be very sensitive to the exact solver you use. In fact, even your operating system will have impact on the exact trajectories you will see. This isn’t a problem, this is numerical chaos.
What you want to make sure of is that dynamic invariants such as lyapunov exponents do not depend on the solver much (provided you use trajectories that go to the same attractor). If they do, you need to lower the tolerances more.
My hunch is further corroborated by what you stated: