I see, yes most SciML caches should be a DI.ConstantOrCache as there’s certain algorithm specific constants that shouldn’t be zeroed (I presume this has to do with make_zero! behavior).
Open an issue on LinearSolve and I can dig into it (though it’s the post-JuliaCon rush so it might be a bit delayed). But I think it’s at least pretty clear there that you’re building an init with a non-dual value and so there’s no way you’re going to forwarddiff that because linsolve.A and linsolve.b aren’t dual carriers. The easiest solution is likely to just use a PreallocationTools.LazyBufferCache, and hopefully that’s a good enough hint to get it done though if not if you open an issue I should answer by the end of the week with something more detailed.