No, I was suggesting that you consider whether you can use the Cholesky factors without explicitly computing the inverse. That is, given C = cholesky!(X'X + Diagonal(d)), you can solve a linear system for any given right-hand-side quickly, so in many cases you don’t need the inverse matrix explicitly.
If you really need the whole inverse matrix, I would suggest
LinearAlgebra.inv!(cholesky!(X'X + Diagonal(d)))
(I don’t see the point of your Diagonal(1 ./ d) reformulation.)
I do need the inverse (more preferably), or at least the diagonals of the inverse…
What are you ultimately trying to compute, and what are your typical matrix sizes?