Have you considered a Tikhonov-regularized Kalman filter? There seem to be several published variants of this notion, but my guess (caveat: without reading the papers) is that they are all conceptually centered on replacing a truncated pseudo-inverse A^+ (where you drop singular values smaller than \sigma_0, i.e. “truncate” the SVD) with a Tikhonov regularization (A^T A + \sigma_0^2 I)^{-1} A^T.
Alternatively, you can compute an analogue of your pseudo-inverse with a “truncated QR” (column-pivoted QR where you simply drop columns where the diagonal elements of R become too small). Then you are replacing one SVD with one QR factorization, which should definitely be faster (even if it is theoretically less reliable than SVD).
(Note that in any of these cases, you may not need to compute any matrix inverse explicitly — my recollection is that you just need to multiply it by another matrix in the Kalman algorithm, which can usually be done more quickly than explicitly forming the inverse and then multiplying it.)