Computing matrix inverses is numerically stable, in the sense that it is backwards stable. This doesn’t mean that it is always accurate, however — in particular, it can give a very inaccurate result if the matrix is badly conditioned (close to singular).
And a rank-1 matrix a * a' is very badly conditioned indeed! Because of roundoff errors, it is not exactly singular, but the condition number is still huge:
julia> a = randn(100);
julia> cond(a * a')
5.413008718471948e21
This means that the result of inverting it is basically garbage from roundoff errors.