this sounds like a fairly controlled level of error given how large your condition number is.
Put another way, think about this in terms of just regular numbers. If I have
x = 1e-10
y = 1e-10 + 1e-18
then
y - x = 1e-18
but
1/x - 1/y ≈ 99.9999
So just because the difference between two numbers is small, does not mean that the difference between their inverses is small.