Is 0.56 a big or a small difference? You always need to ask compared to what? Here, the relevant comparison is probably to the norm of the inverse, i.e. you want to know the relative error:
norm(inv(A)-inv(B),Inf)) / norm(inv(A),Inf)
What is it?
As @Oscar_Smith writes, the relative error could still be large if your matrices are badly conditioned, and generally you want to avoid computing explicit inverses to begin with. But looking at absolute errors is a common mistake.