Inaccurate Matrix Multiplication

I don’t think that the conclusion here is that FMA is “dangerous”. Sometimes it will give more accurate results than non-fused operations, sometimes (probably less often) it will happen to give less accurate results, but in all cases the results are within the tolerance you would expect from floating-point calculations.

I think you are being confused here by the concept of relative error. Is 8.5e41 a “big” error? You have to ask big compared to what? In this case, the summands are ±1.9e58, and 8.5e41 / 1.9e58 ≈ 4.4e-17, so it’s pretty small compared to the inputs.

Compared to the exact output of 0.0, of course, the relative forward error is infinite (you have no correct significant digits), but that would be true of any nonzero answer, and is to be expected because the sum is ill-conditioned (a sum of nonzero inputs with an exact cancellation has an infinite relative condition number).

See also my answer here: "sum([1.0, 1e100, 1.0, -1e100])" fails - #10 by stevengj

11 Likes