Optimizing sums of products (dot products)

In some algorithms you can re-order things without any big problem (this is why many functions like reduce and sum in Julia leave their associativity/ordering/algorithm unspecified). But in others it would lead to catastrophe, or at least undesirable behavior.

The compiler doesn’t know which is which, so it defaults to doing what computer languages are usually supposed to do:

“The good news about computers is that they do what you tell them to do. The bad news is that they do what you tell them to do.” — Ted Nelson

The compiler is not allowed to change the results of your code, even if it thinks the change is small (it isn’t always!). You don’t want a compiler that avoids disastrously rewriting your code only 99% of the time.

That’s why usually languages require some flag like @simd or @fastmath to give them explicit permission to re-order/re-associate floating-point operations in ways that might change the result. This is not unique to Julia, e.g. gcc has -ffast-math to enable “unsafe” floating-point optimizations.

Note that this is wrong — the machine epsilon is a relative tolerance, not an absolute tolerance. This is a common confusion. See also (e.g.) History of `isapprox` in languages - #14 by stevengj

(Nor will different algorithms generally agree to a relative tolerance of the machine epsilon either … you often lose more digits than that. And sometimes you are more accurate than that. Precision ≠ accuracy.)

8 Likes