Fast math in NASA benchmark

[PetrKryslUCSD]

I noticed in this benchmark that the --fast-math GCC option was far more effective in reducing the runtime then the --math-mode=fast Julia option. What do you think is the reason for that?

[simonbyrne]

I think it comes down to being fundamentally bad idea, hence:

https://github.com/JuliaLang/julia/issues/36246

[simonbyrne]

But to give you a better answer: the biggest speed-up in gcc is in the matrix multiplication benchmark: using -Ofast here would let them reassociate arithmetic operations (to exploit SIMD instructions), and use FMA instructions if they’re available. Julia’s @simd macro and muladd functions will do something similar, without the blatant unsafe-ness of fastmath.

[evanfields]

Is there a “reader’s digest” version of why @fastmath is bad?

[mbauman]

It’s an under-specified grab-bag of math-breaking the compiler can do everywhere. Some algorithms actually depend upon math working correctly.

[Oscar_Smith]

One specific problem with it is that it’s recursive. While at first this feels like a good thing (I want all my code fast), the problem is it can end up applying to stuff in Base or other libraries where it is specifically broken.

[tbeason]

Relevant issue https://github.com/JuliaLang/julia/issues/36246 [oops… this was already linked]

[mbauman]

There was once a very long (non-readers-digest) discussion about this, but this post is fairly self-contained:

Accurate summation algorithm

A point I’ve made several times previously is the following: Mathematically, since + is associative and commutative, all correct summation algorithms are algebraically equivalent to each other, in particular they are all equivalent to left-to-right summation. The -ffast-math option gives the compiler license to transform any algorithm into any algebraically equivalent one in the mathematical sense. Therefore -ffast-math gives the compiler license to transform any summation algorithm int…

Especially if you combine it with the fact that you can pick a collection of 2046 numbers that can sum to just about any number depending upon their ordering:

Array ordering and naive summation

One of the first things you learn in numerical analysis is that floating-point operations are not associative. A classic example is this: julia> (0.1 + 0.2) + 0.3 0.6000000000000001 julia> 0.1 + (0.2 + 0.3) 0.6 I was thinking of ways to make floating-point summation independent of the order of the summands without making the performance much worse (this is a hobby of mine). Julia currently uses a pairwise summation algorithm, which is much better than naive left-to-right reduction while havin…

Base’s code actually depends upon smartly re-ordering summations to improve precision that fast math might break.

[PetrKryslUCSD]

I filed an issue to alert developers responsible for the NASA software comparison benchmark suite to the inappropriateness of including fast math in the results.
https://github.com/JulesKouatchou/basic_language_comparison/issues/9

When fast math option is not used, in the Jacobi iteration benchmark
Julia is faster than the C-language implementation. 6.96 s versus 7.51 s.
Tested with WSL 2, Julia 1.6, generic Linux.