Machine Hardware (CPU and GPU)
The IEEE-754 standard [this is the 2008 version; see changes for the 2019 version; also short comment here] defines five rounding rules–2 are “round to nearest rules”, and 3 are “directed rounding” rules:
Roundings to Nearest
Round to nearest, ties to even – rounds to the nearest value; if the number falls midway, it is rounded to the nearest value with an even least significant digit; this is the default for binary floating point and the recommended default for decimal.
Round to nearest, ties away from zero – rounds to the nearest value; if the number falls midway, it is rounded to the nearest value above (for positive numbers) or below (for negative numbers); this is intended as an option for decimal floating point.
Round toward 0 (zero) – directed rounding towards zero (also known as truncation ).
Round toward +∞ – directed rounding towards positive infinity (also known as rounding up or ceiling ).
Round toward −∞ – directed rounding towards negative infinity (also known as rounding down or floor ).
Number 1 (round to nearest, ties to even) is the default rounding mode, and has the advantage that it is without positive/negative bias and without bias toward/away from zero.
- However, it will distort a distribution of values, by increasing the probability of even numbers relative to the number of odd numbers.
There are some obscure variations of #1 (“alternate tie-breaking” – take turns rounding ties to even/odd, and “random tie-breaking”–randomly round towards even/odd).
A method with desirable statistical properties is stochastic rounding (also see here), which is unbiased and the expected rounding error is zero. No hardware support (yet).
The Hidden Elephant In Hardware Floating Point
Rounding might not be your worst nightmare – it might be that floating point numbers are not equally spaced.
The Invisible Pink Unicorn In Numerical Accuracy and Precision
How does the computer calculate exp(), log(), sin(), cos(), atan(), Normal()? Often they use Chebyshev Polynomials to approximate these functions. These approximations are often excellent, as Chebyshev polynomials are close to the best polynomial approximation and minimize Runge’s phenomenon. But … the approximation errors with oscillate around zero.
Yet … Rounding Errors are Not Random
If we look at this page from Higham’s Accuracy and Stability of Numerical Algorithms (Second Edition), p. 25:
Higham’s book has additional references on the topic.