The problem is that the biggest N0f8
not larger than 0.33
is 0.329
, whereas the biggest one not larger than 0.3
is 0.298
. If you floor with only one decimal digit of precision, the algorithm rightly chooses ever smaller numbers until it reaches a number with one digit that is exactly representable as x/256. (Remember, N0f8 is the type representing all fractions of that form!). In this case, that’s 0.2
.
Note the documentation of floor
:
floor(x) returns the nearest integral value of the same type as x that is less than or equal to x.