2.7.12 Arithmetic Tutorial Exercise 1

Dividing two integers that are larger than the current precision may give a floating-point result that is inaccurate even when rounded down to an integer. Consider ‘123456789 / 2’ when the current precision is 6 digits. The true answer is ‘61728394.5’, but with a precision of 6 this will be rounded to ‘12345700. / 2. = 61728500.’. The result, when converted to an integer, will be off by 106.

Here are two solutions: Raise the precision enough that the floating-point round-off error is strictly to the right of the decimal point. Or, convert to Fraction mode so that ‘123456789 / 2’ produces the exact fraction ‘123456789:2’, which can be rounded down by the F command without ever switching to floating-point format.