Rounding Results for Comparison with Specification

The mysteries of rounding are exposed.
Apr 02, 2013


Chris Burgess
Rounding a result to the required number of decimal places is easy, isn’t it? After all, we were all taught at college or university that only two rules are needed.

Rule 1: If the digit after the figure to be rounded is less than 5, then don’t change the rounded figure (i.e., round down).

Rule 2: If the digit after the figure to be rounded is 5 or more, then increase figure to be rounded by 1 (i.e., round up).

Simple really, isn’t it? After all, the United States Pharmacopeia (USP) requires the use of just this method:

When rounding is required, consider only one digit in the decimal place to the right of the last place in the limit expression. If this digit is smaller than 5, it is eliminated and the preceding digit is unchanged. If this digit is equal to or greater than 5, it is eliminated and the preceding digit is increased by 1 (1).

Of course, rounding only takes place after the final calculation has been performed. The USP General Notices make this requirement clear:

Numbers should not be rounded until the final calculations for the reportable value have been completed. Intermediate calculations (e.g., slope for linearity) may be rounded for reporting purposes, but the original (not rounded) value should be used for any additional required calculations (1).

The problem is that simplicity is not always correct all the time. From a statistical point of view, applying rule 2 will bias the data over time because one will always round up particularly if 5 is frequently the figure to be rounded. 4 and below and 6 and above are balanced in rounding but what about 5? It has been known for more than 60 years that applying only rules 1 and 2 will cause biased data. What is even more interesting is that the third rule was well known at that time, at least to statisticians. Sadly, this rule is rarely if ever mentioned in modern textbooks and guidelines.

The purpose of this column is to resurrect this “forgotten” rule because the need for simplicity does not take precedence over scientific correctness. In any event, rule 3 is not hard to understand and is merely buried in older textbooks and standards.