Lynn D. Torbeck is an international consultant and trainer specializing in applied statistics and experimental design for pharmaceutical, biopharmaceutical, and medical device research and development, validation, quality control/assurance, and production, 2000 Dempster Plaza, Evanston, IL 60202, tel. 847.424.1314.

Lynn D. Torbeck

Unused, misused, or misunderstood, statistical tolerance intervals are fundamentally different from statistical confidence intervals, which provide interval estimates for summary statistics, such as mean and standard deviation. Tolerance intervals were developed as a tool to provide confidence intervals for a population proportion. With insights gained, the tolerance interval can be effectively used to develop product and process specifications as well as determine conformance to requirements.

Statistical tolerance intervals are routinely used to assess a product’s conformance to requirements and to provide a range within which one can expect product attributes to be contained. In this paper, the authors focus on parametric tolerance intervals that assume normally distributed data. The authors introduce the idea of asymmetrical tolerance intervals as an aid in fully assessing product performance relative to product or process requirements.Tolerance intervals are generated by collecting data through sampling of representative product and processes and using tolerance interval factors (k) to create the tolerance limits. Tolerance interval factors are widely available in tables and available in many statistical computer programs.

Lynn D. Torbeck is a statistician with Torbeck and Associates. Chuck Pheatt is Professor Emeritus with Emporia State University.


Volume 45, Number 12
December 2021
Page: 38–42

When referring to this article, please cite it as L. D. Torbeck and C. Pheatt, “Using Tolerance Intervals to Assess Conformance to Requirements,” 

Articles

The authors introduce the idea of asymmetrical tolerance intervals as an aid in fully assessing product performance relative to product or process requirements.  

Using Tolerance Intervals to Assess Conformance to Requirements

Has risk management gone mainstream? One might think so given the June 13, 2011 article in the 

 on using the game of poker to train stockbrokers in the nature of risk (1). 

In the pharmaceutical industry, the reception of International Conference on Harmonization Guidelines Q8, Q9, and Q10 has led to widespread discussion, if not implementation, of the Quality Trio guidelines, with ICH Q9 specifically addressing risk management. 

While it is perhaps too early to say that risk management has gone mainstream, it has certainly had its fair share of news coverage of late. Due to a rare 9.0 magnitude earthquake and an unprecedented 45-ft tsunami that washed over the nuclear power plant in Fukushima, three out of six reactor cores melted down—the worst possible outcome. It will take decades and billions of yen to clean up. 

The world was stunned  by that event. If a highly-regulated industry run by a detail-oriented and meticulous culture such as Japan made this mistake, what chance do the rest of us have at risk management? Which raises the questions, what are the risks in risk management and can we reduce them? First it is important to define terms. 

In ICH Q9, risk to a patient equals probability times severity. What is probability? We can do little better than this observation from Bernoulli in 1713: 

"For it should be presumed that a particular thing will occur or not occur in the future as many times as it has been observed, in similar circumstances, to have occurred or not occurred in the past." —

This definition has been refined since then. Here are the three common definitions given: 

 An undefined subjective measure of belief from personal estimate, gut feeling or intuition

 The observed frequency of outcomes, as a fraction, percentage or a proportion

 A theoretical characteristic of the situation as determined by the limiting frequency of an infinite random series. 

I note with concern and dismay that ICH Q9 does not include probability in the list of definitions. Perhaps the authors assume that everyone has the same universal understanding of this complex topic, so a definition is not needed. 

Severity is the second component of risk and is defined in ICH Q9 as "a measure of the possible consequences of a hazard" (i.e., a meltdown). Now, let's turn to key steps for mitigating the risk in risk management. 

Include top management. They can delegate work but not the responsibility of a mission critical process. Note that risk management is a not an event but an ongoing reevaluation. Management must fight the danger of complacency, distraction, and stagnation. Risk analysis cannot simply become window dressing for the regulators. Risk reduction is costly, and management reflects the importance of the task with support, finances, and resources. 

At least one person in the risk team must understand probability well enough to teach it to the other members. Few people have studied probability and even fewer have practical experience of using probability. Real life probability estimation is much more difficult than textbook exercises. Yet, probability theory is the core of determining risk. 

Risk assessment cannot be seen as a check-the-box activity; it is a serious issue and requires serious effort and commitment. The success of the assessment is a direct result of the skill, education, and experience of the company's team members. Technical experts are absolutely necessary even if external experts must be called in. All stakeholders must work to agreement. 

Industry in general is less interested in the risk of an everyday occurrence, such as the risk of failing the moisture test for the next production run. What is more important is estimating risks such as a recall or another Heparin type tragedy. The difficulty in doing so is compounded by the lack of information and actionable data. In the absence of historical data, we are left with a gut-feeling, which can often underestimate the risk. The best experts must be consulted. 

"For every complex and difficult problem, there is a solution that is simple, intuitive, easy to understand—and completely wrong." —

Estimating probability and severity on scales of low, medium, and high (or on a scale of 0–4) are so simplistic as to be almost worthless; like using third grade English to study a college subject. Formal training in probability is needed to advance beyond this simplistic approach. 

Finally, focus on the patient, build for the future, collect relevant data, and store it for easy access and analysis. Expand the knowledge base of the team and the company. Learn from others and learn advanced probability theory. Continue to refine and improve. Only through our best efforts can industry say with confidence that it understands the risks facing patients. 

 is a statistician at Torbeck and Assoc., 2000 Dempster Plaza, Evanston, IL 60202, tel. 847.424.1314, 

A firm grasp of probability and ongoing re-evaluation are key.

Reducing the Risk in Risk Management

We all have a scrupulous eye for subjects we are passionate about. Foodies insist on using the most authentic ingredients for their favorite dishes. Classic-car collectors require the smallest details to be as close to the original as possible. And statisticians cajole nonstatisticians to avoid classic pitfalls in applied statistics. In this latter case, however, it is not a matter of taste or authenticity. Incorrect statistical practices can result in erroneous calculations and poor conclusions. Some errors are small, but others can be monumental, and since we never know which way the apple will fall, we should treat them all the same. 

Although there is always a large scope for error in a statistical project, some mistakes are more common than others. Those that appear to be correct on the surface are what we call pitfalls. 

This section highlights some of the most common challenges facing statisticians. 

Reportable value or result is not defined for the data and the analysis (1). By definition, the reportable value is the end result of the complete measurement method as documented. It is the value compared with the specification and the official value most often used for statistical analysis. If different people or departments have different definitions, confusion reigns and out-of-tolerance and out-of-specification investigations multiply. 

The average of a set of averages is correct only if the sample sizes are the same. Otherwise, the averages are weighted by the sample sizes (2). In addition, avoid averaging standard deviations even when the sample sizes are the same. The variance is the standard deviation squared. Variances can be averaged when the sample sizes are the same. If the sample sizes are not the same, then a weighting formula is used (2). 

The percentage relative standard deviation. 

The percentage relative standard deviation (%RSD) is not a substitute for the standard deviation because they measure different aspects of variation. Report both with the sample size. Also, avoid trying to average %RSDs or calculate %RSD on data expressed as percentage recovery. The data is already a percentage, so the average and the standard deviation will also be percentages. 

Always report the sample size. Remember the famous rat study where one third of the rats got better, one third got worse and the third rat ran away? 

Avoid gratuitous summary statistics without a clear purpose; they cloud the interpretation. Likewise, printing out massive lists of all possible summary statistics of all possible sets, subset and sub-subsets isn't worth the paper it is printed on. 

Give absolute values when looking at relative changes. For example, 2 out of 100 million versus 1 out of 100 million is a 100% increase, but it is still only 2 out of 100 million. Trust your reader to understand the practical implications of the data. 

Ranking anything without giving absolute values and/or some sense of comparison to practical importance can be misleading. For example, consider that we ranked schools using a metric that results in one school being at the bottom of the list and another school at the top. But then, we realize  both schools produced Nobel Prize winners. Does the ranking therefore have any meaning?  

. Avoid pie charts unless your goal is to deliberately confuse your reader. Graph the data before starting a formal statistical analysis. Common graphs include histograms, time plots, and scatter plots. Attempt to get cause and effect on the same page (3). 

Attempt to determine the underlying distribution of the data before starting a formal statistical analysis. While the normal distribution is the most common, many other distributions, such as the log-normal, are also found. 

If the data are symmetrical around the average, use the average and the standard deviation. If the data are skewed, the median and interquartile range is preferred. This is not a hard and fast rule, just good practice. 

Correcting or adjusting one assay result with the result of a second assay that has the same as or larger variance than the first will result in more total variability, not less, because the variances add up. This is known as the "weight to run" problem, in which tablet weight is adjusted using an assay test for potency. This problem can lead to rejecting good lots of materials and products. In most cases, setting the weight equal to the target results in less variation. 

Recognize that the population parameters, such as mu (µ), the population mean, and sigma (σ), the population standard deviation, are single values, whereas sample estimates of the population mean and sample estimates of the population standard deviation are random results from a distribution. Every additional sample gives slightly different results, so problems arise when sample estimates are treated as if they are population values. This leads to treating other sample estimates such as %RSD and Cpk as if they are without variation. Confidence intervals should be calculated for these statistics to estimate their uncertainty.  

 Define in exact detail what the phrase "within the variation of the method" means for specific applications because there is no universally accepted definition. Try not to use use confidence intervals to set specification criteria. Instead, use tolerance intervals to get a starting point. Overlapping or non-overlapping confidence intervals are not a significance test. 

The most egregious pitfall of all is calculating the sample average plus and minus three times the sample standard deviation without considering the sample size and distribution, and then using it for confidence intervals, setting specification criteria, identifying outliers and all other manner of ad-hoc comparisons. It is not a universal statistical tool. In fact, this equation came about in the 1920s via Dr. Walter Shewhart for defining control charts and is no longer of much use today. 

Of course, there are many more potential pitfalls, and when in doubt, contact your local statistician. 

Schaum's Outline of Statistics Fourth Edition

The Visual Display of Quantitative Information (

The hardest errors to spot are the ones that don't look like errors at all.

Pitfalls in Statistics

Statisticians and non-statisticians daily select the level of statistical significance to be used for decisions, experimental designs, data collection, sample sizes, and other formal analysis. They usually choose 5%. Why 5%? Note this definition in a well-known dictionary: "Significance level: The level of probability which it is agreed that the null hypothesis will be rejected. Conventionally set at 0.05" (1). 

Thus, the Cambridge Dictionary of Statistics gives 5% as the definition of the significance level. Why 5%? Why not some other value? Is it acceptable to be wrong 5% of the time? If we choose another value, what value(s) should it be: 10%, 1%, or 0.1%? 

Other writers have reflected on this as well. "Why is the 0.05 level of significance used as the decision point to reject the null hypothesis? Why not 0.06 level of significance? Actually, the 0.05 level of significance is used because of tradition. [Sir] R. A. Fisher (2), the founder of modern statistical methods, chose this value and other scientists have accepted the choice" (3). 

In his landmark 1926 paper, "The Arrangement of Field Experiments," (4) Sir Ronald presented his logic for selecting confidence levels: 

"It will illustrate the meaning of tests of significance if we consider for how many years the [farm] produce (i.e., results) should have been recorded in order to make the evidence convincing. First, if the experimenter could say that in twenty years experience with uniform treatment the difference in favour of the acre treated with manure had never before touched 10 per cent, the evidence would have reached a point which may be called the verge of significance; ... This level, which we may call the 5 per cent. point, would be indicated, though very roughly, by the greatest chance deviation observed in twenty successive trials. ... If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent. point), or one in a hundred (the 1 per cent. point)."  

Apparently, Sir Ronald was not fixated on 5%. 

Fisher worked at the Rothamsted Experimental Station located in Harpenden, England, from 1919–1933 (2, 5). While there, he studied crop yields and animal husbandry using statistics and the theory of experimental designs that he developed. An incorrect decision affected only the distribution of manure on crop fields or the care and feeding of pigs and honey bees. Being wrong for 5% of the decisions wouldn't seem to be a major problem. Also, Fisher was not legalistic in his use of significance levels. Being pragmatic, he was much more concerned with the practical impact or practical significance of the results. It should be noted that Fisher developed many of the valuable statistical tables used for significance testing, and he chose levels of 0.05 and 0.01%. Thus, in effect, he forced the rest of the world to go along with his choices regardless of the application. 

Other statisticians have pointed out that the selected level of 5% determines how often we will be wrong in our decisions. "In rejecting the null hypothesis, the sampler faces the possibility that he is wrong. Such is the risk always run by those who test hypotheses and rest decisions on the tests.  ...  As a matter of practical convenience, probability levels of 5% (0.05) and 1% (0.01) are commonly used in deciding whether to reject the null hypothesis.  ...  This use of 5% and 1% levels is simply a working convention. There is merit in the practice, followed by some investigators, of reporting in parentheses the probability ... (6)." 

"The question arises: at what probability level does a deviation become statistically significant? There is no rational probability level at which possibility ceases and impossibility begins, but it is conventional to regard a probability of 0.05 as the critical level of significance" (7).  

Thus, the hallowed 5% significance level was born from a crop experiment and a manure spreader. I am sure this gives reassurance to those benefitting from the next analysis. 

 (Cambridge University Press, Cambridge, MA, 1998) p. 305. 

 (John Wiley & Sons, New York, NY, 1978). 

Biostatistics, Experimental Design and Statistical Inference

 (Oxford University Press, Oxford, 1993) p. 84.  

 (John Wiley & Sons, New York, NY, 1981) p. 245. 

, 6th ed., (Iowa State Press, Ames, IA, 1971) p. 27 

 4th ed., (John Wiley & Sons, New York, NY, 1951) p. 89.  

A history of the selection of the widely used significance level leaves much to be desired.

Statistical Solutions: On the Verge of Significance: Why 5%

Dr. James Bergum is now getting the recognition he deserves. His technique fundamentally changes the way we should use 

 (USP) standards and in-house specifications. Bergum, an associate director of nonclinical biostatistics at Bristol Myers Squibb, published a landmark paper in 1990 that, with his follow-on work, is now the basis for a new American Society for Testing and Materials (ASTM) Standard, E11 2709-09, adopted in 2009. 

People often ask statisticians about sample sizes, particularly for 

 standards such as <905> Uniformity of Dosage Units and <711> Dissolution. For example, "Is a sample of size 10 or 30 tablets adequate for a lot of 2.1 million tablets?" Or, "How many capsules—6, 12, or 24—are enough to measure dissolution profiles?" 

But, how do we decide the sample size? How do we relate the sample size to the chances of passing a standard? Suppose we collect and test a representative sample of 30 tablets from a lot. Suppose further the specifications are 90% to 110%, and the 30 values are near 100% with a low standard deviation. What is the probability that the lot, if tested again or even several times, would meet the 

 standard? Intuitively, we would say the probability is quite high. In this situation, we may not have control over the sample size, so rather than ask what should the sample size be, perhaps the question should be "What is the probability?" Bergum's method can be used to answer this question. 

Bergum's method is a statistical procedure for calculating the probability of passing multistage 

 standards such as dissolution or uniformity of dosage units. Passing a 

 standard once is no assurance that the lot would pass again if tested later, either on stability, by another company, or the US Food and Drug Administration. But, by using Bergum's method, a lot can be tested for dissolution, and the probability of meeting the standard in the future can be determined. His approach converts an absolute standard into a workable practical "in-house" specification. 

Thus, the test data for multistage dissolution criteria are converted into a probability of passing the standard with a specified confidence level. If the probability is less than a specified criterion, say, 95% with 95% confidence, then a larger sample size may be needed to better assure passage of the standard for a future sample. Or, perhaps the lot is not acceptable and should be rejected. Passing a 

 standard once does not ensure that subsequent samples would also pass the standard. The probability of future tests from the lot can be calculated and reported. 

 chapters such as the one for dissolution are standards and not working specifications. The standard must be met at any time up to the expiry date. It is a bright line that can't be crossed. It is absolute; there is no probability associated with a standard. By contrast, specifications are developed in-house by companies to provide a high degree of confidence of meeting the standard at any time. Specifications have a probability associated with them. 

Unfortunately, many companies don't develop in-house specifications, but rather use the 

 standard as if it is a specification. Because there is no probability associated with the standard, passing the standard once doesn't provide any assurance of passing again in the future. 

Bergum's method evaluates the variability in the dissolution data and calculates a lower probability of passing the standard with a specified confidence. So instead of a pass/fail response, using the 

 standard as a specification, Bergum's method gives a lower bound on the probability.  

Dr. Bergum has provided a valuable service to our industry. He has developed a sound statistical basis for creating in-house specifications based on 

 standards. This paradigm shift fundamentally changes how our industry should develop in-house specifications.  

	1.	J. S. Bergum, Drug Development and Industry Pharmacy, 16 (14), 2153–2166 (1990).

2. 	J.S. Bergum and M. L. Utter, “Process Validation,” Encyclopedia of Bio-pharmaceutical Statistics (Marcel Dekker, New York, 2000).	 

3.	 J. S. Bergum and M. L. Utter, “Statistical Methods for Uniformity and Dissolution Testing,” in Pharmaceutical Process Validation, 3rd ed., Eds. R. Nash, R. and A. Wachter (Marcel Dekker, New York, 2003).

4. 	P. Cholayudth, Pharm. Technol. 28 (9), 86–98 (2004).

5.	 P. Cholayudth, Pharm. Technol. 30 (1),( 88-94 (2006).

6. 	P. Cholayudth, Jrnl. of Validat. Technol. 13 (4), (2007).

7. 	J.S. Bergum and L. Hua, Pharm. Technol. online, http://pharmtech.findpharma.com/pharmtech/article/articleDetail.jsp?id=463577), accessed Apr. 14, 2010, October 2007.

8. 	P. Cholayudth, Jrnl. of Validat. Technol. 14 (2), 62–72, (2008).

9. 	P. Cholayudth, Jrnl. of Validat. Technol. 14 (3) 67–80 (2008).

10. 	P. Cholayudth, Jrnl. of Validat. Technol. 14 (4), 72–78 (2008).

11.	 P. Cholayudth, Jrnl. of Validat. Technol. 15 (4), 77–90, (2009).

12. 	P.D. Lunney and C. A. Anderson, Jrnl. of Pharm. Innov. 4 (2009). PT

A method for determining sample size is finally getting some respect.