Technical Fellow at Northrop Grumman Information Technology

Thomas F. Curry

The attempt to define an "outlier" has a long and diverse history. Despite many published definitions, statisticians in all fields are still interested in objectively determining whether a data point is consistent with the rest of the data set or is, after all, an outlier, signifying a deviation from the norm. For biopharmaceutical companies, the need to evaluate whether a data point is an outlier - inconsistent with the rest of a small set of data - is important in validating process consistency. 	

The power of a statistical tool increases as sample size (n) increases. So, low power outlier tests used on small data sets, such as the production data derived from 3-5 conformance runs, have relatively high beta (β) or Type 2 errors. These errors indicate there is a high chance of leaving deviant results undetected, making these tests inappropriate for pharmaceutical or biopharmaceutical applications.	

The z-test is the most powerful outlier test (the most able to detect a discordant datum) if the data are normally distributed and the standard deviation is known or can be accurately estimated. But z-tests usually require large data sets. Conformance runs from early commercial lots usually produce small data sets, and the standard deviation of the population is not usually known. We show, using representative biopharmaceutical process validation data, that the Weisberg t-test is a powerful outlier test at small values of n. It has a low β error rate in detecting deviations from the mean. Therefore, the Weisberg t-test is suitable for objectively demonstrating consistency of production data.	

During process validation, process consistency is typically demonstrated in 3-5 conformance runs. When historical data are available, even if those data are from a different production scale, statistical comparisons are relatively straightforward. Those data can be used to set acceptance criteria for validation runs at a new production scale. 				

When approximately 15 lots have been produced at commercial scale, control charts, which illustrate a process and its variation with time, are useful for evaluating process stability. The authors' choice of 15 lots for calculating control limits is a balance between the extreme uncertainty of limits based on few data and the diminishing value of each new data point in further decreasing that uncertainty. An individuals control chart based on 15 lots has 8.9 effective degrees of freedom (df), which is sufficient to reduce the uncertainty in the limits to approximately plus/minus 23%. Achieving plus/minus 10% uncertainty requires approximately 45 degrees of freedom, which requires more than 70 individual values.1	

 Until there are 15 lots, the most useful method to statistically evaluate a data point that seems to be anomalous is the Weisberg t-test.2,3 The Weisberg t-test can be used for data sets larger than 15 values as well. The other tests evaluated for application in small data sets were the Dixon4 and the Grubbs.5 The latter is also known as the Extreme Studentized Deviate (ESD).6 The Weisberg t-test can distinguish between normal process variation and a process aberration that yields an outlier. For example, at an alpha (α) error (calling something an outlier when it is not one) of 0.05, the b error is only 0.31, and thus the Weisberg t-test is the most powerful test available for small data sets among those considered (Figure 1).

Figure 1: The operating characteristic curves for a variety of outlier tests are created by repeatedly drawing four values from a population of known mean and S, with a fifth value taken from a population with a known shift in mean.	

Until recently, the US Pharmacopeia (USP) did not address the treatment of chemical test data containing discordant observations. This "silence" was interpreted to mean a "prohibition" during the US versus Barr Laboratories, Inc. case.7 The judge's ruling in that case indicated the need for such guidance8 and in 1999, a new monograph was previewed in Pharmacopeial Forum.9 That monograph states that when appropriately used, outlier tests are valuable tools for analysing discordant observations.				

The discussions in the Barr case and the USP monograph advocate using an outlier test to disregard a data point. In this article, the authors use the Weisberg t-test to objectively identify an outlier as part of a statistical evaluation of small data sets. For process validation purposes, if the Weisberg t-test identifies no outliers, the data can be claimed to be consistent based on an objective statistical method.	

This article describes the application of the Weisberg t-test to data from five conformance runs. The authors examine the ability of the test to demonstrate process consistency. Subsequent uses of this test would include checking a suspect data point from lot six with the previous five, or lot seven from the previous six, for instance. The Weisberg t-test could also be used during a retrospective review of data sets. For example, an earlier value may stand out as a possible outlier, but only after subsequent data show a pattern that distinguishes it as a possible outlier. As standard practice, the authors advocate an investigation of the causes of such statistical differences. 	

At 15 data points, the individuals control chart for each point becomes the preferred tool for detecting discordant observations and for showing process consistency (defined as the absence of discordant observations). If the data are available in subgroups, then an averages control chart is preferred.	

In this article, an outlier refers to a datum that appears not to belong to the same group as the rest of the data; that datum measurement may seem to be either too large or too small in relation to the general pattern of the rest of the data. The method proposed applies to a single outlier, and is similar to the traditional t-calculated (t

) form of the general t-test statistic (Equation 1).10,11		

 or null) can be stated as: the suspected value is not an outlier. Its alternative (H

 or alternative) is stated as: the suspected value is an outlier.	

Working with reduced data. The entire set of data should not be used to estimate the standard error (SE). Such estimates would be biased if the suspected outlier were included. The estimate of variation would be inflated and the estimate of the arithmetic mean would be biased toward the outlier. 	

 After computing the estimates without the suspected outlier, the Weisberg t-test statistic for the suspected outlier (denoted by y

 is its standard deviation after the withdrawal of y

The logic of the Weisberg t-test is that the numerator (y

 is the classic sample standard deviation. 	

) downward and is more conservative for small samples. Specifically, the above factor is identical to	

where h is the leverage matrix;3 that is	

because the data are reduced by one observation. The estimated SE of the mean y

Articles

Determining whether a data point is an "outlier" - a result that does not fit, is too high or too low, is extreme or discordant - is difficult when using small data sets, such as the data from three, four or five conformance runs. In this article, the authors demonstrate that the Weisberg t-test is a powerful tool for detecting deviations in small data sets.