Statistical Solutions: On the Verge of Significance: Why 5% - Pharmaceutical Technology

Latest Issue

Latest Issue
PharmTech Europe

Statistical Solutions: On the Verge of Significance: Why 5%
A history of the selection of the widely used significance level leaves much to be desired.

Pharmaceutical Technology
Volume 34, Issue 7, pp. 36

Lynn D. Torbeck
Statisticians and non-statisticians daily select the level of statistical significance to be used for decisions, experimental designs, data collection, sample sizes, and other formal analysis. They usually choose 5%. Why 5%? Note this definition in a well-known dictionary: "Significance level: The level of probability which it is agreed that the null hypothesis will be rejected. Conventionally set at 0.05" (1).

Thus, the Cambridge Dictionary of Statistics gives 5% as the definition of the significance level. Why 5%? Why not some other value? Is it acceptable to be wrong 5% of the time? If we choose another value, what value(s) should it be: 10%, 1%, or 0.1%?

Other writers have reflected on this as well. "Why is the 0.05 level of significance used as the decision point to reject the null hypothesis? Why not 0.06 level of significance? Actually, the 0.05 level of significance is used because of tradition. [Sir] R. A. Fisher (2), the founder of modern statistical methods, chose this value and other scientists have accepted the choice" (3).

In his landmark 1926 paper, "The Arrangement of Field Experiments," (4) Sir Ronald presented his logic for selecting confidence levels: "It will illustrate the meaning of tests of significance if we consider for how many years the [farm] produce (i.e., results) should have been recorded in order to make the evidence convincing. First, if the experimenter could say that in twenty years experience with uniform treatment the difference in favour of the acre treated with manure had never before touched 10 per cent, the evidence would have reached a point which may be called the verge of significance; ... This level, which we may call the 5 per cent. point, would be indicated, though very roughly, by the greatest chance deviation observed in twenty successive trials. ... If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent. point), or one in a hundred (the 1 per cent. point)." Apparently, Sir Ronald was not fixated on 5%.

Fisher worked at the Rothamsted Experimental Station located in Harpenden, England, from 1919–1933 (2, 5). While there, he studied crop yields and animal husbandry using statistics and the theory of experimental designs that he developed. An incorrect decision affected only the distribution of manure on crop fields or the care and feeding of pigs and honey bees. Being wrong for 5% of the decisions wouldn't seem to be a major problem. Also, Fisher was not legalistic in his use of significance levels. Being pragmatic, he was much more concerned with the practical impact or practical significance of the results. It should be noted that Fisher developed many of the valuable statistical tables used for significance testing, and he chose levels of 0.05 and 0.01%. Thus, in effect, he forced the rest of the world to go along with his choices regardless of the application.

Other statisticians have pointed out that the selected level of 5% determines how often we will be wrong in our decisions. "In rejecting the null hypothesis, the sampler faces the possibility that he is wrong. Such is the risk always run by those who test hypotheses and rest decisions on the tests. ... As a matter of practical convenience, probability levels of 5% (0.05) and 1% (0.01) are commonly used in deciding whether to reject the null hypothesis. ... This use of 5% and 1% levels is simply a working convention. There is merit in the practice, followed by some investigators, of reporting in parentheses the probability ... (6)."

"The question arises: at what probability level does a deviation become statistically significant? There is no rational probability level at which possibility ceases and impossibility begins, but it is conventional to regard a probability of 0.05 as the critical level of significance" (7).

Thus, the hallowed 5% significance level was born from a crop experiment and a manure spreader. I am sure this gives reassurance to those benefitting from the next analysis.

Lynn D. Torbeck is a statistician at Torbeck and Assoc., 2000 Dempster Plaza, Evanston, IL 60202, tel. 847.424.1314,


1. B.S. Everitt, The Cambridge Dictionary of Statistics (Cambridge University Press, Cambridge, MA, 1998) p. 305.

2. J.F. Box, and R.A. Fisher, The Life of a Scientist (John Wiley & Sons, New York, NY, 1978).

3. J.F. Zolman, Biostatistics, Experimental Design and Statistical Inference (Oxford University Press, Oxford, 1993) p. 84.

4. R.A. Fisher, Jrnl of the Ministry of Agric., 33, 504 (1926).

5. J.L. Folks, Ideas of Statistics (John Wiley & Sons, New York, NY, 1981) p. 245.

6. G.W. Snedecor, and W. G. Cochran, Statistical Methods, 6th ed., (Iowa State Press, Ames, IA, 1971) p. 27

7. L.H.C. Tippett, The Methods of Statistics, 4th ed., (John Wiley & Sons, New York, NY, 1951) p. 89.


blog comments powered by Disqus
LCGC E-mail Newsletters

Subscribe: Click to learn more about the newsletter
| Weekly
| Monthly
| Weekly

FDASIA was signed into law two years ago. Where has the most progress been made in implementation?
Reducing drug shortages
Breakthrough designations
Protecting the supply chain
Expedited reviews of drug submissions
More stakeholder involvement
Reducing drug shortages
Breakthrough designations
Protecting the supply chain
Expedited reviews of drug submissions
More stakeholder involvement
View Results
Eric Langerr Outsourcing Outlook Eric LangerRelationship-building at Top of Mind for Clients
Cynthia Challener, PhD Ingredients Insider Cynthia ChallenerRisk Reduction Top Driver for Biopharmaceutical Raw Material Development
Jill Wechsler Regulatory Watch Jill Wechsler Changes and Challenges for Generic Drugs
Faiz Kermaini Industry Insider Faiz KermainiNo Signs of a Slowdown in Mergers
CMOs and the Track-and-Trace Race: Are You Engaged Yet?
Ebola Outbreak Raises Ethical Issues
Better Comms Means a Fitter Future for Pharma, Part 2: Realizing the Benefits of Unified Communications
Better Comms Means a Fitter Future for Pharma, Part 1: Challenges and Changes
Sandoz Wins Biosimilar Filing Race
Source: Pharmaceutical Technology,
Click here