The court's decision
From the outset, the US Supreme Court made clear that the issue at stake was: "Whether a plaintiff can state a claim for securities
fraud … based on a pharmaceutical company's failure to disclose reports of adverse events associated with a product if the
reports do not disclose a statistically significant number of adverse events" (18). The short answer provided by the Supreme
Court was clear and unanimous: yes. The Supreme Court affirmed the decision of the Ninth Circuit in finding that the bright-line
materiality rule sought by Matrixx was inconsistent with its ruling in Basic (19).
The Supreme Court noted that because multiple factors are considered in determining causation, the bright-line rule sought
by Matrixx "would 'artificially exclud[e]' information that 'would otherwise be considered significant to the trading decision
of a reasonable investor'" (20). Thus, Matrixx's argument in favor of statistical significance as the only reliable indication
of causation was deemed flawed (21). After citing a litany of circumstances under which causation is determined in the absence
of statistically significant data using other evidence, the Supreme Court held that "assessing the materiality of adverse
event reports is a 'fact-specific' inquiry ... that requires consideration of the source, content, and context of the reports."
This is not to say that statistical significance (or the lack thereof) is irrelevant—only that it is not dipositive of every
case (22). Consequently, the Supreme Court, citing its ruling in Basic, which in turn relied on the language of the TSC Industries decision, held that because (taking the investors' allegations as true) information provided to Matrixx revealed a plausible
causal relationship between Zicam Cold Remedy and anosmia, it was "substantially likely that a reasonable investor would have
viewed this information as having significantly altered the 'total mix' of information made available" (23).
Statistical discussions are exacting and circumscribed because the field of statistics uses the language of mathematics as
well as English to define, describe, and present statistical concepts and results. Words that are used freely in a lay discussion
take on mathematically precise meanings. Sentences are worded carefully to present the statistical concepts correctly. Generally,
this does not present a problem because the context usually reveals whether the discussion is technical or not. In the both
the Basic and Matrixx cases, it was clear that the discussions were not statistically oriented, and that the word "significant" was used in the
In the Carter–Wallace case, it is clear that the discussion is statistical in nature, and the word "significant" is used in its statistical technical
sense. Thus, there is a need for a definition of the term "statistical significance."
Statistical analysis provides scientists with a tool along with theory and common sense for making scientific interpretations
and conclusions. Often, the analysis focuses on identifying significant differences, that is, practical and statistical differences.
Practical significance comes from comparing an observed difference (i.e., a signal) with an absolute reference. Practical
significance always takes precedence over statistical significance. In fact, statistical significance should not be determined
until practical significance is found.
Statistical significance, on the other hand, comes from comparing an observed difference with a relative reference that incorporates
a noise or random variability. Statistical significance testing compares a signal with noise and is often expressed as a ratio
of signal to noise. The result is not meant to be a statement of causality, truth, or reality. That is because if the signal
can be shown to be stronger than the noise (i.e., more of a difference than expected by chance variation alone), then the
scientist may conclude it to be "statistically significant." Otherwise, it cannot be shown to be significant. If more data
are obtained, the noise could be reduced, perhaps helping to demonstrate that the signal is significant. In fact, if the noise
is small enough or if the sample is large enough, even wildly anomalous differences can be shown to be statistically significant.
This difference is the reason that significance cannot be a single "bright-line" rule for causality. Rather, the primary purpose
of statistical testing is to prevent the declaration of an apparent practical significance when, in fact, it could be caused
by random variation.