The court's decision

From the outset, the US Supreme Court made clear that the issue at stake was: "Whether a plaintiff can state a claim for securities fraud … based on a pharmaceutical company's failure to disclose reports of adverse events associated with a product if the reports do not disclose a statistically significant number of adverse events" (18). The short answer provided by the Supreme Court was clear and unanimous: yes. The Supreme Court affirmed the decision of the Ninth Circuit in finding that the bright-line materiality rule sought by Matrixx was inconsistent with its ruling in Basic (19).

The Supreme Court noted that because multiple factors are considered in determining causation, the bright-line rule sought by Matrixx "would 'artificially exclud[e]' information that 'would otherwise be considered significant to the trading decision of a reasonable investor'" (20). Thus, Matrixx's argument in favor of statistical significance as the only reliable indication of causation was deemed flawed (21). After citing a litany of circumstances under which causation is determined in the absence of statistically significant data using other evidence, the Supreme Court held that "assessing the materiality of adverse event reports is a 'fact-specific' inquiry ... that requires consideration of the source, content, and context of the reports." This is not to say that statistical significance (or the lack thereof) is irrelevant—only that it is not dipositive of every case (22). Consequently, the Supreme Court, citing its ruling in Basic, which in turn relied on the language of the TSC Industries decision, held that because (taking the investors' allegations as true) information provided to Matrixx revealed a plausible causal relationship between Zicam Cold Remedy and anosmia, it was "substantially likely that a reasonable investor would have viewed this information as having significantly altered the 'total mix' of information made available" (23).

Statistical discussions are exacting and circumscribed because the field of statistics uses the language of mathematics as well as English to define, describe, and present statistical concepts and results. Words that are used freely in a lay discussion take on mathematically precise meanings. Sentences are worded carefully to present the statistical concepts correctly. Generally, this does not present a problem because the context usually reveals whether the discussion is technical or not. In the both the Basic and Matrixx cases, it was clear that the discussions were not statistically oriented, and that the word "significant" was used in the general sense.

In the Carter–Wallace case, it is clear that the discussion is statistical in nature, and the word "significant" is used in its statistical technical sense. Thus, there is a need for a definition of the term "statistical significance."

Statistical analysis provides scientists with a tool along with theory and common sense for making scientific interpretations and conclusions. Often, the analysis focuses on identifying significant differences, that is, practical and statistical differences. Practical significance comes from comparing an observed difference (i.e., a signal) with an absolute reference. Practical significance always takes precedence over statistical significance. In fact, statistical significance should not be determined until practical significance is found.

Statistical significance, on the other hand, comes from comparing an observed difference with a relative reference that incorporates a noise or random variability. Statistical significance testing compares a signal with noise and is often expressed as a ratio of signal to noise. The result is not meant to be a statement of causality, truth, or reality. That is because if the signal can be shown to be stronger than the noise (i.e., more of a difference than expected by chance variation alone), then the scientist may conclude it to be "statistically significant." Otherwise, it cannot be shown to be significant. If more data are obtained, the noise could be reduced, perhaps helping to demonstrate that the signal is significant. In fact, if the noise is small enough or if the sample is large enough, even wildly anomalous differences can be shown to be statistically significant. This difference is the reason that significance cannot be a single "bright-line" rule for causality. Rather, the primary purpose of statistical testing is to prevent the declaration of an apparent practical significance when, in fact, it could be caused by random variation.