Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research
This book is aimed at the statistics consumer who has only a rather partial quantitative education—that is, most of us. The basic thrust is to counter the overwhelming emphasis on p-value-centered, null-hypothesis significance testing. The pros and cons of this debate are carefully reviewed. The major point is that drawing inferences from analyses solely in terms of p values is grossly inadequate and easily leads to misunderstandings. To remedy this, confidence limits, effect sizes, and meta-analyses are emphasized.
The extended, lucid treatment of the variety of effect sizes is unique. Also, a helpful page is titled “How to Fool Yourself With Effect Size Estimation.” Particularly trenchant are the statements that “generic definitions of effect size magnitude” are problematic, referring to the convention that considers a d index of 0.2 as small, whereas 0.8 is large. It is generally forgotten that the late remarkable innovator, Jack Cohen, proffered these tentative standards as representative of the range of effect sizes common in the psychological literature. He did not address whether this is due to feeble assessments.
A further strength is the welcome emphasis on replication:
No matter how intriguing the result for a single study, it must be replicated before it can be taken seriously. Replication also is the ultimate way to deal with the problem of sampling error. Indeed, statistical tests are unnecessary with sufficient replication.
Unfortunately, these axioms are quite inconsistent with the practices of most novelty-focused scientific journals as well as the U.S. Food and Drug Administration, where two mammoth trials that achieved statistical significance—despite miniscule effect sizes and no clearly useful clinical impact—were deemed sufficient for marketing.
The author’s firm stand for replication becomes somewhat problematic within the context of “meta-analytic thinking.” The admirable criticism applied to the interpretation and misinterpretation of p values is missing here. He cautiously states, “If the results of a meta-analysis helps researchers conduct better primary studies, then little more could be expected.” However, there is no explication of the many flawed meta-analyses that distort the psychiatric and psychological literature.
The unpublished critical paper “Meta-Analysis at 25” by the pioneer Gene V. Glass is linked as a Web resource in the book’s amplifying Web supplement (www.apa.org/books/resources/kline). Glass forthrightly denounces many statistical approaches to meta-analysis and pleads for massive, public, complete raw data archives to be posted on the Web concurrently with journal publication—to replace meta-analysis. That would really reform behavioral research.
