Dr. Klein Replies

To the Editor: Dr. Gould objects that my criticism of the meta-analysis by him and his colleagues neglected to mention their CD-ROM PsycLIT search from 1974 to 1994, as well as secondary references, etc. I regret these omissions, but my relevant point is the existence of appropriate studies from before 1974 as well as the incompleteness of the computerized review. The authors’ failure to achieve the meta-analytic goal of comprehensive unbiased review still stands.

Dr. Gould states that the Klosko et al. study provided valid data, although active drug could not be differentiated from placebo. Dr. Gould objects to my “circular reasoning.” However, such trials failed because their inability to distinguish an already validated drug from placebo indicates that something was wrong with the sample or the procedures or that a sampling fluctuation called any findings into question, etc. The FDA does not accept such studies (1; Klein, 1996).

Dr. Gould misses my major point. Regardless of effect size calculations, the relative merits of treatments that have not been directly compared in a properly controlled study of a randomized sample cannot be discerned. Psychiatric diagnosis is insufficient. The differences between patients in psychotherapy and patients receiving medication (which are not subtle) confound such comparisons. That cognitive behavior therapy is “at least as effective as pharmacotherapy,” as restated by Dr. Gould, has no factual or logical basis.

Dr. Bakker et al. object to my criticism of their noncomparative effect size without specifying its problems. However, I cited an article (2) that presents such a detailed critique. Dr. Bakker et al. are concerned that I did not refer to two recent meta-analyses; however, like Dr. Gould, they do not address my central criticism of the lack of sample comparability. They justify their meta-analysis by the scarcity of direct comparisons, which requires developing the “next-best answer.” I disagree with this conclusion.

The pseudoexactness of meta-analysis misapplied to such chaotic data provides an altogether unwarranted assurance of well-founded comparative inference. The authors’ wish to support direct comparisons could be better served by tabulating relevant studies with regard to both outcome and validity issues, e.g., randomization, blindness, and nature of control group. This would provide the correct basis for possible recommendations about direct comparisons. Such old-fashioned literature reviews yield a substantially better understanding than effect size manipulations when the data are so partial, limited, and irrelevant. Such meta-analyses are not second best; rather, they are off the validity scale.

Finally, Dr. Rifkin argues that I should have extended my criticisms to the lack of direct valid comparative evidence regarding psychotherapy and medication in the treatment of less severe depression. I entirely agree and regret the constricting space limitations for articles.