Medications Versus Cognitive Behavior Therapy for Severely Depressed Outpatients: Mega-Analysis of Four Randomized Comparisons

Discussions of the comparative efficacy of antidepressant medication and cognitive behavior therapy for the treatment of severely depressed outpatients have been marked by controversy and conflicting claims. Findings from the early comparative studies of Rush and colleagues (1) and Murphy and colleagues (2) led many in the field to conclude that cognitive behavior therapy is at least as effective as antidepressant medication as an acute treatment for depression (3–5). But reviewers who have focused on results from the National Institute of Mental Health Treatment of Depression Collaborative Research Program (Elkin et al. [6]) have concluded that cognitive behavior therapy is not an effective treatment for severely depressed outpatients (7). Indeed, current depression treatment guidelines recommend the use of antidepressant medication, and not cognitive behavior therapy, for more severe depressions. For example, APA’s 1993 “Practice Guideline for Major Depressive Disorder in Adults” (8) includes the statement: “There is some evidence that cognitive therapy reduces depressive symptoms during the acute phase of less severe, nonmelancholic forms of major depression, but not significantly differently from pill-placebo coupled with clinical management.” The Agency for Health Care Policy and Research guidelines (9) state: “For severe and psychotic depressions, there is strong evidence for the efficacy of medication and little or none for the efficacy of psychotherapy alone.” APA’s summary was based exclusively on the study by Elkin et al. (6), which favored imipramine with clinical management over cognitive behavior therapy, especially among the more severely depressed patients. The Agency for Health Care Policy and Research conclusion appears to have relied heavily on data from the Elkin et al. study (6), as well.

The inferences embedded in these guidelines have been questioned by Persons et al. (10), as well as by Jacobson and Hollon (11, 12) in a debate with Elkin et al. (13) and Klein (7). Jacobson and Hollon and Persons et al. have argued that findings from other randomized trials of antidepressant medication and cognitive behavior therapy (1, 2, 14) should be considered alongside those of Elkin et al. (6). One of these studies (14) reported a separate comparison of cognitive behavior therapy and antidepressant medication in severely depressed patients. Using criteria and measures employed by Elkin et al. (6), Hollon et al. (14) found a very small, nonsignificant advantage for cognitive behavior therapy in their more severely depressed subgroup. This finding is at odds with the Treatment of Depression Collaborative Research Program finding that antidepressant medication outperformed cognitive behavior therapy among the more severely depressed patients (15).

Given the discrepancy between the studies by Elkin et al. (6) and Hollon et al. (14), a meta-analysis that combines outcome data from additional sources would advance the debate. Besides the Hollon et al. study, the studies comparing antidepressant medication and cognitive behavior therapy that were most similar to the Elkin et al. study (6) were those conducted by Rush et al. (1) and Murphy et al. (2), but neither reported separate findings for their more severely depressed subgroups. Thus, we enlisted the cooperation of the investigators and obtained the data required to perform the necessary subset analyses. We also obtained raw data from the Hollon et al. (14) and Elkin et al. (6) studies, allowing us to compare the efficacy of antidepressant medication and cognitive behavior therapy for the more severely depressed patients in each of these four studies. We thus were able to pool the individual subject data from the four studies, to conduct meta-analyses with original data (16), also known as mega-analyses. Mega-analytic methods were employed because when original data are available, they are preferred to standard meta-analyses, especially when there are only a few relevant studies available and when subgroup analyses are desired (17).

In the recent debate, Elkin et al. (13) suggested that the discrepancy between the Treatment of Depression Collaborative Research Program (6) and the other studies, in particular, that of Hollon et al. (14), was due to the superior performance of the antidepressant medication condition in the Treatment of Depression Collaborative Research Program. In contrast, Jacobson and Hollon (12) have questioned the performance of the cognitive behavior therapy condition in the Treatment of Depression Collaborative Research Program. With additional data from the severely depressed patients of Rush et al. (1) and Murphy et al. (2), we can now compare the outcomes of the antidepressant medication and cognitive behavior therapy conditions of the Treatment of Depression Collaborative Research Program, relative to what was obtained in the other studies.

METHOD

Patients from the studies by Rush et al. (1), Murphy et al. (2), and Hollon et al. (14) were chosen for analysis alongside patients from the Elkin et al. study (6) because, like the last study, those three studies met the following criteria. 1) Adult outpatients were randomly assigned to cognitive behavior therapy or antidepressant medication. (Studies with additional conditions, such as combinations of cognitive behavior therapy and antidepressant medication, were not excluded.) 2) Patients were diagnosed with depression according to either the Feighner criteria (18) (in the Rush et al. [1] and Murphy et al. [2] studies) or the Research Diagnostic Criteria (RDC) (19) (in the Hollon et al. [14] and the Elkin et al. [6] studies). 3) Patients were assessed with both the 17-item Hamilton Rating Scale for Depression (20) and the Beck Depression Inventory (21) before and after treatment. (In all studies, symptom severity was assessed at intermediate time points as well, allowing the use of last-value-carried-forward analyses for dropouts.) 4) Individual subject data were available, permitting the examination of only those subjects who met specific severity criteria before treatment. 5) Severely depressed adults (Hamilton depression scale score of 20 or more or Beck Depression Inventory of 30 or more) were present in the study group. A list of differences among the studies is given in Table 1.

In the interest of simplicity and because in the antidepressant medication/cognitive behavior therapy outcome literature, analyses of “all assigned” patients have been preferred to analyses only of patients who complete trials, we will present data from the all assigned subgroups only. The pattern of findings we will report is similar to that obtained in parallel analyses of only patients who completed trials, the results of which can be obtained from Dr. DeRubeis.

We used two criteria to select two (overlapping) sets of severely depressed patients (Table 1). One set is defined by an initial Hamilton depression scale score (20 or more), as employed in both the original Elkin et al. report (6) and the report that focused on the severity question (15). The other cutoff is defined by a pretreatment Beck Depression Inventory score (30 or more), as employed in the latter article (15). The group defined by the Hamilton depression scale severity criterion is always analyzed by using pre- and posttreatment Hamilton depression scale scores. Similarly, the group defined by the Beck Depression Inventory severity criterion is always analyzed by using pre- and posttreatment Beck Depression Inventory scores.

The advantages of using the Hamilton depression scale are twofold. 1) It was the primary depression assessment in the Elkin et al. study (6). 2) Unlike the Beck Depression Inventory, which has been associated historically with cognitive behavior therapy and cognitive behavior therapy research, the Hamilton depression scale has not been associated specifically with either treatment.

The advantages of using the Beck Depression Inventory are also twofold. 1) Because it is a self-report measure, it is immune to a confound that limits the validity of cross-study comparisons of interviewer-based measures—differences in the sensitivities or biases of interviewers. 2) In the Hollon et al. study (14), the use of the Hamilton depression scale seriously underestimates outcome for the dropouts, whereas the use of the Beck Depression Inventory does not. Because most of the dropouts in the Hollon et al. subgroup left treatment before the midtreatment assessment, their “final” Hamilton depression scale score was “carried forward” from the pretreatment assessment. Fortunately, Beck Depression Inventory scores were available from each session, allowing estimates of end-point symptom severity at a time close to the time of dropout. The mean Beck Depression Inventory score for the 24 of 27 subjects who dropped out before the midtreatment assessment decreased from 30.0 before treatment to 20.6 by the time of dropout. This symptom reduction in dropouts in the Hollon et al. study (14) was reflected in Beck Depression Inventory analyses but not in Hamilton depression scale analyses. The other studies assessed subjects with the Hamilton depression scale before midtreatment; pretreatment data were therefore used much less frequently as posttreatment data.

RESULTS

Pretreatment Hamilton Depression Scale and Beck Depression Inventory Scores

Mean pretreatment scores on the Hamilton depression scale and Beck Depression Inventory are shown in Table 2, for the purpose of revealing similarities and differences between the study groups. Two-factor analyses of variance (four [study] by two [treatment]) were conducted on pretreatment Hamilton depression scale scores, as well as on pretreatment Beck Depression Inventory scores. For the Hamilton depression scale, there was a main effect of study (F=5.29, df=3, 161, p<0.002), but there was no effect of treatment (F=0.99, df=1, 161, p=0.32) or of the study-by-treatment interaction (F=0.24, df=3, 161, p=0.87). In t test analyses, least squares means in the Hollon et al. study (14) were significantly higher than those in each of the other studies (against Rush et al. (1), t=2.74, df=92, p<0.01; against Murphy et al. (2), t=2.31, df=78, p<0.03; against the Elkin et al. [6] study, t=3.28, df=119, p<0.002). Whereas the least squares means in the other three studies ranged only between 23.3 and 23.5, the least squares mean in the Hollon et al. study was 25.4. This difference of 2 points on the Hamilton depression scale, although significant, was judged insubstantial. Moreover, since the difference cut across treatments, it should not influence comparisons of antidepressant medication and cognitive behavior therapy.

On the Beck Depression Inventory, neither the effect of study (F=0.40, df=3, 124, p=0.75) nor of treatment (F=3.13, df=1, 124, p=0.08) was significant. The study-by-treatment interaction was significant (F=3.19, df=3, 124, p<0.03). In the Murphy et al. study (2), the least squares mean for antidepressant medication was found to be significantly higher than that for cognitive behavior therapy (t=3.04, df=56, p<0.003). Thus, the randomization process did not produce—among Murphy et al.’s study group, defined as severely depressed by the Beck Depression Inventory—equivalent antidepressant medication and cognitive behavior therapy groups. In the Murphy et al. study, the least squares mean for antidepressant medication was also higher than that in the studies by Rush et al. (1) (t=2.88, df=21, p<0.005) and Elkin et al. (6) (t=2.06, df=30, p<0.05). In part because of these differences, all analyses of posttreatment scores will be adjusted for pretreatment scores.

Attrition

Table 2 also displays the proportion of patients who dropped out in each treatment within each study. In order to determine whether rates of attrition differed across studies, or whether attrition occurred differentially as a function of treatment across studies, we conducted four-by-two logistic regressions for the Hamilton depression scale and Beck Depression Inventory groups, with attrition status (completed versus dropped out) as the dependent variable and study, treatment, and the study-by-treatment interaction as predictors.

In the Hamilton depression scale group, the study-by-treatment interaction was not significant (χ²=4.01, df=3, p=0.58). In a model that excluded the interaction term, the effect of treatment was not significant (χ²=1.25, df=1, p=0.47), but there was a study effect (χ²=8.00, df=3, p<0.05). In follow-up pairwise comparisons, only the difference between the Rush et al. (1) and Elkin et al. (6) studies was significant (p=0.01, Fisher’s exact test); dropout was lower in the Rush et al. study (15%) than in the Elkin et al. study (42%).

In the Beck Depression Inventory group, the study-by-treatment interaction was not significant (χ²=3.73, df=3, p=0.29). In a model that excluded the interaction term, neither the effect of treatment (χ²=2.59, df=1, p=0.11) nor of study (χ²=2.05, df=3, p=0.56) was significant. Thus, there is no evidence that differential patterns of attrition threaten the internal validity of conclusions about outcome.

Outcome of Antidepressant Medication Versus Cognitive Behavior Therapy

Figure 1 and Figure 2 display the outcome comparisons of antidepressant medication and cognitive behavior therapy for each study’s severely depressed subgroups. We conducted separate analyses of covariance (ANCOVAs) for each study, on both the Hamilton depression scale and Beck Depression Inventory, with posttreatment Hamilton depression scale (or Beck Depression Inventory) as the dependent variable, pretreatment Hamilton depression scale (or Beck Depression Inventory) as the covariate, and treatment as the independent variable. When the Bonferroni correction was applied for eight separate comparisons of antidepressant medication and cognitive behavior therapy (four studies by two outcome measures), none of the differences was significant. The effect sizes (Cohen’s d [23]) of the within-study comparisons, along with the half-widths of two-tailed 95% confidence intervals (CIs) (24, p. 86), are displayed in Table 3.

Before pooling the patients from the four studies, we conducted analyses to test for heterogeneity in the relative effects of treatments in the four studies. These tests are analogous to tests of the heterogeneity of effect sizes in standard meta-analyses. For both the Hamilton depression scale and Beck Depression Inventory data, we conducted two-way ANCOVAs, with posttreatment Hamilton depression scale (or Beck Depression Inventory) as the dependent variable and pretreatment Hamilton depression scale (or Beck Depression Inventory) as the covariate. Study (four levels), treatment (antidepressant medication versus cognitive behavior therapy), and the study-by-treatment interaction were entered as factors.

In these analyses, the study-by-treatment interaction did not approach significance in the Hamilton depression scale analysis (F=1.52, df=3, 160, p=0.21) or in the Beck Depression Inventory analysis (F=1.37, df=3, 123, p=0.25). This permitted the pooling of patients from the four studies, so that each patient was given equal weight.

In Figure 1, the right-most pair of bars depicts the pooled comparison of antidepressant medication and cognitive behavior therapy, which used Hamilton depression scale data from all four studies, adjusted for pretreatment severity across all patients. Figure 2 displays this comparison for the Beck Depression Inventory data. The corresponding effect sizes for the pooled Hamilton depression scale and Beck Depression Inventory data are given in the bottom row of Table 3. Effect sizes for the pooled data were negative (favoring cognitive behavior therapy) in both the Hamilton depression scale and Beck Depression Inventory subgroups. However, tests comparing the adjusted means were nonsignificant both in the Hamilton depression scale subgroup (t=0.43, df=166, p=0.67) and in the Beck Depression Inventory subgroup (t=1.25, df=129, p=0.21). The 95% CI extended from –0.38 to 0.24 in the Hamilton depression scale group and from –0.56 to 0.12 in the Beck Depression Inventory group.

Comparison of Treatment of Depression Collaborative Research Program With the Other Studies

In order to address arguments that outcomes in the Treatment of Depression Collaborative Research Program’s antidepressant medication and cognitive behavior therapy conditions (6) differed from those obtained in the other studies (11, 13), we examined each of the two treatment conditions separately. We conducted ANCOVAs, with posttreatment Hamilton depression scale (or Beck Depression Inventory) as the dependent variable, initial Hamilton depression scale (or Beck Depression Inventory) as a covariate, and study as a factor.

For the antidepressant medication condition, for the Hamilton depression scale, the effect of study was not significant (F=1.75, df=3, 97, p=0.16). A planned contrast between the Elkin et al. study (6) and the other studies was also not significant (F=0.51, df=1, 97, p=0.48). On the Beck Depression Inventory, neither the overall effect (F=0.04, df=3, 69, p=0.99) nor the planned contrast of the Elkin et al. study with the other studies (F=0.00004, df=1, 69, p=1.00) was significant. As shown in Table 4, the comparisons between the antidepressant medication condition of the Elkin et al. study (6) and those of the other studies were in favor of the Elkin et al. study, but the effect sizes were quite small.

For the cognitive behavior therapy condition, for the Hamilton depression scale, the effect of study was significant (F=3.21, df=3, 62, p=0.03). A planned contrast between the Elkin et al. study (6) and the other studies just failed to reach significance (F=3.95, df=1, 62, p=0.05). On the Beck Depression Inventory, the overall effect was not significant (F=2.33, df=3, 53, p=0.09); the planned contrast of the Elkin et al. study (6) with the other studies just failed to reach significance (F=4.01, df=1, 53, p=0.05). Table 4 displays, in effect size terms, results that correspond to the contrasts of the Elkin et al. study with the other studies.

To summarize, the antidepressant medication condition of the Elkin et al. study (6) did not produce outcomes superior to those obtained in the other studies, and there was no conclusive evidence that the cognitive behavior therapy condition in the Elkin et al. study produced poorer outcomes than did the cognitive behavior therapy conditions of the studies by Rush et al. (1), Murphy et al. (2), and Hollon et al (14).

DISCUSSION

The combined results from four randomized clinical trials do not support the inference that antidepressant medication is superior to cognitive behavior therapy for severely depressed outpatients. The analyses included the Treatment of Depression Collaborative Research Program data (6), upon which treatment guidelines have been based, along with data from three other major comparisons of antidepressant medication and cognitive behavior therapy.

Until we conducted the present analyses, the only authors who had specifically compared antidepressant medication and cognitive behavior therapy among severely depressed outpatients were Elkin et al. (15), who found antidepressant medication to be superior to cognitive behavior therapy, and Hollon et al. (14), who reported a small but nonsignificant advantage for cognitive behavior therapy over antidepressant medication. Elkin et al. (13), referring to Hamilton depression scale data, observed that the apparent discrepancy between the Treatment of Depression Collaborative Research Program (15) and the Hollon et al. study (14) could be understood as arising from an excellent performance of the antidepressant medication condition in the Treatment of Depression Collaborative Research Program, compared to a lesser performance of the antidepressant medication condition in the Hollon et al. study. As we have argued, in the study of Hollon et al., the use of the Hamilton depression scale in the “all assigned” group is problematic, since it was necessary to use the pretreatment Hamilton depression scores of 18 of the 20 dropouts as their posttreatment scores. When we eliminated this problem by analyzing the Beck Depression Inventory, the mean antidepressant medication outcome in the Hollon et al. study was very similar to that observed in the Treatment of Depression Collaborative Research Program. And as shown in Figure 1 and Figure 2, the Hamilton depression scale and Beck Depression Inventory outcomes of the Treatment of Depression Collaborative Research Program’s antidepressant medication condition (6) are also similar to those obtained in the two other severely depressed subgroups we have reported on. Thus, among severely depressed patients, it appears that results from the antidepressant medication condition of the Treatment of Depression Collaborative Research Program were unexceptional, relative to those obtained in other trials of antidepressant medication versus cognitive behavior therapy. However, neither do the present results offer firm support for the suggestion that the cognitive behavior therapy condition of the Treatment of Depression Collaborative Research Program was inferior to cognitive behavior therapy conditions in other studies.

In the initial analyses of Treatment of Depression Collaborative Research Program data by Elkin et al. (6), significant differences were not obtained between their antidepressant medication and cognitive behavior therapy conditions in the severely depressed subgroup. In subsequent analyses of Treatment of Depression Collaborative Research Program data, using random regression methods that exploit data collected at points between the beginning and end of treatment, Elkin et al. (15) found antidepressant medication to be significantly superior to cognitive behavior therapy in the severely depressed subgroup. We did not use random regression methods in the present case in part because intermediate data are not available for all studies. One difference between random regression methods and traditional pretreatment/posttreatment analyses is that the former are typically more sensitive to differences between conditions; that is, these methods are more powerful. Because the main finding in the present report is that across the four studies, cognitive behavior therapy was (negligibly and nonsignificantly) more effective than antidepressant medication, the addition of power, on its own, could not reverse this so as to show antidepressant medication to be superior in this collection of studies.

Other differences between random regression and traditional methods could lead to different results. In particular, random regression methods take into account the course of change during treatment. Indeed, in the Treatment of Depression Collaborative Research Program, the more rapid early effects of antidepressant medication, relative to cognitive behavior therapy, were in part responsible for the significant differences reported (13). Ilardi and Craighead’s analysis (25) of cognitive behavior therapy time-course data suggests that the cognitive behavior therapy condition of the Treatment of Depression Collaborative Research Program yielded an exceptionally slow rate of change in the early weeks of treatment, relative to other studies, including those of Rush et al. (1), Murphy et al. (2), and Hollon et al. (14). Moreover, in the total groups of the Murphy et al. (2) and Hollon et al. (14) studies, there appears to have been no such tendency for antidepressant medication to produce more rapid early change than cognitive behavior therapy (Figure 2 and Figure 1, respectively, in the respective articles). Thus, it is unlikely that random regression analyses of data from this collection of studies would reveal an advantage of antidepressant medication over cognitive behavior therapy.

As Klein (7) has pointed out, the superiority of antidepressant medication to placebo in the Treatment of Depression Collaborative Research Program allows the inference that that study’s antidepressant medication condition was effective. The absence of placebo conditions in the studies by Rush et al., Murphy et al., and Hollon et al. leaves open the possibility that their groups were not responsive to antidepressant medication. This argument is especially cogent in the context of groups that include less severely depressed patients (e.g., those with Hamilton depression scores below 20). In pharmaceutical trials, significant differences between drug and placebo are more likely to be found in moderately or severely depressed than in mildly depressed groups (26, 27). Thus, had there been placebo conditions in the studies by Rush et al., Murphy et al., and Hollon et al., the less severely depressed patients might have yielded null differences between drug and placebo, as occurred in the Treatment of Depression Collaborative Research Program (6). But, just as differences between drug and placebo were obtained in the more severely depressed subgroup of the Treatment of Depression Collaborative Research Program, it is reasonable to expect that Rush et al., Murphy et al., and Hollon et al. would have obtained them as well, although this cannot be known with certainty.

Unreplicated results from a single study, even one as large and as well controlled as the Treatment of Depression Collaborative Research Program, cannot determine the relative efficacy of antidepressant medication and cognitive behavior therapy for severely depressed outpatients. Until more data become available, we recommend that the field recognize, and treatment guidelines reflect, the findings obtained thus far in the major comparisons of antidepressant medication with cognitive behavior therapy: that antidepressant medication and cognitive behavior therapy have not differed in acute efficacy in the treatment of severely depressed outpatients.

Presented at the ninth annual meeting of the American Psychological Society, Washington, D.C., May 23–25, 1997. Received March 19, 1998; revision received Sept. 15, 1998; accepted Jan. 20, 1999. From the Departments of Psychology and Psychiatry, University of Pennsylvania. Address reprint requests to Dr. DeRubeis, Department of Psychology, University of Pennsylvania, 3815 Walnut St., Philadelphia, PA 19104-6196; [email protected] (e-mail). Supported in part by NIMH grants MH-45178 and MH-55877. The authors thank Steven Hollon, Ph.D., for providing access to the Rush et al. database and Irene Elkin, Ph.D., for her assistance with this research.

TABLE 1

Enlarge table

TABLE 2

Enlarge table

TABLE 3

Enlarge table

TABLE 4

Enlarge table