The Evolution of Meta-Analysis in Psychotherapy Research

Psychodynamic psychotherapies have a smaller evidence base than do cognitive-behavioral treatments (1). However, their study quality is not worse, and as with all psychotherapy research, quality improves with year of publication (1, 2). Nonetheless, the relative lack of research into outcomes of psychodynamic therapy, along with politicization of psychotherapy guilds (3, 4), has led to its increasing marginalization and endangerment (3). It could conceivably become difficult for patients to find dynamic treatment in the relatively near future if this problem is not addressed.

Christiane Steinert and colleagues (5) have contributed a ground-breaking meta-analysis evaluating the equivalence of efficacy of psychodynamic psychotherapies with other efficacious treatments “for mental disorders.” In as well-controlled a meta-analysis as any to date, the authors demonstrate equivalence of psychodynamic psychotherapy with other active treatments, a finding with important public health implications. This brings potential hope for improved patient access to dynamic therapy (6). As the authors note in their conclusion, however, all treatments for mental disorders have a wide margin for improvement, as rates of response and remission remain relatively modest.

At first blush, it might seem that addressing such a broad question—efficacy of psychodynamic psychotherapy for “mental disorders”—which these authors have been attempting to do for years (7), is too far reaching. Indeed, it is: the 23 randomized controlled trials in the meta-analysis cover six diagnostic categories: mood, anxiety, posttraumatic stress disorder, eating, personality, and substance use disorders, but not psychosis or the fuller range of psychopathologies. Although the study has limitations, it nevertheless represents an important advance for two reasons. The primary reason is that, for the first time, a meta-analysis of psychotherapies has incorporated into its analytic approach measures of psychotherapy study quality and researcher allegiance, two design flaws in past meta-analyses of psychodynamic psychotherapy (8). Secondly, this is the first well-enough conducted meta-analysis of psychodynamic psychotherapy efficacy for multiple mental disorders to generate credible conclusions about its equivalence with other efficacious treatments: cognitive-behavioral therapy (CBT) and selective serotonin reuptake inhibitors. The study findings lend additional support to the growing body of randomized controlled trial evidence demonstrating efficacy equivalence of dynamic therapies with CBT, particularly for depression (9, 10).

Meta-Analyses and Their Limitations

Research into psychotherapy outcomes is a cottage industry that depends on scarce, hotly contested, and increasingly vanishing federal and international funds. Studies are expensive and require meticulous stewardship. They are perforce relatively small, even when supported by grants from the National Institutes of Health or the European Union. These and other operational challenges make it difficult to draw broad conclusions about the effectiveness of various psychotherapies on the basis of even the largest individual studies. Hence, meta-analysis is an approach that can apparently address the problem of small samples: Why not combine outcomes from a large group of studies so as to have a larger N to evaluate? This seemingly sensible approach hides a world of compromise. Few psychotherapy studies, even for the same target disorder, boast identical study entrance criteria or research benchmarks monitoring “response” or “remission,” so combining cross-study data can lead to inaccuracies. When meta-analyses include studies addressing outcomes for a range of psychiatric disorders, tracking outcomes across various disorders and outcome measures can become a free-for-all, a progressively messy endeavor with the addition of each target illness (7). The Steinert study suffers from this drawback, despite the study’s relative strengths.

It is even more problematic that a meta-analysis, or any large-scale lumping of data derived from various individual studies, has heretofore not incorporated formal psychotherapy-specific measures of study quality, thereby diluting data from less biased studies so they carry the same weight as heavily biased, poor-quality studies (6). It is worth noting that Cuijpers et al. (2), who have conducted more than 100 meta-analyses of the effects of various treatment interventions on a wide range of psychiatric and psychological problems, employ the Cochrane study quality measure (11). However, this measure is not specifically designed to track a range of potential bias encountered in psychotherapy outcome studies as accurately as the Randomized Controlled Trial Psychotherapy Quality Rating Scale (12) does, which Steinert et al. (5) employ. Thus, meta-analytic findings are less reliable as to true efficacy of specific interventions than are individual, well-conducted randomized controlled trials of very good quality. The incorporating of investigator bias into the relative weight that a study carries also has not been done before in meta-analyses evaluating the efficacy of psychodynamic psychotherapy. It is an important improvement.

Additional advances in the Steinert study include considering only psychotherapy studies with active comparison interventions (either CBT or medication) and focusing primary analyses only on primary, standard, accepted measures of symptomatic improvement, albeit across disorders. These advances minimize the usual meta-analytic “noise.”

A chief problem of psychotherapy research meta-analyses is study inclusion criteria. Such choices, presumably determined from an a priori rationale, profoundly influence findings (13). This study, like all meta-analyses, suffers from this problem, as the authors note. Specifically, inclusion of several older studies (five published before the late 1990s) means that untracked medications and psychotherapy interventions during the intervention and follow-up periods were likely to have had a diluting, unreported effect on outcomes, a ubiquitous problem in older psychotherapy research literature before study quality improved (2, 12, 14). Including outcome studies across “mental disorders” is an overly broad target and is not ideal for a meta-analytic approach. Nonetheless, in light of dwindling funding around the world for psychotherapy outcome studies in the foreseeable future, the authors can be forgiven this.

Conclusions

It is heartening that in this first truly well-conducted meta-analysis designed to limit poor study quality and allegiance bias, dynamic psychotherapies were found equivalent to other efficacious treatments across a range of mental disorders. This study, by design, cannot engage specific treatment approaches within the varied “psychodynamic” domain. What we can do to improve our outcomes must become a research focus. In the future, improvements in these outcomes may well be achieved through better incorporation of recent mediation and mechanism findings about specific dynamic therapy approaches and targets, thereby helping clinicians to hone even more active, useful patient interventions (4).