A Meta-Analysis of Cognitive Remediation for Schizophrenia: Methodology and Effect Sizes

As defined at the Cognitive Remediation Experts Workshop (Florence, Italy, April 2010), cognitive remediation therapy for schizophrenia is “a behavioral training based intervention that aims to improve cognitive processes (attention, memory, executive function, social cognition or metacognition) with the goal of durability and generalization.” Its development was fueled by a number of studies demonstrating links between cognition and future functioning (1), the effects on rehabilitation outcomes (1–6), and from complaints by service users themselves about the interference with their everyday lives.

A large body of data on the efficacy of cognitive remediation therapy has been produced, and a number of meta-analyses have shown moderate to large effects on cognitive outcomes (7–13). All reviews pointed to variability in cognitive remediation technology, in the target population, and in the targeted outcomes (11, 14). Some have even suggested that methodology might bias the effect estimates and inflate their effects. Previous reviews also have indicated that trials of cognitive remediation have not sufficiently addressed functioning outcomes, since few have involved these measures and even fewer have examined longer-term effects. Following these criticisms, remediation experts have produced a series of studies that address functioning, almost doubling the amount of available studies.

Cognitive remediation therapy aims to enhance cognition with a further goal that improved cognition will affect community functioning. There have been historic concerns about identifying appropriate cognitive targets. Should we choose those that are more malleable or that have the greatest chance of boosting functioning (15)? Previous reviews indicate that nearly all cognitive domains, except visual learning and memory, respond to remediation in the treatment of schizophrenia (e.g., 9), and thus there is no indication of more malleable cognitive targets. Heterogeneity of cognitive difficulties is also well established in schizophrenia (16), and therefore a broader approach that targets multiple domains may be of benefit to the most patients (17). There is also no indication that boosting one cognitive domain rather than another might improve community function (e.g., 18). Given these findings, we consider the key challenge at this stage of the field's development to be an understanding of which therapeutic techniques will provide the most benefit (9). While cognitive remediation therapies all involve massed practice on training tasks, there are key differences in how therapies attempt to bring about cognitive benefits. The greatest difference emerges between studies that have taken a strategy-based approach and a drill and practice approach. A previous meta-analysis (9) found that drill and practice had a greater effect on cognition than a strategy-based approach. However, the reverse was found for functioning.

To determine the most effective treatment approach for key outcomes, it is crucial to identify whether particular groups of patients benefit more (or less) from cognitive remediation therapies in order for scarce resources to be deployed to the greatest effect. There are a number of reasons why participant characteristics are likely to limit the treatment effect. First, age has been found to be a predictor of outcome in two studies (19, 20) and could be an important moderator variable given that some studies have recruited from markedly older populations (21) relative to others (22, 23). Baseline symptoms have been cited as a barrier to engagement in cognitive-behavioral therapy (CBT) for psychosis trials (24, 25). It is conceivable that cognitive remediation therapy could be similarly affected by patients who are distracted by symptoms or distressed.

The methodology of cognitive remediation studies has been regarded by both the National Institute for Clinical Excellence and the Patient Outcomes Research Team as a problematic factor in judging the effect size of studies (26, 27). A lack of rigor is known to affect the treatment effect estimates in both medication and psychological treatment trials of people with schizophrenia (28, 29). In particular, poorer quality masking of allocation to treatments is associated with up to a 40% increase in estimated treatment benefit in mental health studies as well as for studies in circulatory and digestive diseases and in obstetrics. A recent systematic review of CBT for psychosis showed the treatment effect inflation to be 50%–100% (29). To date, no meta-analysis of cognitive remediation has investigated methodological rigor. Since it is now on the brink of being incorporated into health services, such an analysis is essential not only to guide current service providers but also to help new research.

The present meta-analysis assesses treatment effects using double the number of studies included in the most recent analysis, and thus we could investigate more moderating effects of treatment and patient characteristics. However, our analysis is also novel in its investigation of the internal and external validity of the studies driving these treatment effects.

Method

Data Sources

Multiple systematic searches up to June 2009 were conducted using Embase, MEDLINE, Current Contents, Web of Science, PsycINFO, and the Cochrane Collaboration Controlled Trials Register. The following search terms were used as either key terms or keywords: (“cognitive” or “cognit*”) AND (“training” or “remediation” or “rehabilitation” or “enhancement”) AND “schizophrenia” AND (“random” or “randomized control trial” or “clinical trial”). The reference lists of articles fulfilling all criteria were hand-searched for other relevant studies. In addition, members of the Cognitive Remediation Experts Working Group who represent English-, French-, German-, Spanish-, and Italian-speaking countries were contacted regarding studies that did not appear in our literature searches.

Study Inclusion

All articles with an English abstract were included, producing 109 reports that were examined to identify the following criteria: 1) intervention fulfilling the standard Cognitive Remediation Experts Workshop definition for cognitive remediation; 2) a majority (≥70%) of participants with a diagnosis of schizophrenia; 3) all participants receiving standard care, including appropriate medication; 4) a comparison group and allocation procedure; and 5) a cognitive or functional outcome distinct from the trained tasks. These criteria excluded 46 reports (criterion 1, N=5; criterion 2, N=2; criterion 4, N=20; criterion 5, N=6; [13 reports provided inappropriate or insufficient data, which could not be resolved through contact with authors]), leaving 39 key reports of 40 independent studies among the remaining 63 reports.

Data Extraction

Trial details for deriving effect sizes as well as other variables were extracted by two authors of the present review (Drs. Huddy and Cellard). Uncertainty concerning a study was resolved through contact with the study authors. Few assumptions were made, with the exception that effect sizes were chosen when all treatment was concluded and follow-up effect sizes were calculated on the basis that there was a period with no treatment.

Measures

Trial quality

Trial quality was assessed using the Clinical Trials Assessment Measure (29, 30), a 15-item reliable and valid measure of trial methodology for psychological treatment studies. It assesses 1) sample characteristics (geographical cohort, referred sample, etc.); 2) allocation to treatment (randomization, blind allocation, etc.); 3) comparison treatments (e.g., a comparison group for nonspecific treatment effects); 4) outcome assessments (e.g., standardized outcomes, participants unaware of group allocation); 5) treatment description (e.g., a protocol, fidelity assessment); and 6) appropriate analysis (e.g., intention to treat). The maximum score is 100.

Two authors independently rated the studies with high reliability (intraclass coefficient=0.9). We sent all agreed-upon ratings to the study authors for approval. In cases where there was disagreement, supplementary evidence was requested. Only three study authors failed to reply. Following author feedback and evaluation of supplementary information, 57% of total Clinical Trial Assessment Measure scores changed by an average increase of 7.3 points.

Putative treatment moderators

Putative moderators were study vintage (years since publication date), therapy duration, computer presentation, presence of adjunctive psychiatric rehabilitation, and type of therapy (drill and practice versus drill plus strategy training as reviewed by McGurk et al. [9]). In drill and practice, the main focus is to engage cognitive processing by training on progressively more difficult exercises. Some researchers assume that this leads to more efficient neural responses and facilitates other aspects of cognition (e.g., 31). In drill and practice, there is no explicit input on how tasks might be more effectively completed. Adopting more efficient processes is determined by chance as participants try (or do not try) different approaches. In contrast, the drill plus strategy approach involves an explicit focus on teaching strategy use (e.g., initially identifying situations that trigger cognitive problems, such as memory lapses, then learning strategies, such as mnemonics, to aid remembering in therapy and then transferring these skills to everyday life). Studies were only categorized as drill plus strategy if they had some training procedures involving how to carry out tasks.

Putative patient characteristic moderators

There is evidence of potential effects of age (19, 20) and symptoms (24, 25). Data for age were taken from the mean ages in the studies. For symptoms, we converted baseline Positive and Negative Syndrome Scale and Brief Psychiatric Rating Scale scores (N=26) to Clinical Global Impression scores, using the Leucht et al. procedure (32).

Skewness

All continuous moderator variables were examined for skewness, and both study vintage and treatment duration were skewed. Treatment duration was corrected using square-root transformation. However, study vintage was affected by an outlier study (33) that was published 16 years before the next oldest study. When this outlier was removed, the study vintage variable was no longer skewed, and thus it was investigated without any transformation (except removal of the outlier).

Meta-Analytic Procedure

Effect size calculation

Effect sizes (Cohen's d) of cognitive skill, symptom, and functional differences were calculated using the following equation:

Effect size (d)=(M_t–M_c)/SD_pooled

The M_c indicates the mean for the comparison group, and SD_pooled indicates the pooled standard deviation for the two groups.

The estimate of the standard error of the effect size (Cohen's d) was calculated using the following formula:

SE(d)=sqrt([n_t+n_c]/[n_t×n_c])]+(d×2/[2×(n_t+n_c–2)])

The n_t and n_c denote the sample size for both the treatment and comparison groups (see Cooper and Hedges [34]).

The analysis default was to use the active comparison group whenever possible. If the group means were not available, analysis of variance F values that covaried baseline scores or t values were used to derive effect sizes using methods described by Thalheimer and Cook (35).

Primary outcomes

No therapy targets a sole cognitive domain, and cognitive outcomes reflect processing in several cognitive domains. We therefore designated the primary outcome as the global cognitive effect (averaged across all reported cognitive outcomes) that best represents the target for cognitive remediation therapy. This method has been used in most other meta-analyses (e.g., 7, 9, 36). All extracted measures of cognition, symptoms, and functioning are described in Table 1 of the data supplement accompanying the online version of this article. There were 99 cognitive outcomes at posttreatment refined into the Measurement and Treatment Research to Improve Cognition in Schizophrenia (37) group domains. Composite effect sizes were derived by averaging the available effect sizes of individual measures making up the composite.

Composites do not take into account covariation and constitute a conservative estimate, since the covariance would increase the composite effect size (38) so that it would always be greater than that of single measures. To establish whether composite measures were in the same range, we contrasted them with individual tests (Digit Span task, Wisconsin Card Sorting Test, Trail Making Test, and Continuous Performance Test) that were used in more than 10 studies.

Symptoms were reported at baseline in 29 studies and at outcome in 20 studies using eight measures. Functioning was reported in 19 studies using 16 measures (see Table 1 in the data supplement). Fourteen separate meta-analyses were carried out for posttreatment effects (global cognition, N=1; cognitive domains, N=7; individual cognitive measures, N=4; symptoms, N=1; and functioning, N=1). Three additional analyses were conducted for durability of global cognition, symptoms, and functioning.

Statistical procedures

Descriptive statistics and methodological quality were explored to investigate changes over time or specific biases. Meta-analyses used random-effects models and weighted effect sizes and were conducted with STATA software (StataCorp, College Station, Tex.) (39). The chi-square value of the homogeneity of effects was determined with the Q statistic (40). Study effect sizes were examined to identify whether any pretreatment cognitive differences biased the results or whether they were driven by outlier effects.

Mediator and moderator effects

When there was significant heterogeneity and at least 20 studies that provided data, metaregression analyses were carried out. The following individual moderators were considered: methodological rigor (Clinical Trial Assessment Measure variables [total score, masking, comparison group, and randomization]); participant characteristics (age, gender, proportion of participants with a confirmed diagnosis of schizophrenia, baseline symptom severity, impaired cognition as an inclusion criterion); and treatment characteristics (length, therapy type [drill and practice versus drill plus strategy], computer use, adjunctive rehabilitation). Treatment and patient moderator analyses were also conducted after covarying for methodological rigor.

Results

Corpus of Studies

Thirty-nine separate reports of 40 studies, with 2,104 participants, fulfilled all inclusion criteria. Summary characteristics of included studies are provided in Table 2 of the data supplement.

Sample characteristics

Study samples consisted of individuals in their mid-thirties (mean age=35.8 years [SD=7.1], range=15.3–48.3) who were mostly men (N=37, mean=67% [SD=14.2], range=30%–100%) with approximately 11.7 years of education (N=27, range=10.2–13.4). The proportion of inpatients and outpatients was balanced (47% versus 45%, respectively), with 7% of studies recruiting from both in- and outpatients. Symptom severity, when reported (N=26), was in the mild to moderate range. Within-study ranges suggested that some studies included more severely symptomatic individuals.

Study characteristics

Sample sizes were small (mean=53, range=10–145), with larger sample sizes appearing among the more recent studies ([N=40] Spearman's rho=0.42, p=0.007). Most studies were carried out in the United States (N=21), although 11 countries were represented. The average number of cognitive domains measured was 3.4, with a range of 1–6. Studies concentrating on only one domain appeared among the earliest studies and measured mainly speed of processing and attention/vigilance.

Treatment characteristics

Thirty-one studies used individual remediation and a total of nine group treatments. One-quarter of therapy involved the use of drill and practice exercises on a computer, without additional psychiatric rehabilitation. For computerized studies, it is not clear how much time was spent with a therapist, and author feedback indicates that this varied by participant. Fourteen different treatments were represented, with only three (21, 31, 41) not being replicated or independently replicated.

Only three studies (2, 31, 42) paid participants to undergo therapy, and in each of these payment was made to participants in both the experimental and comparison groups. In the Bell et al. (2) and Sartory et al. (42) studies, payment was made so that the experimental treatment group would not lose money, since payment was contingent upon taking part in vocational rehabilitation. Overall, the attrition rate was 11.0%, with a range of 0%–47.5%.

Twenty-one studies used drill and practice, and 19 used drill plus strategy. The average length of treatment was 32.2 hours (range=4–130), provided across 16.7 weeks (range=2–104). Therapy intensity was 2.2 sessions per week (range=0.6–5). Drill and practice treatments were shorter than drill plus strategy training (Mann-Whitney U test: hours: U=114, p=0.02; weeks: U=109, p=0.01), but the two approaches did not differ in intensity. Among studies using drill and practice, those that employed a computer-assisted program outnumbered those that did not (N=15 versus N=6, respectively), but the opposite was true for studies using drill plus strategy training (N=8 versus N=11). However, the difference was not statistically significant. Few programs utilized adjunctive psychiatric rehabilitation (27.5%), with no difference between drill and practice and drill plus strategy.

Trial Quality

Table 1 summarizes the trial methodology, year of publication, and breakdown of Clinical Trial Assessment Measure scores for all 40 studies examined. In this set of trials (N=39), the mean score was 57.4 (SD=12.3; range=35–87) out of a maximum of 100. Internal validity problems were sample size (60% were too small), lack of independent randomization (70%), lack of treatment fidelity assessment (80%), and group allocation masking (73%). The latter factor is known to inflate effect sizes.

Methodological quality improved over time, with some significant improvements, for example, for the method of group allocation ([N=39] Spearman's rho=0.40, p=0.01). However, some quality ratings diminished, for example, for analyses that included all participants randomly assigned ([N=40] Spearman's rho=–0.33, p=0.04). The dropout rate was higher than 15% in 12 studies, which is above the minimum level that would cause many statisticians to question the validity of the study findings.

There were no relationships between trial quality and type of remediation approach or whether or not the approach was computer assisted.

Meta-Analysis Results

Effects of cognitive remediation therapy on cognitive skills posttreatment

Two studies reported only functioning, and thus 38 studies provided measures to the global cognition effect size. Figure 1 illustrates a forest plot for global cognition, depicting an overall positive effect of cognitive rehabilitation of 0.45 (95% confidence interval [CI]=0.31–0.59). The statistical data for all cognitive and functioning outcomes at posttreatment as well as cognitive outcomes at follow-up assessment are presented in Table 2.

Eleven studies had pretreatment differences >0.2, with seven already showing better cognition in the allocated treatment group as well as a large global cognition effect size (0.7, 95% CI=0.1–1.2). However, even when these studies are excluded, there is still a significant effect size ([N=27] 0.4, 95% CI=0.28–0.54).

With only two exceptions, visual learning and memory and Continuous Performance Test ratings, all cognitive domains demonstrated significant effect sizes from 0.25 to 0.65. Domain effect sizes are affected by the set of studies that contributes to them, and thus no further inferences can be drawn about differences between them.

Individual test effects completely overlapped the composite effect sizes, suggesting that our conservative method of combining measures did not enhance their effects. Heterogeneity was found in global cognition as well as in speed of processing, verbal learning and memory, reasoning and problem solving, and Trail Making Test ratings. However, only 13 studies contributed to the Trail Making Test, and thus no further analyses were carried out on this measure.

A sensitivity analysis that excluded each study once made little difference to the global cognition effect size estimate (range=0.43–0.45). The same was found for an analysis that excluded three outliers (effect size=0.37 [SD>1.5], 95% CI=0.28–0.47).

Durability

The effects of treatment on global cognition were durable ([N=11] effect size=0.43, 95% CI=0.18–0.67).

Therapy effects on symptoms and functioning

There was a significant small-to-medium effect on functioning outcomes at both posttreatment and follow-up assessment. There was also a small, significant effect of cognitive remediation on symptoms at posttreatment, but it was no longer significant at follow-up assessment.

Publication Bias

Funnel plots (see Figure 1 in the data supplement) provided no evidence of publication bias producing a “file draw problem.” Statistical tests using Begg's method confirmed this for global cognition (z=0.99, p=0.32), symptoms (z=1.10, p=0.27), and functioning (z=0.26, p=0.80). The relationship between study vintage and global cognition was examined using metaregression, and no significant relationship was found.

Factors Affecting Treatment Benefit

Domains with significant heterogeneity and 20 contributing effect sizes are included in Table 3. Domains with fewer than 20 effect sizes (i.e., functioning) were explored using categorical analysis where possible.

Methodological quality

Overall trial quality (total Clinical Trial Assessment Measure score) had no effect on most cognitive outcomes. There was also no effect on functioning outcome (poor method [Clinical Trial Assessment Measure score <65]: effect size=0.44, 95% CI=0.19–0.69 versus good method [score ≥65]: effect size=0.39, 95% CI=0.02–0.75).

Masking group allocation had no effect on cognitive (Table 3) or functioning outcomes (unmasked: effect size=0.41, 95% CI=0.15–0.67 versus masked: effect size=0.43, 95% CI=0.08–0.79).

Three studies that failed to meet our stringent criteria for random allocation were excluded from the analysis, but the cognitive effect size was not affected ([N=35] effect size=0.47, 95% CI=0.32–0.62). There was no effect of independent randomization on cognition or functioning (nonindependent allocation: effect size=0.41, 95% CI=0.18–0.64 versus independent allocation: effect size=0.42, 95% CI=0.05–0.80). However, eight studies with participants who had better pretreatment cognition had larger effects, and of these only one used independent allocation of participants (14%), while the rate of independent allocation was 38% in the wider corpus.

There was a significant effect of cognitive rehabilitation on global cognitive outcome irrespective of the type of control, suggesting a specific effect of cognitive remediation on a comparable treatment. For functioning, trials that used an active comparison group produced a significant effect, which was not present in trials that applied a passive comparison group, although these were few in number (active comparison [N=14]: effect size=0.47, 95% CI=0.24–0.71 versus passive comparison [N=3]: effect size=0.16, 95% CI=–0.16 to 0.492) and the difference was not significant.

Participant characteristics

There was no indication of an effect of age. Lower baseline symptoms were related to larger global cognitive effects at posttreatment. However, the effect of treatment continued to be significant. Consequently, symptoms did not prevent improvements in cognition, but at high levels the effect size was modest.

Cognitive impairment as an entry criterion did not affect cognition outcomes. For functioning, studies using impairment as an inclusion criterion showed a larger effect size than those studies that did not (effect size=0.37, 95% CI=0.05–0.70 versus effect size=0.48, 95% CI=0.17–0.79, respectively), although with no significant statistical difference.

Clinical factors

Although 38 studies reported medication status, only 13 reported more detailed data regarding dosage. A median split on chlorpromazine equivalence into high (791 mg/day [N=6]) and low (375 mg/day [N=6]) doses demonstrated equivalent global cognition effect sizes in high (0.48) and low groups (0.55). There was no significant difference between these two groups of studies.

Larger effect sizes were found in studies that included only participants with schizophrenia or schizoaffective disorder (0.5) compared with those that also included participants with other diagnoses (0.2), and these groups tended to be statistically different (Q_b=2.7, p=0.1).

Therapy characteristics

There were no significant effects of remediation approach (drill and practice versus drill plus strategy), therapy duration, or computer use on cognitive outcomes.

Further exploratory analyses, as reviewed by McGurk et al. (9), on the effect of remediation programs on psychosocial functioning indicated significantly stronger effects (Q_b=5.51, p=0.02) in studies that provided adjunctive psychiatric rehabilitation to all patients (effect size=0.59, 95% CI=0.30–0.88) compared with studies that examined cognitive remediation alone (effect size=0.28, 95% CI=–0.02 to 0.58). In addition, there was a significant treatment effect for remediation only when a strategic approach was used (drill and practice: effect size=0.34, 95% CI=–0.11 to 0.78; drill plus strategy: effect size=0.47, 95% CI=0.22–0.73), although the difference was not significant. When only studies that provided additional psychiatric rehabilitation (N=8) were included, those using drill plus strategy programs (N=8) produced significantly larger effects ([N=4] effect size=0.8, 95% CI=0.4–1.2; Q_b=6.6, p=0.01) compared with those using drill and practice programs ([N=4] effect size=0.3, 95% CI=0.05–0.6).

None of the results changed when overall methodology was controlled in the metaregression analyses of moderators.

Discussion

The present meta-analysis on cognitive remediation therapy is the most comprehensive to date, containing 2,104 participants in 39 reports on 40 independent studies. In comparison, the most recent reviews conducted by the National Institute for Clinical Excellence (27) (17 studies, 1,084 participants), Patient Outcomes Research Team (26) (33 studies), and McGurk et al. (9) (26 studies, 1,151 participants) contained far fewer studies, treatments, and participants.

Is Cognitive Rehabilitation Effective?

This meta-analysis demonstrates a small to moderate effect of cognitive rehabilitation on cognitive outcomes at posttreatment and follow-up assessment in individuals with a diagnosis of schizophrenia. This result provides a fair representation of the published data, and there is little evidence of bias produced by trial methodology. There is also no evidence of publication bias, which was unlikely because current clinical equipoise about the usefulness of cognitive rehabilitation means that studies are publishable irrespective of whether they are successful. Older studies did not show the usual statistical trajectory toward higher effect sizes. In terms of external validity, most studies include relatively stable patients with a diagnosis of schizophrenia, both inpatients and outpatients. This is the identified group of patients considered to be in need of additional intensive treatment to improve recovery trajectory.

Our results replicate other reviews and meta-analyses (7–10, 12, 13) and further suggest that irrespective of therapy characteristics, cognitive rehabilitation can provide benefits to patients with cognitive difficulties. This is in contrast to two recent guidance reviews (26, 27), neither of which suggested that cognitive remediation should be adopted. The discrepancy with the United Kingdom National Institute for Clinical Excellence (27) is explained by their entry criteria, which excluded large-scale methodologically sound studies that included work rehabilitation within treatment as usual.

Is the Effect of Cognitive Remediation Therapy Dependent on Trial Methodology?

Bias in both the choice of populations and study method can affect the study results and may lead to the unsafe conclusion that a treatment is effective when it is not. Cognitive remediation trials were similar in rigor to studies of other psychological therapies (e.g., average Clinical Trial Assessment Measure scores for CBT for psychosis [29] and cognitive remediation). However, although trial methodology has serious consequences concerning CBT for psychosis trial estimates of effectiveness, particularly masked assessment, it had no effect on cognitive outcomes from cognitive remediation studies. This result should not be surprising, since the outcomes for CBT for psychosis are open to rater bias because they are based on patient interviews for symptom scores. However, cognitive test outcomes are specifically designed to exclude the effects of rater bias.

Clinical Implications

Do participants' characteristics make a difference? Although having more symptoms is associated with smaller effects, these participants still benefit from cognitive rehabilitation. Our medication data did not indicate that this was a bar to improvement through cognitive remediation, neither did the use of cognitive impairment as an inclusion criterion. We therefore conclude that it should be offered to all.

There was no evidence that age was a key moderator variable despite individual study evidence (19, 20). However, there was a narrow age range, with 25 studies (62.5%) reporting a mean age between 30 and 40 years. Our recommendations are therefore not as strong until other age groups have been tested.

Which cognitive remediation treatment works best? The present corpus of studies varies in terms of the way in which rehabilitation is provided. Most treatments were independently replicated, and many have multiple replications. We did not discover any variables to account for the heterogeneity in cognitive effects. Better differentiation can be made for treatment to improve functioning. The results of this meta-analysis suggest that functioning outcomes are best achieved by adding cognitive remediation to other rehabilitation programs, which boosts their effects. When other rehabilitation is included, strategic approaches rather than drill and practice achieve better outcomes, probably because they support transfer of training. These two factors may provide the most potent form of therapy for improving functional outcome.

Variations in treatment presentation have been commented on by several reviewers (26, 27). Our interpretation is that there are approximately 14 different treatments that have few salient differences; rather, they lie along dimensions of learning support as well as support for transfer to community functioning. All cognitive remediation therapies address a broad range of cognitive skills, even if they also include specific outcomes such as social cognition. This is shown in the cognitive outcomes tested within a study. What is more important than the surface characteristics (e.g., using a computer) is the technique of specific and explicit training of strategies and the use of various transfer techniques, as shown in the improved functioning outcomes for these approaches.

Research Implications

Given the evidence across a number of studies of the efficacy of cognitive rehabilitation, research must try to establish what variation in participants, as well as in treatments, will provide the most benefit for the most people. Those studies that do not produce significant effects have high information value because they allow us to consider variables that are important to both the personalization agenda and study design.

A number of these negative studies have appeared in the literature but have not always been well controlled or well explicated. This is not the case with a recent study conducted by Dickinson et al. (21), which has excellent methodology, with a total Clinical Trial Assessment Measure score of 82 out of 100, but produced a small nonsignificant effect size (0.06). This study has the second oldest group of participants (mean age=48 years [SD=7.65]) among the studies in our meta-analysis. Although age was not a predictor in our meta-analysis, there are data at a study level showing that older participants have poorer treatment response (e.g., 19, 20). Those investigators with study data from across the age span might carry out secondary analyses to identify whether age is a moderator. If age is found to be a moderator, then this would mean adaptations of treatment for older participants to increase effectiveness.

The Dickinson et al. study was designed to identify specific treatment effects and thus had an active comparison group that shared several elements in common with the experimental treatment group. This is an excellent way of identifying ingredients that are crucial to outcome, but this design does not allow us to detect whether the experimental therapy was beneficial. In the future, we recommend that cognitive rehabilitation studies include both active and passive comparison groups, since this allows for not only a decision on essential treatment elements but also an assessment of whether the specific therapy demonstrates a significant effect and is therefore worth the investment of scarce healthcare resources.

Even though we have not discovered the essential elements of successful treatment, researchers need to begin to identify key issues for therapy implementation, such as acceptability. One measure is the dropout rate from treatment. The Dickinson et al. study had a 2.5-times greater dropout rate in the comparison arm than the experimental therapy arm, which suggests that the therapy was at least engaging. One independent consumer-led acceptability study (43) showed that cognitive remediation therapy (44) was acceptable and valued. It also showed that if participants did not perceive improvement in therapy, then they “felt worse,” and this finding was demonstrated in secondary analyses. Individuals whose memory improved were more likely to improve their self-esteem, whereas those with no memory improvement experienced a decrement. These studies and detailed analyses allow an informed evaluation of therapy acceptability as well as of identifying side effects. Both these features would influence which therapy to adopt. In addition, since health services are unlikely to be able to pay patients to take part in therapy, those treatment programs that pay patients for attendance (31) would need to stop in order to allow a balanced view of engagement.

Conclusions

It is safe to conclude that there is a small to moderate durable effect of cognitive remediation on cognition and functioning that is not affected by study methodology. If the target is to improve functioning, then adjunctive therapy is essential, with the best effects being shown when a more strategic cognitive remediation approach is adopted.