Repeatable Battery for the Assessment of Neuropsychological Status as a Screening Test in Schizophrenia, I: Sensitivity, Reliability, and Validity

Neuropsychological studies of patients with schizophrenia have demonstrated that cognitive impairment is a prominent feature of the illness, that it is present at illness onset, and that it is generally stable over time (1, 2). Cognitive impairments appear to be largely unresponsive to treatment with conventional antipsychotic medications (3). There is a growing literature documenting possible differential cognitive effects of the new generation of medications (4–7). The documented benefits have generally been modest, however; patients continue to demonstrate marked deficits compared with the performance of normal subjects. Such data suggest that the symptomatic and cognitive manifestations of schizophrenia may represent relatively independent dimensions of the illness with different underlying pathophysiology, course, and treatment responsivity. There is evidence (8) suggesting that cognitive measures may be related to functional outcome more reliably than positive symptomatic variables. Thus, cognitive impairment may be prognostically important, and the enhancement of cognitive functioning has become an important target for both psychosocial and pharmacological interventions.

One factor that may have delayed appreciation of the importance of cognitive impairment in schizophrenia is the fact that neuropsychological assessment batteries are quite extensive. Such testing tends to be expensive and time-demanding and is frequently unavailable in clinical settings. In populations of elderly patients with schizophrenia, very brief tools like the Mini-Mental State (9) are sufficient to document the severity of cognitive impairment (10). In young-to-middle-aged patients with schizophrenia, the Mini-Mental State may not be adequately sensitive given its limited assessment of memory and attention functions. However, the Mini-Mental State has proven to be of great value in studies of dementia, where it is a commonly used “standard” that enhances communication between clinicians and researchers alike. In many studies of Alzheimer’s disease, the Mini-Mental State is supplemented by specific measures of cognitive constructs of interest and provides an interpretive context for experimental findings.

There is no similar cognitive screening instrument in wide use in the schizophrenia literature, making it difficult to compare results across studies. Cognitive assessment has not become central to everyday clinical practice, partly because of the lack of a brief, easily administered, standard cognitive instrument that provides clinically meaningful information. To fulfill this clinical and research function, we propose that an instrument would need to have the following characteristics: 1) availability of normative data to interpret performance, 2) demonstrated sensitivity to the level and pattern of impairment typically seen in schizophrenia, 3) simple and brief administration to facilitate assessment of severely impaired patients, 4) high correlation with more extensive assessments, 5) demonstrated test-retest reliability, 6) relation to important aspects of functional outcome, and 7) relative independence of the positive symptoms of the illness, as is typically the case with more extensive assessments (3).

In this report, we present data suggesting that the recently published Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) (11) appears to meet these seven criteria in a large group of patients with schizophrenia drawn from a tertiary care research setting. In the companion article in this issue of the Journal(12), we present data from a large diagnostically heterogeneous group drawn from an inner city public mental health system to examine the generalizability of our results.

METHOD

Subjects

One hundred twenty-nine patients with a DSM-III-R diagnosis of schizophrenia from the Maryland Psychiatric Research Center were assessed with the RBANS. All subjects were diagnosed by using a best-estimate approach, combining structured interviews with all available medical records and information from collateral informants. The outpatient study group was drawn from two clinics—the Outpatient Research Program and the Motor Disorder Clinic. The majority of patients (N=78 [61% of the total study group]) received their care at the Outpatient Research Program. This program cares for a diverse patient population ranging from competitively employed patients with few residual symptoms to continuously psychotic patients who are able to maintain community residence. Twenty-five patients were drawn from the Motor Disorder Clinic, which provides consultative services focused on tardive dyskinesia to patients from other Maryland public mental health facilities. The 26 inpatients were all voluntary research subjects at the Maryland Psychiatric Research Center.

The patients had a mean age of 39.4 years (SD=7.8). The majority were men (N=92 [71%]). Patients had a mean education of 12.1 years (SD=2.5); 32 (25%) had less than 12 years, 52 (40%) finished high school, 32 (25%) had 1–3 years of college, and 13 (10%) completed 4 or more years of college. The patients’ mean age at illness onset was 21.2 years (SD=6.6). The majority (N=88 [68%]) were Caucasian, and all were native English speakers.

Patients received the following schizophrenia subtype diagnoses: 69 (53%) undifferentiated, 37 (29%) paranoid, eight (6%) disorganized, one (1%) catatonic, three (2%) residual, and 12 (9%) schizoaffective. All patients were tested when judged to be in a period of clinical stability by their treating clinicians. The majority of patients (N=83 [64%]) were receiving one of the new generation of antipsychotics, most frequently clozapine (N=56 [67% of those receiving new-generation medications]). In addition, 34 (41%) of the patients receiving new-generation antipsychotics were also taking adjunctive antidepressants, mood stabilizers, or antianxiety agents, and four (5%) were receiving a combination of new and conventional antipsychotics. In the 41 patients who were receiving conventional antipsychotics (32% of the total study group), haloperidol and fluphenazine were the most commonly used agents, accompanied by anticholinergic agents in 25 (61%) of the cases. Other agents, including antidepressants, mood stabilizers and antianxiety agents were used by 15 (37%) of the patients taking typical agents. Five patients were not receiving any antipsychotic medications.

Written informed consent was obtained from all patients after they had been given a complete description of the study.

Procedure

All 129 patients were given the RBANS; 101 also received the Wide-Range Achievement Test 3 (WRAT 3) reading subtest (13); 38 patients from the Outpatient Research Program received the Wechsler Adult Intelligence Scale, 3rd ed. (WAIS-III) (14), and 37 of the 38 patients also received the Wechsler Memory Scale, 3rd ed. (WMS-III) (14). Forty-eight patients were rated on the Brief Psychiatric Rating Scale (BPRS) (15) within 14 days of being assessed on the RBANS by raters trained at the Clinical Research Center. Fifty-three patients received alternate forms of the RBANS.

The RBANS was designed for several purposes, including use as a screening test for dementia in the elderly, as a screening battery in adults when lengthier assessments are not appropriate, and for repeat evaluations to assess disease progression or outcome of therapeutic trials. The test is composed of 10 subtests that are combined to form five index scores and a total scale score. Each score is expressed as a standard score with a normal mean of 100 and a standard deviation of 15 based on a normative study group of 540 healthy subjects, ranging in age from 20 to 89 years, matched to the U.S. Census on sex, ethnicity, and level of education.

The immediate memory index is composed of a 10-word list of learning tasks presented for four trials and a brief story that is presented for two recall trials. The language index is composed of a 10-item confrontation naming test and one 60-second category fluency trial. The visuospatial/constructional index is composed of a complex figure copy and judgment-of-line orientation task. The attention index is composed of a digit span test and a coding task. The delayed memory index is composed of delayed recall of the story, complex figure, and word list as well as a 20-item recognition test for word list items.

The RBANS has two alternate forms that are psychometrically equivalent. These alternate forms contain different test items.

Data Analysis

The overall analyses of test sensitivity used the total study group of 129 patients. Test-retest reliability data were available for 53 patients who were tested, on average, 12 weeks apart (range=2–50, SD=11.5). Reliability was assessed by using intraclass correlation coefficients and Pearson correlation coefficients. Convergent validity with the WAIS-III (N=38) and WMS-III (N=37) was assessed in the outpatients by using correlations and multiple regressions. The employment outcome analyses were performed with 77 of the patients, all of whom were drawn from the Outpatient Research Program; these analyses used both t tests and linear discriminant functions.

RESULTS

Impact of Demographic Factors

To examine the impact of age and education, Pearson correlations were calculated with the RBANS total and index scores. Only one significant correlation was observed with age, involving the language index (r=0.22, df=127, p<0.05): relatively older patients had better language performance. Education significantly correlated with all five index scores (r values ranged from 0.22 to 0.28, all df=127, all p values <0.05) and with the total score (r=0.33, df=127, p<0.001), suggesting a modest, expectable association of cognitive performance and educational achievement.

Male and female patients differed only on the visuospatial/constructional index, where men performed significantly better (mean=82, SD=17.7, versus mean=71, SD=19.2) (t=–3.13, df=127, p<0.05). None of the other index comparisons or total score differences approached significance (all p values >0.25).

The RBANS total score of the Caucasian patients (mean=74.9, SD=16.1) was higher than that of the African American patients (mean=63.5, SD=11.8) (t=–3.9, df=124, p<0.001). This difference appeared to be explained by differences in educational achievement: the mean WRAT 3 reading score of the Caucasian patients was 92.5 (SD=11.3), compared with 78.5 (SD=14.4) for the African American patients (t=–5.1, df=97, p<0.001). When the WRAT 3 was covaried, ethnicity differences were not found on the total score (F=0.18, df=1, 96, p=0.67).

Test Sensitivity

The patients’ mean RBANS total score was 71.4 (SD=15.7). Their mean index scores were as follows: immediate memory, 72.4 (SD=18.9); visuospatial/constructional, 79.1 (SD=18.8); language, 84.7 (SD=14.3); attention, 74.7 (SD=18.3); and delayed memory, 74.9 (SD=18.8). Cronbach’s alpha, calculated by using the total score and the five index scores, was 0.88, suggesting a high degree of internal consistency. Patients demonstrated clear impairment relative to normal standardization scores across all the RBANS index scores.

To test for differences among the five index scores, we performed an analysis of variance (ANOVA) followed by post hoc comparisons with Bonferroni t tests. The ANOVA was significant (F=17.5, df=4, 644, p<0.0001), evidence that performance differed as a function of index. Post hoc comparisons demonstrated that the language index was significantly higher than the other four indexes and that the visuospatial/constructional index was significantly higher than the immediate memory index. This profile of general impairment of approximately two standard deviations below the normal mean with maximal impairment in memory function and less severe compromise of language and visual perceptual skills is consistent with the findings of studies using comprehensive neuropsychological batteries (1, 16, 17).

Standard scores provide an estimate of how impaired patients are compared with healthy subjects. Of equal clinical interest is how impaired a patient is relative to others with the same illness. The percentile distribution of patients’ performance is presented in table 1, along with the normal standardization distribution. Almost half of the patients had a total scale score less than 70, a score observed in less than 2% of healthy subjects. Only 4% of patients obtained a total scale score above 100, the 50th percentile of the normal distribution. Thus, the patient score distribution is dramatically shifted downward.

Further evidence that the RBANS total score is sensitive to the type of impairment observed in schizophrenia can be seen in the comparison of the WRAT 3 and RBANS total scores. The WRAT 3 is a measure of single word reading and is often considered an indicator of premorbid competence (18). In 101 patients, the mean WRAT 3 score was 88.9 (SD=13.5), whereas their RBANS total score mean was 71.8 (SD=16.1) (t=–12.7, p<0.00001). WRAT 3 scores were higher than RBANS scores in 88 (87%) patients, and discrepancies of 18 or more points favoring the WRAT 3 occurred in over 50% (N=51) of the patients.

Convergent Validity

To determine if the RBANS assesses cognitive abilities similar to those assessed by standard tests of intellectual and memory function, we examined the relationship between the RBANS and the WAIS-III and WMS-III. The correlations between the RBANS total and index scores with WAIS-III and WMS-III index scores are shown in table 2. These correlations demonstrate that the RBANS total score is highly correlated with the major summary WAIS-III and WMS-III index scores. The correlational data provide clear evidence that the RBANS immediate memory, delayed memory, and attention index scores are assessing similar constructs as those assessed at greater length on the WAIS-III and WMS-III. The correlations of the language and visuospatial/constructional indexes are less clear. Although the language index score correlated more strongly with the verbal comprehension than perceptual organization index scores, the index demonstrates similar magnitude of relationship with memory and processing speed. Similarly, the visuospatial/constructional index score correlates at similar levels with the verbal comprehension, perceptual organization, and working memory index scores. Thus, for these two indexes, there is evidence for shared variance with the expected factor scores, but this shared variance does not appear to be highly selective.

Given the high correlation of the RBANS total score with many of the WAIS-III and WMS-III index scores, we performed a stepwise regression to specify which index scores contributed the most unique variance with the RBANS total score. A two-variable solution emerged (F=45.26, df=2, 34, p<0.0001): the WAIS-III working memory index entered the equation first (t=5.35, p<0.0001) with an R² of 0.66; the WMS-III auditory delay recognition index entered second (t=2.99, p<0.01), increasing the R² to 0.73. No other variables entered. The apparent specificity of this result should be considered in the light of the RBANS-WAIS-III/WMS-III correlation matrix and the high intercorrelation of many of the WAIS-III and WMS-III scores.

Comparisons of the absolute level of scores on the RBANS and similar constructs assessed by WAIS-III and WMS-III are shown in table 3. In the assessment of attention functions, both WAIS-III indexes are compared with the RBANS attention index. The RBANS scores were systematically lower than the WAIS-III and WMS-III index scores with the exception of delayed memory measures. Only three of 38 patients had RBANS total scores greater than their full-scale IQ score, and 23 (61%) patients had IQ scores 10 or more points higher than the RBANS. Thus, with the exception of the delayed memory measures, the RBANS yields consistently lower scores than observed on the Wechsler scales, a difference in performance level that is important for interpretation of test findings.

Test-Retest Reliability

Fifty-three patients received different forms of the RBANS on two occasions with a mean test-retest interval of 12 weeks. All but five patients were tested with Form A followed by Form B. Scores from the two test occasions are shown in table 4, along with the intraclass correlations and Pearson correlations. The intraclass correlation is the preferred measure of whether subjects scored at similar levels across test occasions, but Pearson correlations are indicative of whether subjects retained their relative rank. The fact that the values were so similar suggests a high degree of actual score and relative position stability. The total score and the attention index demonstrated excellent reliability and are the primary scores to consider in assessing change over time. The other four index scores had lower reliability values. There was no indication of a practice effect; in fact, Form B scores tended to be slightly lower than Form A scores. The Form B delayed memory index score was significantly lower than Form A (t=2.5, df=52, p<0.05, paired t test), the only such difference observed. It is unclear if this represents a chance finding or a genuine difference in the difficulty levels across the two forms of this index. A similar 5-point difference favoring the Form A delayed memory score relative to Form B was reported in the RBANS test manual in a study group of normal subjects tested twice in counterbalanced fashion (11).

Relationship to Symptoms

To examine the relationship between the RBANS and symptoms, we examined correlations between cognitive performance and BPRS ratings. Forty-eight of the patients who were tested on the RBANS also received BPRS ratings within 14 days, during a period of clinical stability. There were no significant relationships between BPRS total score and the RBANS total or index scores (r values ranged from –0.15 to 0.13). None of the correlations between the BPRS conceptual disorganization and withdrawal-apathy (a proxy for negative symptoms) factors were significant (conceptual disorganization r values ranged from –0.26 to 0.06; withdrawal r values ranged from –0.14 to 0.24). When only patients who received the BPRS and RBANS on the same day (N=17) were examined, a similar lack of significant correlations was observed. With a larger number of subjects, some of these modest relationships might achieve statistical significance. However, it does not appear that symptom severity accounts for the extent of impairment observed on the RBANS.

Relationship to Functional Outcome

To assess the relationship between RBANS performance and functional outcome, we focused on competitive employment because this outcome dimension is easily rated. We limited these analyses to 77 of the 78 patients from the Outpatient Research Program, a homogeneous group in terms of treatment services. Employment status was reported by the patient’s primary therapist. “Employed” was defined as competitive employment of 20 hours or more a week, a rating achieved by 17 of 77 patients. All patients not meeting this criterion were considered unemployed. The resulting two groups were similar in age (employed mean=38.2 years, SD=6.0, versus unemployed mean=40.2, SD=7.7) but differed significantly in number of years of education completed (employed mean=14.4, SD=2.3, versus unemployed mean=11.8, SD=2.2) (t=4.1, df=75, p<0.001).

Mean RBANS scores of the employed and unemployed groups are shown in table 5, along with t test results and percent correct classification results based on discriminant function analyses using each variable alone. Employed patients performed significantly better than unemployed patients on four of the five index scores as well as on the total score. The 95% confidence intervals for the means on the total, delayed memory, attention, and immediate memory index scores did not overlap between groups. The discriminant function results suggest that it is possible to classify employment status correctly 64% of the time or better using four of the five index scores or the total score.

To determine if the between-group cognitive differences might be accounted for by differences in education level, we performed analyses of covariance, covarying years of education. Group differences remained significant on the immediate memory (F=14.89, df=1, 74, p<0.001), delayed memory (F=7.06, df=1, 74, p<0.01), and attention (F=4.04, df=1, 74, p<0.05) indexes and on the total score (F=8.74, df=1, 74, p<0.01). Thus, RBANS performance is related to employment status after education differences are controlled.

DISCUSSION

These data strongly support use of the RBANS as a screening measure for cognitive impairment in patients with schizophrenia. The extent and pattern of impairment noted on the RBANS are broadly consistent with the results of more comprehensive neuropsychological batteries (1, 16, 17), and the correlational data suggest that the test is assessing similar constructs assessed in more detail by the WAIS-III and WMS-III. Further, test performance is relatively independent of symptomatic state but is related to employment status. The total score demonstrates impressive alternate form reliability. Thus, the test provides reliable, clinically useful data in 25 minutes of examiner and patient time. These findings suggest that the RBANS may be a useful tool for the assessment of clinical status.

It should be made emphatically clear that the RBANS does not provide a comprehensive neuropsychological evaluation; rather, it provides a screening assessment of several important cognitive functions. Such a screening instrument is appropriate to address certain clinical and/or research questions but does not take the place of a comprehensive examination. For example, if it is important to derive a profile of a patient’s cognitive strengths and weaknesses, the limited assessment offered by the RBANS index scores is often likely to be inadequate. However, if the goal of the assessment is determine whether an individual patient, or group of patients, demonstrates the type of impairment frequently seen in schizophrenia, the RBANS appears to be quite useful, and the percentile data presented in table 1 should facilitate a comparison of how a patient or group performs relative to this clinical “reference” study group.

Neuropsychological research in schizophrenia has been primarily motivated by the desire to localize areas of maximal impairment in cognitive-anatomical terms. Although controversy remains on the existence of differential deficits, it is clear from multiple studies that the cognitive impairment in schizophrenia is broad, affecting most cognitive functions (1, 2, 16, 17). Therefore, a brief measure that provides for an assessment of this general level of impairment may be valuable in facilitating the use of formal cognitive assessment in patients with schizophrenia as part of a standard clinical evaluation. In research settings, the RBANS may be a useful tool to document the general level of functioning in a study group, similar to the role that the Mini-Mental State has played in the literature on Alzheimer’s disease.

It is important to realize that despite the fact that the RBANS is standardized like an IQ test with a mean of 100 (SD=15), the total score is often extremely low, and low scores are not an indication of mental retardation. The six patients who had RBANS total scores less than 50 had a mean WRAT 3 reading standard score of 83.2, suggesting that these patients had low average levels of premorbid academic skills. Examination of the test suggests that the total score is heavily weighted by measures of attention and memory, critical areas of impairment in schizophrenia. Further, variance in the language index is largely driven by verbal fluency performance, another area of impairment in schizophrenia. Thus, the RBANS is weighted toward the assessment of areas of marked deficit in schizophrenia, with fewer items assessing areas of relative strength such as in preserved verbal knowledge. Thus, the total score is often much lower than might be expected based on WRAT 3 or WAIS-III scores.

Green (8) reviewed evidence that verbal memory may be an important predictor of community functioning. Our data are consistent with his review: employed patients performed significantly better than unemployed patients on the immediate memory index, and this score was the most discriminating of employment status. However, the relationship between memory and outcome appears to be only mildly specific: employed patients outperformed unemployed patients on each of the index scores, and only slightly lower rates of correct classification were observed when either the attention or language index was used.

In the companion article in this issue of the Journal(12), we present evidence that the data presented here concerning test sensitivity and convergent validity with other cognitive measures appear to be reliable. Additional data are needed to determine the relationship of RBANS performance and aspects of functional outcome to evaluate the utility of the measure as either a predictor or primary outcome measure. It is important to note the limitations of the test-retest reliability data. The present data analyses confounded test order and test form: 48 of 53 patients received Form A first. Thus, it is possible that the impressive reliability observed on the total score may be partially confounded. To address this issue, we are currently collecting a study group that is being tested first on Form B, followed by Form A. The reliability data are limited to two test administrations. Additional data are necessary before it is possible to recommend the test in situations where multiple assessment occasions are required.

CONCLUSIONS

The RBANS appears to be a highly useful screening measure of cognitive functioning in patients with schizophrenia. The test is sensitive to the level and pattern of impairment typically observed in schizophrenia, is highly correlated with standard measures of intelligence and memory, and is largely independent of symptom severity. The test appears to have good alternate form reliability and is related to employment status.

Received Dec. 28, 1998; revision received May 1, 1999; accepted May 13, 1999. From the Maryland Psychiatric Research Center, Department of Psychiatry, University of Maryland at Baltimore. Address reprint requests to Dr. Gold, Maryland Psychiatric Research Center, P.O. Box 21247, Baltimore, MD 21228; [email protected] (e-mail). Supported by NIMH grants MH-57749 and MH-40279.

TABLE 1

Enlarge table

TABLE 2

Enlarge table

TABLE 3

Enlarge table

TABLE 4

Enlarge table

TABLE 5

Enlarge table