Repeatable Battery for the Assessment of Neuropsychological Status as a Screening Test in Schizophrenia, II: Convergent/Discriminant Validity and Diagnostic Group Comparisons

The relation between neurocognitive performance and functional outcome suggests that cognitive assessment may be an important aspect of a full clinical diagnostic evaluation (1). Such an assessment may not need to be extensive to be clinically informative. In the companion article in this issue of the Journal(2), we presented data indicating that the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) (3), a 25-minute cognitive screening test, is sensitive to the types of impairments observed in schizophrenia, correlates highly with standard measures of intelligence and memory, and is related to functional outcome. Those results, obtained in a tertiary care research setting, suggest that the test may be a useful measure in everyday clinical practice.

The present study examined the validity of the RBANS in a very different study group, a diagnostically mixed group of 150 outpatients drawn from an inner city community mental health system. This patient group is predominately African American, generally poorly educated, economically disadvantaged, and has high rates of comorbid substance abuse or dependence. The present analyses were undertaken to determine if we could replicate evidence of the convergent validity of the RBANS with standard measures of memory and intelligence as well as to examine the relationship of the RBANS to a comprehensive neuropsychological battery. The goal of these analyses was to determine what the RBANS does or does not measure as an approach to assessing the convergent and discriminant validity of the test. Such analyses of the basic psychometric properties of the RBANS are necessary to establish its potential clinical utility as a general cognitive screening instrument.

In addition, we present data comparing the performance of 23 of the patients with schizophrenia on the neuropsychological battery and the RBANS with that of a demographically matched group of 23 of the patients with bipolar disorder. These data are presented for two reasons: 1) to examine the extent to which any differences between these groups documented on the larger battery are, or are not, reflected on the RBANS as another approach to assessing test validity, and 2) the comparison of these two diagnostic groups is of interest given the conflicting findings reported in the literature. Several studies (4–6) have reported clear differences in level of performance between these two diagnostic groups: patients with schizophrenia were found to be more impaired than patients with bipolar disorder across multiple measures of memory, attention, and problem solving. Other studies (7–9), however, have reported far more limited, if any, differences in overall performance levels of patients with schizophrenia and those with bipolar disorder.

The present study group included a large proportion of patients with substance abuse and dependence. Although this kind of comorbidity may complicate the interpretation of cognitive findings, investigation of the cognitive features of such patients may help inform clinical care because these patients are often seen in everyday clinical practice.

METHOD

Subjects

The study group consisted of 150 patients who were recruited from the Maryland site of the Employment Intervention Demonstration Project, an eight-site study funded by the Center for Mental Health Services of the Substance Abuse and Mental Health Services Administration. The Employment Intervention Demonstration Project is designed to examine the effectiveness of vocational rehabilitation in patients with psychiatric disability. Inclusion in the Employment Intervention Demonstration Project study required that participants be receiving disability-based income support or have a diagnosis of schizophrenia or schizoaffective disorder. Patients could also qualify for participation if they had required multiple psychiatric hospitalizations for any psychotic or mood disorder diagnosis over the previous 3 years or had been unable to work for more than 75% of the time during the past 12 months.

All patients received their primary psychiatric care from the Carter Center, which is part of the adult services program in the University of Maryland Department of Psychiatry in Baltimore. The Carter Center serves the most severely psychiatrically disabled people in the South/Southwestern regions of Baltimore City. This catchment area is predominately low income. The majority of the patients (N=102 [68%]) received assertive community or mobile treatment; the remaining 48 patients (32%) received more conventional outpatient clinic care.

DSM-IV diagnoses were made by research psychologists and psychiatrists from the Employment Intervention Demonstration Project study team, who used a best-estimate approach, combining information obtained from the Structured Clinical Interview for DSM-IV Axis I Disorders, Patient Edition (10), medical records, and clinical reports. The study group included three large diagnostic groups: 59 patients (39%) with schizophrenia, 42 (28%) with schizoaffective disorder, and 28 (18%) with bipolar disorder. The remaining patients received a diagnosis of another mood disorder (12 patients [8%]) or another psychotic disorder (nine patients [6%]).

Detailed substance use information was available for 139 (93%) of the subjects. The majority of these patients (N=90 [65%]) had a current co-occurring substance abuse or dependence disorder. Of the patients with a substance disorder, 66 (73%) met criteria for dependence and 24 (27%) met criteria for abuse. The majority of the patients meeting criteria for dependence used multiple substances (38 patients [58%]). Alcohol was the most frequently cited substance (48 patients [73%]), followed by cocaine (28 patients [42%]) and cannabis (18 patients [27%]). Of the patients meeting criteria for abuse, cocaine was the most frequently used substance (six patients [25%]), followed by cannabis (five patients [21%]), and alcohol (four patients [17%]).

The study group demonstrated significant functional disability. The vast majority of subjects (N=131 [87%]) were receiving either Supplemental Security Income or Social Security Disability Income. Although most patients (N=143 [95%]) had a lifetime work history, only 75 (50%) had held a job in the last 5 years and only one (0.7%) was marginally employed when entering the study.

All participation was voluntary, and patients were reimbursed. Written informed consent was obtained from all subjects after complete description of the study.

Procedure

All patients received Form A of the RBANS (3) (the companion article in this issue of the Journal[2] provides a full description of the test). The RBANS yields a total score as well as five index scores: immediate memory, visuospatial/constructional, language, attention, and delayed memory; each index score has a normal mean of 100 and standard deviation of 15 based on the performance of a standardization sample matched to the U.S. Census on sex, ethnicity, and level of education.

The patients also received a battery of standard neuropsychological measures. General intellectual function was assessed by using four subscales of the Wechsler Adult Intelligence Scale, 3rd ed. (WAIS-III) (vocabulary, block design, picture arrangement, and comprehension) that were prorated to estimate full-scale IQ scores (11). Auditory and visual memory were assessed by using the immediate and delayed recall subscales of the Wechsler Memory Scale, 3rd ed. (WMS-III) (logical memory stories and visual reproduction figures) (12). Attentional capabilities were assessed by using the Stroop Color and Word Test (13), Trail Making (14), and the Gordon Diagnostic System vigilance and distractibility versions of the Gordon Continuous Performance Test (15). Executive functioning was evaluated by using the Wisconsin Card Sorting Test (16) and letter-number sequencing from the WAIS-III (11). Language skills were measured by the reading subscale of the Wide-Range Achievement Test 3 (WRAT 3) (17) and the listening comprehension subscale of the Woodcock-Johnson Revised Tests of Cognitive Ability (18). Motor skills were measured by using the Purdue Pegboard (19) and the Halstead-Reitan Finger Tapping (14).

All tests were administered according to standardized testing procedures in a fixed order so that longer or more difficult tasks were separated by tasks that were shorter or less strenuous. The Wisconsin Card Sorting Test administration and scoring differed from the standard administration in two ways: 1) subjects received a total of only 64 cards (20) and 2) to enhance variance in the achieved measure of the categories, subjects were given partial credit (in tenths) if they finished the test with a series of correct responses (20). The testing was generally conducted in one 3-hour session, but testing was separated into two sessions when clinically indicated. Breaks were taken at scheduled intervals and as needed.

Data Analyses

Pearson correlations were performed between the RBANS total score and the WAIS-III and WMS-III subtests that were common to the patients reported on this article and the patients reported on in the companion article in this issue of the Journal(2). The magnitude of the correlation between the two groups of patients was compared by using the z test. We first performed Pearson correlations to examine the relationship between the RBANS and the broader neuropsychological battery. We then created a composite z score from the neuropsychological battery and correlated it with the RBANS to assess the relationship of the RBANS and global ability.

These univariate approaches were followed by a principal component analysis of the neuropsychological battery. The N for this analysis was 146 because data were missing data from four of the 150 patients. The RBANS total score was correlated with the six factor scores to determine which constructs the RBANS does or does not measure. Diagnostic group differences on the RBANS and the neuropsychological battery were investigated by comparing 23 demographically matched pairs of patients with schizophrenia or bipolar disorder with t tests and a descriptive examination of effect sizes.

RESULTS

Sociodemographic Characteristics

The mean age of the 150 subjects was 39.8 years (SD=8.2); their mean education level was 10.8 years (SD=2.4). Seventeen patients (11%) had completed grades 1–8, 76 (51%) had completed grades 9–11, 46 (31%) had finished high school, and 11 (7%) had 1–3 three years of college. Most of the patients were African American (N=117 [78%]); 30 (20%) were Caucasian, and three (2%) were Hispanic. There were more men (N=86 [57%]) than women (N=64 [43%]). Most of the patients were right-handed (N=138 [92%]).

The mean RBANS total and index scores for all 150 patients were as follows: total=66.02 (SD=14.04), immediate memory index=72.34 (SD=17.12), visuospatial/constructional index=68.35 (SD=12.96), language index=79.44 (SD=15.60), attention index=70.85 (SD=17.47), and delayed memory index=69.25 (SD=19.62). Cronbach’s alpha for the total and five index scores was 0.88, suggesting a high degree of internal consistency. The RBANS total score and the five index scores were correlated with patients’ age and education level. No significant correlations where observed between the RBANS scores and age (r values ranging from –0.002 to –0.09, df=148, all p values >0.1). Education was significantly correlated with the total score (r=0.38) and with four of the five index scores (r values ranging from 0.23 to 0.42, df=148, p values <0.005). The correlation with the visuospatial/constructional index did not reach significance (r=0.12, df=148, p>0.05). Thus, there is a modest but significant effect of education on test performance, consistent with data presented in the RBANS manual.

Replication of Convergent Validity

To determine if the RBANS measures abilities that are similar to those assessed by the WAIS-III and WMS-III (i.e., convergent validity in relationship to established measures), and if these correlations are stable across study groups, we performed correlations between the RBANS total score and the WAIS and WMS measures that were available for 38 of the patients reported in the companion article in this issue of the Journal(2) and in the current independent study group of 150. The magnitude of the correlations observed in the two groups of patients were compared by z test (21).

As seen in table 1, there is a remarkable degree of similarity in the magnitude of the correlations across study groups. None of the z tests approached the value of 1.96 necessary to demonstrate a significant difference in the value of the observed correlations. Thus, it appears that the RBANS total score demonstrates a reliable relationship with standard measures of IQ, working memory, and episodic memory across very different study groups.

The Relationship of the RBANS to Other Neuropsychological Measures

These results suggest that the RBANS total score may be a useful measure of general cognitive competence. To evaluate this in relation to the total neuropsychological battery, we created a composite z score to represent overall performance on the neuropsychological battery. This global score was the mean z score observed across 22 dependent measures, including scores on the four WMS-III scales and five WAIS-III scales; Stroop word, color, and color-word scales; Trail Making A and B; Halstead-Reitan Finger Tapping (average of dominant and nondominant hands); Purdue Pegboard (average of dominant, nondominant and tandem conditions); Gordon Continuous Performance Test vigilance and distractibility (total correct detections minus false alarms); Wisconsin Card Sorting Test (categories achieved and percent of perseverative errors); WRAT 3 reading; and Woodcock-Johnson listening comprehension. This global z score was highly correlated with the RBANS total score (r=0.79, df=148, p<0.001), suggesting that there is a high degree of overlap between what is assessed by the RBANS and the broader battery.

Although this composite score correlates highly with the RBANS total score, there was a wide range in the Pearson correlations between the RBANS total score and the individual measures from the neuropsychological battery. Correlations ranging from 0.60 to 0.75 were found on the Woodcock-Johnson listening comprehension, Stroop interference condition, and Trail Making B; from 0.40 to 0.59 on the WRAT 3, Wisconsin Card Sorting Test categories, Stroop color naming and word reading, Trail Making A, and the Purdue Pegboard; and from 0.20 to 0.39 on the Wisconsin Card Sorting Test percent of perseverative errors, Halstead-Reitan Finger Tapping, and the Gordon Continuous Performance Test vigilance and distractibility.

These univariate correlations between the RBANS and the neuropsychological battery do not directly address the question of how many different cognitive abilities were assessed by the larger neuropsychological battery. Therefore, we performed principal component analyses of the neuropsychological battery to determine how many discrete constructs were being assessed by the 22 scores. We first performed an exploratory analysis and found that six factors had eigenvalues greater than 0.70, the cutoff recommended by Jolliffe (22) for defining a factor. The data were then reanalyzed with a varimax rotation. These six factors accounted for 74% of the total variance in the battery. Tests that have factor loadings greater than 0.60 are shown in table 2, along with the amount of total variance explained by each factor.

The first factor appears to mark a general cognitive ability factor; it is composed of intelligence, language, and memory measures and accounts for the largest amount of variance. Although five of the six tests with loadings greater than 0.60 involve verbal functions, five other tests had factor loadings greater than 0.40, including the following mix of visual, verbal, attention, and working memory measures: WAIS-III block design and letter-number sequencing, Stroop color and word, WRAT 3 reading, and WMS-III visual reproduction 2.

Factors two through six appear to be more discrete: factor two is composed of attention measures that involve performance speed; factor three is a visual memory factor; factor four is a vigilance factor; factor five is an executive factor; and factor six is a motor factor.

In broad terms, the principal component results are consistent with how the tests are interpreted clinically with one possible exception. A clear memory factor, including both auditory and visual measures, did not emerge from the analysis. Instead, auditory memory loaded highly on the general ability factor, and visual reproduction defined a discrete factor.

We correlated these factor scores with the RBANS total score and found that the test covaried maximally with the general ability factor (r=0.57, df=144, p<0.0001), the attentional factor (r=0.44, df=144, p<0.0001), and the visual memory factor (r=0.36, df=144, p<0.0001). The RBANS correlations with the other three factors were relatively modest: motor (r=0.17, df=144, p<0.05), vigilance (r=0.12, df=144, p=0.15), and executive (r=0.12, df=144, p=0.16).

We also examined the correlations between the RBANS index scores and the six factors. All five index scores correlated significantly with the general ability factor (r values ranging from 0.23 [visuospatial/constructional index] to 0.52 [immediate memory index]), the attention factor (r values ranging from 0.31 [visuospatial/constructional index] to 0.41 [attention index]), and the visual memory factor (r values ranging from 0.22 [attention index] to 0.36 [language index]). No r values greater than 0.20 were observed between the five RBANS index scores and the vigilance, motor, or executive factors.

Diagnostic Group Comparisons

To control for the impact of demographic differences and substance abuse on cognitive performance, patients with schizophrenia and those with bipolar disorder were matched on the following characteristics: age plus or minus 3 years, sex, education, ethnicity, and presence or absence of substance abuse. From the original groups of 59 schizophrenia patients and 28 bipolar disorder patients, 23 matched pairs were obtained. The neuropsychological and RBANS results from the 23 matched patients along with t test results and effect sizes of the observed differences are shown in table 3. Effect sizes were calculated as the mean score of the bipolar disorder patients minus the mean score of the schizophrenia patients, divided by the mean standard deviation (23).

Both diagnostic groups demonstrated substantial cognitive limitations compared with normal population values. In part, these deficits may be explained by the limited educational background of the groups combined with the possible negative effects of substantial drug abuse. However, the schizophrenia group performed consistently worse than the bipolar group on the RBANS and other neuropsychological measures with the exception of the Halstead-Reitan Finger Tapping. Thirteen of the 29 comparisons yielded effect sizes greater than 0.50, Cohen’s criterion for a medium effect (23). An additional eight tests equaled or exceeded an effect size of 0.20, Cohen’s criterion for a small effect (23). The largest effect sizes were observed on measures of intellectual ability (full-scale IQ), verbal memory (vocabulary), and the categories achieved on the Wisconsin Card Sorting Test, consistent with the report of Goldberg et al. (6).

To compare the extent of intellectual decline of the two groups, a within-subject discrepancy score was calculated between the WRAT 3 reading scale (a premorbid estimate) (24) and the subject’s current estimated IQ. The groups differed significantly on the discrepancy score: for schizophrenia patients, mean=9.04 (SD=14.2); for bipolar disorder patients, mean=0.21 (SD=6.3) (t=2.86, df=44, p<0.01). This difference suggests that the schizophrenia group experienced more of an apparent decline in intellectual functioning than the bipolar disorder group. Similar results were observed between the WRAT 3 and the RBANS total score; again, schizophrenia patients demonstrated a greater discrepancy between estimated premorbid level and current level of functioning: for schizophrenia patients, mean=15.7 (SD=16.2); for bipolar disorder patients, mean=7.0 (SD=10.2) (t=2.18, df=44, p<0.05).

It appears unlikely that this pattern of findings can be explained by medication effects. Sixteen (70%) of the 23 schizophrenia patients were taking typical antipsychotics, five (22%) were taking atypical antipsychotics, and two (9%) were taking a combination of both. Thirteen (57%) of the 23 bipolar disorder patients were taking typical antipsychotics, three (13%) were taking atypical antipsychotics, and 17 (74%) were taking either an antidepressant or a mood stabilizer. Further, the fact that the schizophrenia group did not perform worse than the bipolar disorder group on measures of motor performance, tasks that are potentially most sensitive to the negative effects of typical antipsychotics (25), would also argue against the role of medication effects in explaining the overall group differences (25).

As seen in table 3, the effect size of 0.55 for the RBANS total score meets the criterion for a medium effect, and the RBANS immediate memory scale effect size is highly similar to that observed on the logical memory and visual reproduction subtests of the WMS-III. However, the language scale effect size of 0.17 is clearly lower than that observed on several of the other language measures in the larger battery. The RBANS attention scale also yields somewhat smaller effect sizes than several of the attention measures that require performance speed. Thus, the RBANS is somewhat less sensitive than several of the neuropsychological measures to the diagnostic effects present in this study group.

DISCUSSION

These data strongly support the validity of the RBANS as a screening test for general level of cognitive impairment in psychiatric populations. The test correlates highly with measures of IQ, memory, and general performance level across a broad neuropsychological battery. These results are highly consistent with those reported in our companion article (2), in a very different patient study group, suggesting that the observed relationships are quite robust.

The current data extend beyond those presented in our companion article in demonstrating the cognitive domains that are not measured by the RBANS. As seen in the principal component analyses, the RBANS loads primarily on general ability, attention, and visual memory factors and does not correlate substantially with the vigilance, executive, and motor factors identified in the current study group. Therefore, the RBANS needs to be supplemented by additional measures of these constructs if these abilities are of specific interest. This may be a particular concern in the design of test batteries to assess the impact of different pharmacological treatments that may affect these cognitive functions. Thus, the RBANS clearly is not a replacement for a comprehensive neuropsychological battery when detailed assessment of multiple cognitive functions is required.

Given the time limitations in many clinical settings, however, the typical choice is not between the administration of a comprehensive battery and a screening instrument. Instead, the choice is between the administration of a screening instrument or not obtaining a formal cognitive assessment. In our view, the data from our two studies strongly suggest that the RBANS total score is a highly useful single number that captures general performance level. The test is quickly administered and is well tolerated by patients, including severely ill patients who would not be able to cooperate for the 2 to 4 hours often required to do a more comprehensive neuropsychological evaluation. As discussed in our companion article (2), the total score may be useful in both clinical and research settings to enhance communication and description of general level of cognitive function in individual cases or study populations.

The cognitive differences between patients with schizophrenia or bipolar disorder are largely consistent with those reported by Goldberg et al. (6) and others (4, 5). The fact that such differences were observed in such a low-functioning group with substantial substance abuse comorbidity might be seen as evidence that the cognitive differences between groups are quite reliable. However, there are multiple studies in the literature that have not found diagnostic effects (7–9), and there does not appear to be a clear explanation for these conflicting results. As a way to interpret the literature, we suggest that, although patients with schizophrenia reliably demonstrate evidence of cognitive impairment, the inconsistent findings across studies of patients with bipolar disorder suggest that cognitive impairment may not be a primary feature of the disorder but, rather, a characteristic of a subgroup of patients. Impairment in this subgroup varies in severity, at times closely resembling that seen among patients with schizophrenia. The clinical features of this subgroup cannot be determined from the available literature because many reports have combined patients with different types of mood disorder. Additional research will be needed to determine the clinical and biological correlates of cognitive impairment among patients with bipolar disorder. The fact that cognitive impairment does appear to be present in some patients with bipolar disorder suggests that a screening cognitive examination such as the RBANS may be a useful addition to standard clinical evaluations.

CONCLUSIONS

The data from our two studies suggest that the RBANS yields clinically meaningful information. It correlates highly with general level of cognitive competence, is related to employment outcome, and has some sensitivity to diagnostic group differences. In the light of the relatively poor predictive power of most symptomatic variables for functional outcome in patients with schizophrenia, the addition of this type of cognitive data could enhance everyday clinical practice at minimal cost. Such information should be informative for treatment planning, and the alternative form reliability of the test suggests that it may be useful for monitoring clinical status.

Received Dec. 28, 1998; revision received May 11, 1999; accepted May 13, 1999. From the Maryland Psychiatric Research Center, Department of Psychiatry, University of Maryland at Baltimore. Address reprint requests to Dr. Gold, Maryland Psychiatric Research Center, Department of Psychiatry, University of Maryland at Baltimore, P.O. Box 21247, Baltimore, MD 21228; [email protected] (e-mail). Supported by grants from the Stanley Foundation and NIMH grant MH-40279.

TABLE 1

Enlarge table

TABLE 2

Enlarge table

TABLE 3

Enlarge table