Performance-Based Measures of Everyday Function in Mild Cognitive Impairment

Mild cognitive impairment is a less-than-benign diagnosis because it is associated with an elevated risk of incident Alzheimer's disease and more rapid cognitive decline (1, 2). The rate of conversion from mild cognitive impairment to Alzheimer's disease may be 10%–12% per year (3). Neuropathologically, mild cognitive impairment (in many but not all cases) appears to be a transitional state of evolving Alzheimer's disease (4, 5). Recent in vivo quantified imaging using the amyloid binding ligand carbon-11-PIB has suggested that the amyloid burden in mild cognitive impairment is intermediate between healthy comparison subjects and patients with Alzheimer's disease (6, 7). By recommended diagnostic criteria, individuals with mild cognitive impairment have an impairment of 1.5 standard deviations in episodic memory but essentially preserved everyday function (8, 9).

Several lines of evidence suggest that the diagnostic criterion relating to function may be problematic. First, neuropsychiatric conditions with associated cognitive impairments are also reliably associated with sequelae in everyday function. These include such disorders as Alzheimer's disease itself, various amnestic syndromes, traumatic brain injury, focal epilepsy, stroke, multiple sclerosis, and HIV infection (10–16). Second, if the cognitive impairment and underlying neuropathology of amnestic mild cognitive impairment and Alzheimer's disease are on a continuum, it would be unlikely that functional impairment could be dichotomized as preserved (in mild cognitive impairment) or impaired (in Alzheimer's disease). Thus, when pathology and cognitive impairment are dimensional, they are also likely to exact a graduated cost on function in mild cognitive impairment.

Another factor that might artifactually maintain views of intact or preserved function in mild cognitive impairment is related to test measurement issues. If more sensitive measures of everyday function were used, they might indeed be found to be compromised in populations with mild cognitive impairment. There has been little empirical study of the exact nature of activities of daily living or instrumental activities of daily living in individuals with mild cognitive impairment (see the Discussion section for more on previous work), and several investigators have suggested that new instruments capable of objectively measuring functional impairments and their decline in patients with mild cognitive impairment would advance the field (17–20). Basic activities such as bathing, toileting, dressing, and eating remain preserved in the face of mild cognitive deterioration, but more complex abilities, such as managing schedules, managing finances, planning trips, shopping, and using public transportation, which usually involve interactions with technology, use of higher-level cognitive operations, or knowledge of cultural expectations (i.e., instrumental activities of daily living), are sometimes thought to be impaired, albeit subtly (21). Because instrumental activities of daily living are more complex and demanding and rely less on routines and habits, they may be more sensitive to the early effects of cognitive deterioration. Another measurement issue that can affect test sensitivity is the use of informant-based measures, which can suffer from the inherent problems associated with halo effects, memory lapses, or lacunae in knowledge, generally resulting in an overestimation of the patient's abilities (22). Thus, one way to increase sensitivity in these functional measures might be to use performance-based measures. In these functional outcome assessments, individuals are tested and objectively scored on their ability to perform real-world task analogues.

In this study, we carefully assayed functional disability using a performance-based measure of everyday function that included many ecologically relevant items in healthy comparison subjects, patients with mild cognitive impairment, and patients with Alzheimer's disease and then assessed its relationship to performances on a variety of cognitive tests, as well as its psychometric properties. We were also able to compare it to a widely used informant-based measure of function.

Method

Staging Instruments

The Mini-Mental State Examination (MMSE) (23) was used for screening of cognitive level. The Clinical Dementia Rating scale (24) was used to assess dementia; this instrument consists of items relating to memory, orientation, problem solving, personal care, function at home and in hobbies, and function in community affairs. Domain scores were integrated in an algorithm in order to yield the global value.

Participants

All participants were between the ages of 55 and 85 years. There were no restrictions based on gender or ethnicity. Patients were recruited because they had subjective complaints. All potential participants underwent examination by a neurologist or a geriatric psychiatrist to determine study eligibility.

Patients with evidence of clinically significant and active pulmonary, gastrointestinal, renal, hepatic, endocrine, or cardiovascular system disease were excluded, as were individuals with clinically significant folate or vitamin B₁₂ deficiency and those undergoing cancer treatment. Individuals with Hachinski Ischemic Scores >4 were excluded (25). Patients with evidence of other neurological disorders, including but not limited to stroke, Parkinson's disease, seizure disorder, hydrocephalus, and head injury with loss of consciousness greater than 30 minutes within the past 5 years, were excluded. Patients with a current DSM-IV axis I disorder other than Alzheimer's disease (including substance use diagnoses within the previous year) were excluded. Controlled diabetes, hypertension, and hypothyroidism were not among the exclusion criteria.

Alzheimer's disease. Twenty-two individuals met National Institute of Neurological and Communicative Disorders and Stroke–Alzheimer's Disease and Related Disorders Association criteria for probable Alzheimer's disease. Diagnostic criteria include memory impairment (defined below for mild cognitive impairment) and at least one other area of impaired cognition, including speed of processing, executive ability, and/or semantic processing/language; report of decline in memory and other areas of cognition; and impairments in activities of daily living. Patients in the Alzheimer's disease group had MMSE scores <24 and >12 (i.e., in the mild to moderate range) and Clinical Dementia Ratings of 1.0 or greater on the global scale.

Mild cognitive impairment

The diagnosis of mild cognitive impairment was made according to the criteria of Petersen et al. (8) for "amnestic" mild cognitive impairment in 26 individuals. These individuals had memory impairment of greater than 1.5 standard deviations on either selective reminding or logical memory and had ostensibly preserved activities of daily living (i.e., were "functioning well"). Individuals who had additional impairments in other nonmnemonic domains of cognition were also included, as long as activities of daily living were ostensibly preserved (i.e., "amnestic plus"). This approach is similar to that used in the study of Tabert et al. (26). All participants with mild cognitive impairment had MMSE scores ≥24 (i.e., they did not have dementia) and Clinical Dementia Ratings of 0.5 (as specified by the National Institute of Aging Working Group on Study Design) (27).

Cognitively healthy elders

Fifty elders had MMSE scores ≥24 and did not meet psychometric criteria for mild cognitive impairment or Alzheimer's disease. All formal neurocognitive test scores for these participants were within 1.5 standard deviations of normative data in published studies or manuals. Cognitively healthy elders were usually the spouses of study participants.

Medications

Of the cognitively healthy participants, 34 were not receiving any medication; nine were receiving a selective serotonin reuptake inhibitor (SSRI) and seven were receiving levothyroxine. Of the patients with mild cognitive impairment, 11 were not receiving any medication, while eight were receiving cholinesterase inhibitors, four were receiving memantine, eight were receiving an SSRI, and two were receiving a second-generation antipsychotic. Of the patients with Alzheimer's disease, four were not receiving any medication, seven were receiving cholinesterase inhibitors, eight were receiving memantine, seven were receiving an SSRI, and two were receiving a second-generation antipsychotic.

Cognitive Measures

For verbal list learning, we used a version of the Buschke Selective Reminding Test (28) that consisted of six trials of a 12-word list, followed after 20 minutes by free recall and recognition; total recall (trials 1–6) and delayed free recall were the dependent measures. To test memory for stories, we used the Wechsler Memory Scale–Revised logical memory subscale (29); immediate memory and memory after a delay were the dependent measures. To test verbal working memory, we used the WAIS-R version of the digit span test; the raw score was the dependent measure.

To assess speed of processing that involves visual scanning and psychomotor speed, we used the Trail Making Test, Part A (30); time in seconds was the dependent measure. We used the WAIS-R version of the digit symbol test to measure speed of processing; the raw score was the dependent measure. We used the clock drawing test (31) to measure semantic memory-related time, number, and/or linguistic facility and visual motor processing; we used a 10-point scoring system. Finally, we used the semantic fluency test to measure word production and speed; we measured number of words generated in 60 seconds.

Functional Measures

The Alzheimer's Disease Cooperative Study–Activities of Daily Living Inventory (ADCS-ADL) (32) has a range of items assessing both complex abilities (shopping, hobbies, personal appliances) and more basic activities of daily living (including walking and eating). Some of these latter items might dilute the instrument's sensitivity at preclinical stages of the disease, as they are rarely impaired. This 23-item questionnaire was completed by the cognitively healthy group and by caregivers for the participants in the mild cognitive impairment and Alzheimer's disease groups. The total score (possible scores range from 0 to 78) was the dependent variable.

The University of California, San Diego (UCSD) Performance-Based Skills Assessment (UPSA) is a performance-based measure of functional abilities that has ecological validity; it includes analogue measures of real-world activities—for example, planning an activity, such as a trip to the beach or the zoo, determining a route, dialing a phone number, and writing a check. It was validated in middle-aged and older healthy individuals and in an outpatient sample of older patients with schizophrenia (33, 34). Interrater reliability was high, correlation with another measure of function (informant based) was high, and test-retest reliability was 0.92 (33). A second study demonstrated its high correlations with both neurocognitive measures derived from the Clinical Dementia Rating Scale and level of independence in community living in older stable psychotic outpatients (35). Bowie et al. (36) demonstrated in a path analysis that the UPSA appeared to occupy a key node as a mediating variable in the path from cognition to functional disability. The UPSA includes domains involving comprehension and planning; transportation; communication; and financial procedures. Accommodations to the New York metropolitan area (e.g., local maps) were made for all relevant items after consultation with the test's author. We used the composite score derived as the mean of percent correct for each UPSA subtest as our primary dependent measure. We did not administer the household management section of the UPSA because of time constraints and concerns about its lack of contribution to the total score.

Procedure

To establish eligibility, a medical and psychiatric history and a neurological examination were conducted, and the MMSE, the Clinical Dementia Rating scale (using a trained rater), the Geriatric Depression Scale (37), and the Hachinski Ischemia Scale were administered. The cognitive tests listed above, followed by the UPSA, were then administered to each participant. Cognitively healthy participants completed the ADCS-ADL. Informants for participants in the mild cognitive impairment and Alzheimer's disease groups completed the ADCS-ADL.

Statistical Analyses

Statistical analyses were conducted using SAS, version 9.1.3 (SAS Institute, Cary, N.C.). All parametric analyses that contrasted groups on key measures of function and cognition were analyses of covariance (ANCOVAs) (using Proc GLM) in which age and education served as covariates. Multiple regression models (using Proc Stepwise) were used to assess the relation between function and cognition; age and education were forced to enter prior to other independent variables. Receiver-operating-characteristic curve analyses were conducted using Proc Logistic; such analyses have the advantage of providing a single estimate of diagnostic accuracy.

Results

Demographic Characteristics and Cognition

The basic demographic and clinical characteristics of the study participants are summarized in Table 1. The groups differed significantly in age and education by analysis of variance; they did not differ in sex ratio (cognitively normal men, N=19; men with mild cognitive impairment, N=8; men with Alzheimer's disease, N=10).

Enlarge table

The performance of the three groups on cognitive tests and functional measures is summarized in Table 2. On all cognitive tests, significant between-group differences were observed by ANCOVA. Post hoc contrasts generally indicated that cognitively healthy participants outperformed those in the mild cognitive impairment group, who in turn outperformed those in the Alzheimer's disease group.

Enlarge table

Functional Measures

We subjected UPSA scores to ANCOVA parametric analyses. (Age and education served as covariates, given group differences.) Differences among the groups were highly significant (Table 2). Notably, post hoc Bonferroni-corrected contrasts indicated that the mild cognitive impairment group performed significantly worse than the cognitively healthy group (p=0.002). The effect size (Cohen's d) of this difference was large at 0.86.

We examined the ADCS-ADL similarly. Differences among the groups were highly significant by ANCOVA (Table 2). However, post hoc contrasts indicated that the cognitively healthy participants did not significantly differ from those with mild cognitive impairment. The effect size of this latter difference was moderate at 0.56.

Refinement of Analyses in Restricted Samples

Next, we performed a more rigorous analysis in which we excluded any patient with mild cognitive impairment whose ADCS-ADL score was more than 1.5 standard deviations below the mean of the cognitively healthy group. By so doing, we created a "purified" sample that directly and objectively met criteria for preserved function as reflected in the responses of spouses or other family members. Critically for our hypothesis, when we compared the resulting purified mild cognitive impairment group to the cognitively healthy group on UPSA performance by ANCOVA, the groups continued to differ at a highly significant level in Bonferroni-corrected post hoc contrasts (p=0.008).

Because of the difference in age and education between the cognitively healthy group and the mild cognitive impairment and Alzheimer's disease groups, we also conducted an analysis in which we compared a subset of 34 older, slightly less educated cognitively healthy participants (mean age=74 years, mean education=14 years) now matched to the mild cognitive impairment and Alzhei­mer's disease groups on these demographic variables. Differences among the groups on the UPSA remained highly significant (F=64.52, df=2, 80, p=0.0001). Notably, post hoc contrasts indicated that the mild cognitive impairment group performed significantly worse than the cognitively healthy group (p<0.001). The effect size of this difference was large at 1.43. Similarly, the difference between the mild cognitive impairment and Alzheimer's disease groups was significant (p<0.001), with a large effect size at 1.71.

Psychometric Properties of the UPSA

The UPSA was not generally prone to ceiling effects in the cognitively healthy group, nor to floor effects in the Alzheimer's disease group. Thus, only one participant in the cognitively healthy group obtained a score above 90% and only one in the Alzheimer's disease group obtained a score under 10%. For the ADCS-ADL, 14 participants were at ceiling.

As shown in Table 3, the UPSA was generally less skewed than the ADCS-ADL. It also generally demonstrated less extreme kurtotic distributions than did the ADCS-ADL. The ADCS-ADL distribution violated assumptions of normality in the cognitively healthy group at least partly because of ceiling effects. The coefficient of variation was consistently greater in the UPSA than in the ADCS-ADL.

Enlarge table

Receiver-operating-characteristic curves for the UPSA are shown in Figure 1. The area under the curve was 0.84 for the comparison between the cognitively healthy and mild cognitive impairment groups (odds ratio=1.16, 95% confidence interval [CI]=1.07–1.25, p<0.0001)—that is, it indicated that for any randomly drawn pair of participants from these two groups, the probability that the participant with mild cognitive impairment would have a lower UPSA score was 0.84. At a cutoff of p=0.50, sensitivity for identification of cognitively healthy participants was 0.88 and specificity was 0.58. At p=0.44, correct classification was maximal (0.816). For the mild cognitive impairment and Alzheimer's disease contrast (odds ratio=1.16, 95% CI=1.06–1.26, p=0.0009), the area under the curve was 0.88—that is, it indicated that for any randomly drawn pair of participants from these two groups, the probability that the patient with Alzheimer's disease would have a lower UPSA score was 0.88. At p=0.50, sensitivity for identification of participants with mild cognitive impairment was 0.85 and specificity was 0.73. At p=0.44, correct classification was maximal (0.813). Thus, sensitivity and specificity were acceptable for both contrasts, and group separation was robust.

Relationship Between the Functional and Cognitive Measures

Using a variant of stepwise regression in which age and education were forced to enter, we determined that a three-variable solution comprising the Trail Making Test, Part A (a test of processing speed, complex visual attention, and scanning); logical memory, immediate (a measure of verbal episodic memory); and semantic fluency (speeded semantic access and executive control), predicted 51% of the remaining variance in the UPSA. Beta weights, partial R2 values, and significance values are listed in Table 4.

Enlarge table

The UPSA and the ADCS-ADL were also significantly correlated using Spearman's rank order method (r=0.63, p=0.0001). The scatterplot is shown in Figure 2. However, the magnitude of the correlation suggested that the amount of nonshared variance was also substantial (>0.60).

Discussion

Our study yielded four important findings. First, we demonstrated that patients with mild cognitive impairment showed significant compromises on the UPSA, a performance-based measure of everyday function that comprises ecologically relevant tasks. The effect size between the cognitively healthy participants and those with mild cognitive impairment was large (>0.85). Second, when we restricted our analysis to only participants whose everyday function was deemed within normal limits on an informant-based measure, we nevertheless continued to observe significant impairments on the UPSA. Third, we found that cognitive scores in speed of processing, episodic memory, and semantic processing and fluency accounted for a significant share of the variance on the UPSA (about 0.50), thus suggesting a principled relationship between the two types of measures. Fourth, the psychometric properties of the UPSA were good.

Our results indicate that patients with mild cognitive impairment have functional impairments, as indicated on a performance-based measure of everyday functional ability. Moreover, in our stringent and critical analysis of a purified sample, we were able to demonstrate that patients with mild cognitive impairment had impairments even when there was otherwise "objective" informant-based information suggesting preserved function. These results argue against the notion of mild cognitive impairment exceptionalism—that is, the idea that patients with mild cognitive impairment, unlike those with nearly all other neuropsychiatric disorders, and despite their marked impairment in an important domain of cognition, have preserved everyday function.

Early conceptions of mild cognitive impairment suggested that little change would occur in daily function in the face of ongoing cognitive decline (38). As this view evolved, subtle impairments in instrumental activities were included in the consensus criteria for amnestic mild cognitive impairment (21). However, the specific instrumental performances of patients with mild cognitive impairment have not been well characterized (38). Thus, the diagnosis of mild cognitive impairment did not necessarily imply that there were no functional consequences, but rather it indicated that patients did not exhibit gross functional impairment in the course of normal daily activities as observed and reported by a knowledgeable observer but could have subtle, undetected impairment. (See the Patient Perspective box for some of the clinical implications.)

The method we used in this study allowed us to align cognitive and functional impairments. Thus, the majority of the variance in the UPSA could be predicted by key cognitive measures, including verbal episodic memory, semantic processing, executive ability, and speed/attention. This approach also provided convergent validity for findings of compromised function in the mild cognitive impairment group. Moreover, such a principled relationship has often been difficult to discern in informant-based measures of function (39).

A pragmatic advantage of performance-based measures is that they are free of the possible informant biases or lacunae in knowledge that may distort informant-based reports. From a psychometric standpoint, the UPSA was also sensitive, was not prone to ceiling or floor effects, and demonstrated acceptable receiver-operating-characteristic curves. As the field moves to earlier diagnostics and interventional strategies, the psychometric strengths of the UPSA also make it attractive. It might also be an appropriate coprimary endpoint in clinical trials of Alzheimer's medications designed to improve cognition and function on instrumental activities. Certainly the need for more sensitive and objective measures of everyday function has been widely discussed in the literature (20, 21, 36, 39–41).

It could be argued that we have created a "straw man" in this article. While several reports have identified functional impairments in cohorts with psychometrically defined mild cognitive impairment (40, 42, 43), including in one study using a narrowly based performance-type measure of function (44), no study has directly contrasted performance-based and informant-based measures, as we did here, and provided empirical support for the predictions that the magnitude of these compromises are large and can be robustly and systematically related to cognitive failures in mild cognitive impairment and Alzheimer's disease. In this manner, it becomes possible to place our results within a broader context of neuropsychiatric disorders so as to better understand the implications of a variety of cognitive impairments for function. Moreover, we note the rather large number of studies of mild cognitive impairment in which preserved function remains a diagnostic criterion, along with its implicit assumptions.

It should be noted that the ADCS-ADL was designed for a very different purpose than that used here, namely, to stage decline in Alzheimer's disease. We deployed it to bring into sharper relief points about conceptualizing mild cognitive impairment, performance-based measures of function, and the pragmatics of assessing function at earlier points in the Alzheimer's disease process. Even so, the effect size of 0.56 that we observed on this measure was moderate and would have been statistically significant in a larger sample. Newer versions of the test specifically tailored to emphasize the more complex activities that are likely to be impaired in mild cognitive impairment show much promise (45). These issues can be thought of as involving detection and sensitivity, which may in turn be dependent on what is being asked and who is being asked. Of course, both types of scales (informant-based and performance-based) need to address cultural variability and sensory and motor disabilities.

Several criticisms have been raised about performance-based measures. These include the idea that they are neuropsychological tests by another name. Our view is more nuanced, in that we believe that the analogue tasks in performance-based measures engage fundamental cognitive operations. We believe that the UPSA is ecologically valid because it assesses the performance of tasks that must be frequently and efficiently managed by individuals living in the United States and similar cultures. The tasks tested by the UPSA are important in and of themselves and may be surrogates for a wider range of activities (e.g., remembering key documents to bring to a doctor's office in the UPSA assessment may also have relevance to remembering documents or items to bring to other types of appointments). Our multiple regression results empirically support this argument. Another argument has to do with the possibility that these measures may be too sensitive—that patients appear to be doing well in their home environment but score poorly on tests. In our view, patients' adaptation to their environment may be dependent on procedural learning and relatively automatic routines that do not accurately reflect the ability to perform more novel, yet ecologically critical, tasks. Such tasks might require the integration of a variety of attentional/speed, semantic, and episodic memory demands that are indirectly captured by the UPSA.

In summary, we found that patients with mild cognitive impairment had significant impairments on a performance-based measure of everyday function and that the magnitude of the contrast between cognitively healthy participants and those with mild cognitive impairment (in effect size units) was greater for the performance-based measure than for an informant-based measure. Furthermore, as predicted, we found a strong relationship between cognitive performance and the performance-based measure of function using multiple regression models. Our work also suggests the need for a reconceptualization of the relationship between cognition and function in mild cognitive impairment so that they can more effectively be aligned.