Combining Cognitive Testing and Informant Report to Increase Accuracy in Screening for Dementia

The assessment and diagnosis of dementia in the elderly have assumed new importance with the increasing availability of drugs to treat this condition. Cognitive impairment is central to the diagnosis of dementia. The use of psychometric tests of cognitive performance forms the cornerstone of screening and assessment of the disorder. Alternative approaches to screening for dementia have been proposed (1–3). Of these, informant report is the most promising. This approach involves the administration of a questionnaire to a person who knows the patient well. Questionnaires of this type inquire about aspects of memory and intellectual function in everyday situations compared to earlier in life. In a recent meta-analysis (4), Jorm demonstrated no significant difference between the performance of cognitive tests and informant report in screening for dementia. It would, however, be a mistake to believe that informant report and cognitive tests measure the same attribute of a patient. Correlation between these tests is moderate (2, 5). Further, joint factor analyses of test and informant report items have produced separate factors for each item domain (6).

The potential desirability of using both types of tests when assessing patients at risk of dementia is reinforced when the complementary shortcomings of each type of test are considered. Cognitive tests may be affected by education and premorbid ability. Poorly educated patients may be misclassified as demented, and dementia in well-educated subjects may be missed (7). Cognitive testing also requires intact sensorimotor and language faculties. Informant report questionnaires have been shown to be uncontaminated by premorbid intelligence and education, and they are unaffected by patients’ physical disabilities. However, scores on these tests may be influenced by noncognitive factors, such as the affective state of the patient and the informant, the personality of the patient, and the quality of the relationship between the patient and informant (4).

While clinical practice in the assessment of dementia often includes an informal interview with a relative of the patient, only the administration of an informant report questionnaire permits a quantitative rating of the changes observed. More important, to our knowledge, there are currently no procedures that allow this information to be combined formally with the results of cognitive tests administered to the patient. Informally, when test results concur, greater confidence in the diagnosis, based on the tests, can be had. Where informant report and cognitive testing suggest different conclusions, the optimal clinical impact of the tests is uncertain. Given the moderate correlation of the tests, such discrepancies will not be infrequent. Beyond simply resolving this problem, there may be an opportunity to combine tests in such a manner that the performance of the combination, as measured by its sensitivity, specificity, and other indices, exceeds that of either test used alone.

The aim of this study was to determine whether information from an informant report questionnaire could be combined with a standard cognitive test to improve performance in screening for dementia in clinical settings. To achieve this aim we studied the most widely used cognitive test, the Mini-Mental State (8), and the most established informant report measure, the Informant Questionnaire on Cognitive Decline in the Elderly (9). There are a large number of ways in which two or more test scores might be combined to make an assessment or reach a screening decision. The choice of methods investigated was influenced by the following factors: 1) the rules should be broadly consistent with the current use of each test, 2) the method of combination should be “transparent”—a black box producing a diagnosis by combining scores in unknown ways was not acceptable, and 3) no special equipment or resources should be needed to undertake the combination.

Methods of analyzing psychometric tests as diagnostic or screening tools are well developed and include calculation of indices such as sensitivity and specificity, as well as techniques such as receiver operating characteristic analysis (10). These methods are generally restricted to individual tests. The development of optimal combinations of tests has received little attention and has focused on issues of sequencing and cost (11). Three methods meeting the aforementioned criteria by which two test scores might be combined were identified: 1) a compensatory rule, 2) a conjunctive rule, and 3) the combination of test scores.

THREE SCREENING METHODS

Compensatory (“Or”) Rule

Two tests may be combined by classifying patients as “cases” if either test has a positive result. Only patients with negative results on both tests are considered as noncases. Cutoff scores for the two tests might be determined empirically and may differ from those used when each test is used alone. This is an appropriate rule to consider when it is believed that one test may pick up information missed by the other because it allows evidence missed by one to be compensated by that found by the other. This is an attractive feature given the different sources of bias that have been identified in the cognitive testing and informant report methods. Consistent with this interpretation, combining two tests in this manner will increase sensitivity above that with either test used alone. However, because only patients with negative results on both tests are classified as noncases, the specificity of the combination cannot be greater than that of either test.

Conjunctive (“And”) Rule

With this combination, only patients with positive results on both tests are classified as cases. Patients with a positive result on only one test or on neither are classified as noncases. This approach may be considered as requiring diagnostic consensus. Reflecting this, such a combination of tests can improve specificity over that of either test used alone, but the sensitivity of the combination can be no higher than that of either test. In the assessment of dementia using the cognitive testing and informant report, this approach may be seen as similar to the structure of the DSM-IV definition of dementia of the Alzheimer’s type, which requires evidence of cognitive deficits (criterion A) and a process of decline from previous levels (criterion C).

Combining Test Values

When the preceding rules are used, cutoff points are determined and applied to each test before combination. An alternative approach involves finding a mathematical function of the two test scores that differentiates cases from noncases. This may be recognized as a classical statistical question. Procedures such as discriminant function analysis (12) and logistic regression (13) are methods of constructing such a function. Both yield a classification or prediction score for which a cutoff point can be determined.

This approach is similar to the “or” rule because a high score on one test may compensate for a low score on the other. However, unlike the situation with the simple logical rule, information about the level of each test is retained in the combination so that the weight given to each test and the form of the function can be determined empirically. A disadvantage of this approach for clinical practice is that it requires some calculation to arrive at the classification score. Further, the calculated score is on a new, arbitrary scale, having none of the attributes of either of the scales of the original tests.

Comparing Methods

While some specific characteristics of the two logical combination rules can be estimated from the characteristics of each test (14), whether cognitive testing and informant report can be combined to improve overall screening performance can only be determined empirically. Therefore, the performance of the three preceding rules was compared to that of each test used alone, and the three rules were compared with each other.

METHOD

Subjects

The subjects were patients at the Geriatric Hospital or at the Memory Clinic of the Hospitals of the University of Geneva School of Medicine. Their ages ranged from 62 to 98 years (mean=80.33, SD=8.12). There were 76 women and 30 men, and 59 patients had an educational attainment below the completion of secondary schooling. Patients were excluded if they had no informant, were not fluent in the French language, or had a serious medical condition or a severe sensorimotor deficit. After complete description of the study to participants, written informed consent was obtained from the patients or their legal guardians and from informants.

Instruments

The Mini-Mental State (8) is a brief test of mental status. It has been demonstrated to have satisfactory psychometric properties. A number of cutoff points for distinguishing dementia cases from noncases are in general use. A score of less than 23 is most commonly used to classify patients as cases (7). The French translation of the Mini-Mental State used in this study has been shown to exhibit properties similar to those of the English version (15).

The Informant Questionnaire on Cognitive Decline in the Elderly is a self-administered questionnaire that asks an informant about changes in an elderly patient’s cognitive performance over the previous 10 years. A French translation has been shown to have properties comparable to those of the English version (2). Studies of this instrument with a variety of subjects have shown high reliability and a good ability to differentiate demented from nondemented subjects (9, 16, 17). A 16-item short form of the original questionnaire has been shown to have discriminatory power almost identical to that of the original 24-item version (18), and thus these 16 items were used in this study.

Procedure

The Mini-Mental State and Informant Questionnaire on Cognitive Decline in the Elderly were used in accordance with the directions given by the tests’ developers. Some patients and informants omitted a small number of responses to the tests. In these cases, a total score was calculated by prorating the items available. The tests were administered separately by trained staff. Diagnoses of dementia according to DSM-IV criteria were made by one of two senior psychiatrists (including R.M.), who used a checklist and were blind to the outcome of the psychometric tests.

Statistical Analysis

Logistic regression was used to determine whether the two tests considered together provided any additional information for predicting dementia over that given by either test used alone. This technique was also used in preference to discriminant function analysis to calculate the function used to classify patients as being demented or not. The choice of techniques was influenced by the unpalatable assumption of equal dispersion of test values that underlies discriminant function analysis (19). The sensitivity and specificity of the results of different rules of combination were calculated over a range of cutoff points for each test. Receiver operating characteristic analysis was used to assess the utility of each test or combination as a screen for dementia (11). Areas under curves for different tests were compared by using the method of DeLong and colleagues (20). Differences in the sensitivity and specificity of different tests or combinations were evaluated by using two-tailed binomial tests.

RESULTS

The DSM-IV criteria for dementia were met by 58 patients.The patients were also clinically assessed for DSM-IV major depressive disorder. Of the 11 patients who met the criteria for depression, two also met the criteria for dementia. The Mini-Mental State scores for all patients ranged from 4 to 30 (mean=22.5, SD=5.6), while the ratings on the Informant Questionnaire on Cognitive Decline in the Elderly ranged from 3.00 to 5.00 (mean=3.87, SD=0.56). As would be expected, the mean scores on both scales differed substantially between the demented and nondemented patients (Mini-Mental State: 19.3 versus 26.5; informant questionnaire: 4.22 versus 3.49) (p<0.0001). A scatterplot of the scores on the two tests is shown in figure 1. The correlation (r) between the tests was –0.58.

Logistic Regression Analysis

Logistic regression was used to predict DSM-IV caseness from the scores on the two instruments. Model 1 in table 1 shows the regression equation built by adding the Mini-Mental State score to a null model. This shows that the Mini-Mental State score alone is a strong and statistically significant predictor of caseness. The possibility of a nonlinear relationship between caseness and Mini-Mental State score was assessed by then adding a quadratic term to the model. This term was nonsignificant and negligible in size. Adding the rating on the Informant Questionnaire on Cognitive Decline in the Elderly to the equation resulted in a significant improvement in prediction. Adding a multiplicative term containing scores on both tests, to represent the interaction between the tests, was not significant. Model 2 in table 1 used the informant questionnaire rating as the basis of the prediction equation. The results mirrored those achieved with the Mini-Mental State. Thus, it can be concluded that each test yields nonredundant information in the prediction of dementia. There was no evidence of a nonlinear relationship for either test or of any interaction between them.

Receiver Operating Characteristic Analysis

Receiver operating characteristic curves were constructed for each test (figure 2), and the area under each curve was calculated. Sensitivity, specificity, and positive and negative predictive values at conventional cutoff points are shown in table 2. The area under the Mini-Mental State receiver operating characteristic curve was 0.92 (95% confidence interval=0.87–0.97), while that for the Informant Questionnaire on Cognitive Decline in the Elderly was 0.89 (95% confidence interval=0.83–0.95). These areas were not significantly different (χ²=0.60, df=1, p>0.05). The performance of each test in this study group was thus comparable to previously obtained values.

The performance of the “or” rule was investigated by calculating the sensitivity and specificity across all possible cutoff points for the informant questionnaire while implementing the rule with fixed cutoff points on the Mini-Mental State. For example, when we used a Mini-Mental State cutoff point of 23/24, the sensitivity and specificity of the rule classifying patients as demented if their Mini-Mental State score was below 24, or if their informant questionnaire score was above a specific value, were calculated for all values of the informant questionnaire scores. This was repeated for all possible Mini-Mental State cutoff points and yielded a series of receiver operating characteristic curves, one for each Mini-Mental State cutoff point. Two such curves are illustrated in figure 2. Unlike normal receiver operating characteristic curves, each curve is bound by the sensitivity and specificity of the cutoff point of the Mini-Mental State used to implement the rule. Thus, it was not possible to use the area under the curve to compare the omnibus performance of the “or” rule with either test used alone. Nevertheless, the curves in figure 2 suggested that useful improvements in test performance could be obtained by using Mini-Mental State cutoff points in the range 22 to 24 in conjunction with the Informant Questionnaire on Cognitive Decline in the Elderly. Table 2 shows test performance statistics for combinations using several cutoff points. Using an informant questionnaire cutoff point greater than 4.0 in conjunction with a Mini-Mental State cutoff point of 23/24 increased sensitivity significantly (p<0.01, two-tailed binomial test) over that obtained by using the Mini-Mental State alone, while not significantly reducing specificity (p>0.50, two-tailed binomial test).

The procedure used to investigate the “or” rule was repeated to test the “and” rule. While modest increases in specificity could be obtained by using a Mini-Mental State cutoff of 25/26 in conjunction with the informant questionnaire, this improvement was not statistically significant for any informant questionnaire cutoff point. The performance of most combinations could be met or exceeded simply by changing the cutoff point when using the Mini-Mental State alone.

Figure 2 shows the receiver operating characteristic curve for the weighted sum of the Mini-Mental State and informant questionnaire derived from the following logistic regression equation:

logit(case) = −1.08 + 3.70 × IQCODE − 0.54 × MMSE [1]

or

where IQCODE is the score on the Informant Questionnaire on Cognitive Decline in the Elderly and MMSE is the score on the Mini-Mental State.

The area under the curve was 0.96 (95% confidence interval=0.92–0.99). Consistent with the logistic regression value already described, this area was significantly greater than that under the curve of either test used alone (informant questionnaire versus combination: χ²=8.44, df=1, p<0.01; Mini-Mental State versus combination: χ²=4.52, df=1, p<0.05).

The logical cutoff point for this combination is that associated with the patient having a greater probability of being demented than not demented (i.e., greater than 0.50). The choice of cutoff points above or below this value would reflect the relative importance of missing true cases and false positive diagnoses in specific situations. Table 2 shows test performance statistics for cutoff points having probabilities of 0.40, 0.50, and 0.60 of meeting the criteria for dementia. Sensitivity at all these points was significantly greater than that achieved by using the Mini-Mental State alone at the 23/24 cutoff point (p<0.001, two-tailed binomial test), while specificity was not significantly reduced (p>0.05, two-tailed binomial test). Conversely, compared to the Mini-Mental State at the 26/27 cutoff point, specificity was significantly greater (p<0.001, two-tailed binomial test) while sensitivity was not significantly impaired (p>0.05, two-tailed binomial test). No statistically significant differences between the weighted sum and the “or” rule were found.

Test Implementation

In clinical settings the use of the “or” rule should present no problems. Implementation of the linear combination rule minimally requires the weighted combination of the two test scores as shown in equation 1. In research settings, such calculations might be regarded as relatively simple. In many clinical settings such calculations may be difficult or error prone and also have the disadvantage of rescaling the outcome on a scale that does not have the familiar characteristics and interpretation of the Mini-Mental State. This problem has been overcome by the construction of a graphical template similar to figure 1, on which the scores of individual patients on the Mini-Mental State and Informant Questionnaire on Cognitive Decline in the Elderly may be plotted. This achieves classification of the patient according to the weighted sum rule while leaving the original scores apparent, and it does not require any calculation. This template is available on request from the authors or from the following World Wide Web site: http://www.mhri.edu.au/biostats/demegraph/

DISCUSSION

The analyses presented here demonstrate that the informant report and cognitive testing are complementary approaches to the assessment of dementia. Used together, these tests provide more information than either test alone. This information can be used to improve the accuracy of screening and diagnosis.

This study has demonstrated that the use of a compensatory rule to combine test scores produces superior performance over either single test. On statistical grounds the performance of the simple logical combination cannot be distinguished from the weighted sum in this study group. The logical rule is extremely simple to use and thus would be expected to be attractive in many clinical situations. The implementation of the weighted sum rule is more complex, requiring either calculation or use of our template.

The advantage of the weighted sum is that the trade-off between the scales is graduated: the more evidence there is that the patient is unimpaired according to one test, the more evidence is required that he or she is impaired according to the other test. Consider a patient with a score on test B that is marginally above the cutoff value. By using the “or” rule, such a patient would be classified as a case regardless of the extent to which the score on test A indicates absence of impairment. As depicted visually in figure 1, greater evidence of absence of impairment on one scale requires greater evidence of impairment on the other, in order for a patient to be classified as demented.

Consistent with the ability of the “or” rule to increase the joint sensitivity of the tests at the possible cost of specificity, there are a priori grounds for preferring the weighted sum rule unless the application is focused on screening with the objective of maximizing the detection of cases of dementia. In situations where the primary goal is screening patients before further investigation, the use of the “or” rule can be readily justified, as it can only increase the sensitivity of the testing protocol and thus help ensure that all actual cases are identified. However, this may or may not be at the expense of reduced specificity. In applications where the sensitivity and specificity are both important and must be balanced, the weighted sum may be preferable.

There was no support for the use of a rule requiring evidence of impairment on both scales. This is despite the resemblance between the structure of the formal classification criteria for the disorder and this rule. The failure of this rule to improve diagnostic performance stems from the high specificity of the Mini-Mental State in the current group of subjects. As a conjunctive rule can improve only specificity and may, at the same time, reduce sensitivity, there was little room to improve overall diagnostic performance at conventional cutoff points. While applying this rule with a higher Mini-Mental State cutoff point did increase specificity, the associated reduction in sensitivity resulted in test performance comparable to that achieved by using the Mini-Mental State alone at a low cutoff point.

It is perhaps too rarely appreciated that test performance statistics including sensitivity and specificity are influenced by the characteristics of the population sampled as well as the instrument being evaluated (21). In the same way that the sensitivity, specificity, and positive and negative predictive values of the individual tests vary according to whether they are being applied in general population or clinical samples, so too the actual performance of their combinations would be expected to vary. This underlines the desirability of replication of this study in a range of populations. It should be noted that the Mini-Mental State performed better than has been observed in many previous studies (7). Thus, in some populations there may be considerably more scope to improve the performance of the Mini-Mental State alone by combining it with the Informant Questionnaire on Cognitive Decline in the Elderly.

From a clinical perspective, the combination of cognitive testing and informant report adds little to the clinician’s burden. The use of the Mini-Mental State in the assessment of dementia is ubiquitous (22). The short form of the Informant Questionnaire on Cognitive Decline in the Elderly comprises only 16 items and is self-administered. In many cases, it is possible for an informant to complete this questionnaire while waiting during the time the patient is examined. If a suitable informant does not accompany the patient, the informant questionnaire can be completed by mail. Thus, adding a formal measure of informant report to clinical assessment need not prolong the clinician’s time. Similarly, either of the proposed methods for combining this test with the Mini-Mental State is straightforward, and the role of the two tests in the decision reached is transparent. While different levels of performance might be obtained and different cutoff values might be necessary to optimize performance, it is unlikely that the mechanism by which performance is enhanced (i.e., the information collected by one test that is not available from the other) would differ in different settings. Thus, there should be some confidence in the generalizability of the method of combination beyond the current study group.

This study has demonstrated that it is possible to achieve a significant improvement in the diagnostic performance of a test of cognitive status (the Mini-Mental State) and a method of informant report (the Informant Questionnaire on Cognitive Decline in the Elderly) by appropriate combination of test results. In the subjects examined, comparable levels of sensitivity and specificity were obtained either by using a weighted sum of test scores on which a suitable cutoff point value was determined or by identifying patients with abnormal scores on either or both tests as cases of dementia. While each method has particular advantages, both are simple to use in clinical settings and can improve screening performance.

Revised version of a paper presented at the 1997 World Congress of Gerontology, Adelaide, Australia, Aug. 19–23, 1997Received Nov. 19, 1997; ; revision received March 13, 1998; accepted April 17, 1998. From the Mental Health Research Institute of Victoria, Melbourne, Australia; and Consultation Mémoire, Hôpitaux Universitaires de Genève, Geneva, Switzerland. Address reprint requests to Dr. Mackinnon, Mental Health Research Institute, Locked Bag 11, Parkville, Victoria, 3052, Australia; [email protected] (e-mail). The authors thank Dr. P. Giannakopoulos for assistance in the diagnosis of patients in this study and Dr. R. Conroy for suggesting the possibility of a graphical implementation of the test combination rule.

TABLE 1

Enlarge table

TABLE 2

Enlarge table