Reward/Punishment Reversal Learning in Older Suicide Attempters

Worldwide, suicide is more common in the elderly than in any other age group (1). Suicide attempts in late life are more lethal than those in midlife (2), with up to one-half ending in death in older men (3). Although cognitive decline is prevalent in old age and may contribute to the heightened suicide risk, little is known about the role of cognitive factors. Initial evidence links late-life suicidal behavior to poor performance on screening measures of cognitive control (4, 5).

Evidence from younger suicide attempters indicates impaired performance on tests of cognitive control (6–9) and poor problem solving (10). To the extent that findings from studies of suicide in younger adults generalize to late-life suicide, they suggest a deficit in cognitive abilities relevant to dealing with life's problems and making decisions. Yet cognitive mechanisms that underpin these relationships remain unclear. In a pioneering study, Jollant and colleagues (11) found impaired decision making in otherwise cognitively intact euthymic younger suicide attempters on the Iowa Gambling Task. Indeed, since a suicide attempt may be viewed as an outcome of a nonoptimal decision process, it is plausible that impaired decision making is a causal cognitive factor in suicidal behavior. However, the mechanisms of impairment on the Iowa Gambling Task are unknown, since the typically reported "net score" (total number of risky choices minus total number of safe choices) does not easily allow for dissection of underlying components. Suicide attempters may fail to represent the reinforcement contingencies on the task, may be deficient in adapting a decision-making strategy, or may have a genuine preference for high-risk choices (12–14).

In this study, we sought to identify the components of decision making that are associated with attempted suicide in old age. We selected a probabilistic reversal learning task, which requires one to "map" an uncertain and changing environment by choosing between two stimuli variably associated with reward or punishment. Poor performance on this task can reflect a problem in mentally representing the environment, such as a representation that is overly static in time (slow reinforcement learning), leading to perseverative responses; an unstable representation that changes too rapidly based on recent feedback and discounts past reinforcement history, leading to excessive switches in choice; and abnormal sensitivity to rewards or punishments. Alternatively, it may result from a suboptimal choice process, regardless of the nature of the representation. Depressed people have been found to switch excessively in response to noncontingent punishment, which suggests that their representations either are unstable or overemphasize punishment, or both (15, 16). Behavioral and neural responses during decision making on this task may be described by Bayesian computational models, which predict the activity of the ventromedial prefrontal cortex (17) and the striatum (18).

To determine whether impaired reward/punishment-based learning is specifically related to attempted suicide, we attempted to dissociate performance on the probabilistic learning task from executive abilities that do not involve reward or punishment. We hypothesized that suicide attempters would show impaired probabilistic reversal learning, reflecting unstable representations that discount past history and are overly focused on recent events. One may call such a view of one's past history "myopic," akin to "myopia for the future," which Bechara and colleagues linked to ventral prefrontal lesions (19). This "myopia for the past" could be related to habitually decreased consideration of one's personal past in the face of recent events (20) during a suicidal crisis. We predicted that this myopia for the past would be distinct from oversensitivity to punishments. We further hypothesized that the suicide attempters' executive abilities would appear intact on a test of forward planning and working memory. Another limitation of the existing literature is that studies that contrast suicide attempters with nonsuicidal individuals (7, 11) cannot determine whether cognitive markers are specific for suicidal behavior or whether they also extend to suicidal ideation—a distinction of great clinical importance. To more strongly link decision making with suicidal behavior, we included a comparison group of elderly persons with serious suicidal ideation but no history of attempt.

Method

Participants

Between May 2006 and October 2008, we recruited 65 participants age 60 and older: 15 depressed suicide attempters, 12 depressed suicide ideators, 24 nonsuicidal depressed elderly, and 14 nondepressed, nonsuicidal elderly comparison subjects. Suicide attempters, suicide ideators, and nonsuicidal depressed elderly were diagnosed as having major depression without psychotic features by the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID; 21, 22). To exclude individuals with clinical dementia and to ensure that participants could engage in computerized assessments, all were required to have a score ≥24 on the Mini-Mental State Examination (23). We excluded elderly persons with sensory disorders that precluded cognitive testing, mental retardation, delirium, neurologic disorders, bipolar disorder, schizophrenia, schizoaffective disorder, and exposure to ECT in the previous 6 months.

All participants provided written informed consent. The University of Pittsburgh Institutional Review Board approved the study.

Suicide attempters. Participants in the suicide attempters group (N=15) had made a suicide attempt; nine of them made their first suicide attempt after age 60; all were inpatients. These participants displayed a high level of suicidal intent during their attempts and expressed severe suicidal ideation; five had made repeat attempts, six had made an attempt within a month of assessment, and nine within the previous year. Suicide attempt history was verified by a psychiatrist, using the interview, medical records, information from the treatment team, and information from family members or friends. We excluded participants with significant discrepancies between these sources. Although none of the suicide attempters had experienced head injuries directly related to attempt, we assessed potential anoxic-ischemic or toxic brain injury, based on the Beck Lethality Scale (24), medical records, and the clinical interview. A study psychiatrist (A.Y.D.) assessed participants for any attempts with a score ≥4 on the Beck Lethality Scale and any history of systemic hypotension lasting >5 minutes or asphyxia or neurotoxic ingestion. One participant met these criteria, and for two participants it was impossible to rule out brain injury during past attempts; we excluded these three participants from sensitivity analyses.

Suicide ideators. Participants in the suicide ideators group (N=12) had thoughts of suicide with specific plan, serious enough to precipitate an inpatient admission (N=10) or an increase in the level of outpatient care (N=2), and no lifetime history of suicide attempt. Suicidal ideation was also very severe in this group.

Nonsuicidal depressed elderly. Nonsuicidal depressed elderly (N=24) were included in the study to detect an association between decision making and suicidal behavior above and beyond cognitive effects of depression. These participants had no current or lifetime history of suicide attempts or suicidal ideation as established by clinical interview, review of medical records, the SCID, and the Scale for Suicidal Ideation (lifetime). Two participants in this group were inpatients. Participants were excluded from this group if they had indirect self-destructive behaviors.

Nondepressed comparison subjects. Nondepressed comparison subjects (N=14) were included as the benchmark group. They had no lifetime history of any psychiatric disorder as determined by the SCID.

Procedures

The study was conducted at a university psychogeriatric inpatient unit and a specialty outpatient clinic for late-life depression. Since we aimed to capture decision making in a state similar to a suicidal crisis, we assessed participants with major depression during an acute depressive episode. Participants were tested within 2 weeks of inpatient admission or at the beginning of outpatient treatment. Depressed participants continued to receive psychotropic medications as clinically indicated. We ensured that none were intoxicated or had withdrawal symptoms at the time of testing. Neuropsychological testing took place in one to two sessions over 1–3 days. Assessors were blind to clinical history and ratings.

Assessments

Cognitive and clinical characterization. Current global cognitive function was assessed with the Dementia Rating Scale (25). Depression severity was measured with the 17-item Hamilton Depression Rating Scale (HAM-D) (26). Burden of physical illness was assessed with the Cumulative Illness Rating Scale adapted for geriatrics (27). We obtained medication lists from pharmacy records. We measured the intensity of pharmacotherapy for the current episode of depression with the cumulative strength score from the Antidepressant Treatment History Form (28), based on antidepressant trial duration, dose, and use of augmenting agents. To capture exposure to psychotropic medications not included in this score, we assessed exposure to sedative-hypnotics, anticholinergic drugs, and opioid analgesics. Intraclass correlation coefficients measuring interrater reliability among our assessors were 0.95 for the HAM-D, 0.97 for the Cumulative Illness Rating Scale, and 0.99 for the Dementia Rating Scale.

Probabilistic reversal learning. The probabilistic reversal learning task requires participants to learn to choose one of two colored rectangles in each of 80 trials. In an initial 40-trial acquisition stage, on 80% of the trials the participant is rewarded for selecting stimulus 1 and punished for selecting stimulus 2; on 20% of the trials, false feedback is delivered such that the participant is punished for selecting stimulus 1 (noncontingent or "probabilistic" punishment) or rewarded for selecting stimulus 2. In the second 40-trial reversal stage, the probabilities are reversed (Figure 1). Our version of the task used symbolic reward (green frame around the stimulus, green "CORRECT" display, high-frequency tone) and punishment (red frame around the stimulus, red "WRONG" display, low-frequency tone). Previous studies of probabilistic reversal learning have found similar behavioral and neural responses to symbolic reward and punishment (29), monetary reward and loss (17), and angry and happy faces (30). Participants were considered to "pass" a stage if they reached a predetermined learning criterion of eight consecutive correct trials. It is difficult to achieve this learning criterion by chance, without mastering the task, since at least one or two trials are likely to include noncontingent punishment. In addition, we used the number of trials before achieving the learning criterion in each stage as a parametric measure of performance.

To perform well on this task, one needs to trade off "staying" (i.e., choosing the previously reinforced stimulus despite occasional noncontingent punishment) and "switching" (i.e., reversing the preferred choice after punishment is received). One strategy is to integrate the reinforcement across a number of trials. The tendency to stay while ignoring punishment feedback leads to perseverative errors, which result from selecting the previously correct stimulus after the reversal of probabilities. Conversely, the tendency to switch too easily results in a large number of probabilistic switch errors when one switches away from the preferentially reinforced stimulus in response to noncontingent punishment.

Control measures: forward planning and spatial working memory. To determine whether suicidal elderly demonstrate deficits that are specific to decision making and can be dissociated from other aspects of cognition, we employed a test of forward planning and spatial working memory, the Stockings of Cambridge test, a modification of the Tower of London task (31).

Statistical Analysis

We used SPSS 15.0 (SPSS Inc., Chicago) and MATLAB 7.6 (MathWorks, Natick, Mass.). All tests were two-sided. We first compared groups on demographic and clinical characteristics using F tests and chi-square tests. For these and all subsequent F tests, we examined post hoc contrasts using the Tukey honestly significant difference test. We then used the chi-square test to compare proportions of participants in each group passing acquisition and reversal stages of the probabilistic reversal learning task. In our follow-up contrasts, we compared each of the two suicidal groups to comparison subjects and to nonsuicidal depressed participants. We then examined the performance pattern in the reversal stage by group using multinomial regression with group as dependent variable and two alternative markers of poor performance—perseverative errors and probabilistic switch errors—as covariates. Our sensitivity analyses used binary logistic regression with passing/failing the reversal stage as the dependent variable. The predictors included group status (dummy-coded) and potential confounding factors: global cognitive functioning, years of education, lifetime or current substance use disorders, and gender.

Computational Model

To test our hypothesis that poor decision making in suicide attempters is explained by abnormally high discounting of past rewards and punishments, we employed a computational modeling procedure in which the degree to which participants incorporate the past history into their decisions ("memory") is estimated on the basis of their responses. We adapted a reinforcement learning model common in studies of reinforcement/punishment-based learning (for examples, see references 17, 32, and 33). The model (described in detail in the data supplement that accompanies the online edition of this article) used a Rescorla-Wagner learning rule to choose parameters that afforded the strongest match between choices produced by the model and participants' actual choices, based on the reinforcement history experienced by each participant. The four free parameters were memory, learning rate from rewards, learning rate from punishments, and exploration. Memory reflected the effect of the prior reinforcement history on the choice. Learning rate from rewards and learning rate from punishments reflected the impact of reward or punishment on the last trial on the participant's subsequent choice. Exploration reflected whether choice was random or determined by feedback.

Results

Group Characteristics

The groups were similar in their demographic characteristics (Table 1). Global cognitive functioning as measured by the Dementia Rating Scale was lower in the three depressed groups relative to the nondepressed comparison group. Suicide attempters had a higher lifetime prevalence of substance use disorders compared to nonsuicidal depressed participants (χ²=1,615, df=1, p=0.001); six of the suicide attempters and none of the participants in the other groups had current substance use disorders. Psychotropic exposure was similar across the three depressed groups.

Enlarge table

Probabilistic Reversal Learning

Groups did not differ in their performance in the acquisition stage, suggesting similar overall learning ability. On the other hand, significant differences in performance emerged in the reversal stage (Figure 2). After the reinforcement contingencies were reversed, suicide attempters were less likely than nondepressed comparison subjects (χ²=12.5, df=1, p<0.001) and nonsuicidal depressed participants (χ²=6.5, df=1, p=0.011) to achieve a learning criterion. Suicide ideators did not differ from nondepressed comparison subjects and nonsuicidal depressed participants. These results were mirrored on trials needed to achieve the learning criterion in the acquisition (F=1.4, df=3, 61, p=0.24) and reversal stages (F=5.3, df=3, 61, p=0.003).

Measures of excessive switching (probabilistic switch errors) and perseveration in the reversal stage differentiated the groups (Table 2). Examination of suicide attempters' performance revealed two patterns. First, most attempters made multiple probabilistic switch errors, switching away from the newly reinforced stimulus after noncontingent punishment. Second, a smaller subgroup of suicide attempters made multiple perseverative errors. Suicide ideators did not differ from other groups.

Enlarge table

Reinforcement Learning Model-Based Analyses

Suicide attempters displayed lower memory relative to nondepressed comparison subjects (omnibus analysis of variance, F=2.77, df=3, 61, p=0.049; Tukey honestly significant difference, post hoc: suicide attempters < comparison subjects, p=0.039; Figure 3A). That is, in their choice, attempters relied less on their previous reinforcement history and more on feedback on the last trial. As expected, participants with lower memory made more total switches in their choice (Figure 3B) and more probabilistic switches (Figure 3C). While the three depressed groups, particularly suicide ideators, tended to have a lower learning rate from punishments, group differences were not significant (F=2.52, df=3, 61, p=0.066; suicide ideators < controls, p=0.087; Figure 3D). This seemingly surprising trend was due to perseveration in the depressed groups (Table 2): participants who were less likely to switch their choice following punishment made more perseverative errors (Figure 3E). The learning rate from punishments was also positively correlated with the proportion of probabilistic switches among all switches (Figure 3F). The learning rate from rewards and exploration did not differ between groups (see the online data supplement).

Forward Planning and Spatial Working Memory

On the Stockings of Cambridge test, groups did not differ in the number of problems solved in minimum moves, the number of moves used to solve the problems (adjusting for level of difficulty), or deliberation times (see the online data supplement).

Sensitivity Analyses

The groups still differed in achieving the learning criterion in the reversal stage (Wald statistic=5.8, df=1, p=0.015) after controlling for global cognitive functioning as measured by the Dementia Rating Scale, years of education, and the effects of lifetime history of substance use disorders. After all participants with lifetime substance use disorders were excluded, suicide attempters were still the least likely to pass the reversal stage (χ²=11.6, p=0.009, N=52; suicide attempters versus nonsuicidal depressed: χ²=5.6, p=0.018, N=29). The same was true after excluding participants with current substance use disorders and after limiting the attempter group to the nine participants with a first attempt after age 60 (see the online data supplement). Similarly, the difference between suicide attempters and nonsuicidal depressed participants remained after excluding the one suicide attempter with potential anoxic brain injury from the suicide attempt and the two for whom brain injury could not be ruled out (χ²=10.2, p=0.001). Gender was not related to performance in the reversal stage.

Discussion

We found that in depressed elders, a deficit in probabilistic reversal learning, a component of decision making, is associated with attempted suicide but not with suicidal ideation. Suicide attempters discounted their reinforcement history to a high degree relative to nondepressed comparison subjects, basing their choices largely on the reward or punishment received in the last trial. Some suicide attempters also made multiple perseverative errors. This impairment was not explained by lower global cognitive function, effects of lifetime substance use disorders, or possible brain injury from suicide attempts. Furthermore, it was dissociated from cognitive abilities engaged outside the context of punishment and reward—forward planning and working memory.

Time and Decision Making in Suicidal Behavior

Our results extend earlier findings of impaired decision making in younger suicide attempters with affective disorders (11, 34) to a group of depressed elders with a history of suicide attempt. Suicide attempters in our study showed unstable decision making, which has been described in midlife depression (15, 16) but was evident to a more extreme degree in our study participants. Furthermore, decreased reliance on past history was dissociated from abnormal sensitivity to rewards or punishments. Thus, in counterpoint to the prevailing view that suicidal individuals' representations of reality are distorted in the valence domain (negative cognitive biases; see reference 35, for example), our findings indicate distortions in the time domain. This notion is supported by early empirical findings of altered time perception (36–39) and by self-report evidence that future orientation is weakened in older suicide ideators and attempters (40). A similar process could have contributed to the decaying performance of suicide attempters on the Iowa Gambling Task (11); that is, they may have failed to integrate reinforcement history from a card deck over a number of trials. Assuming that these laboratory observations are representative of individuals' lives, they suggest that people vulnerable to suicidal behavior, faced with change and uncertainty, fail to access their past experiences, making decisions based largely on their present state.

Decision Making and Possible Neural Substrates of Suicidal Behavior

Human lesion (13, 41–43) and imaging (17, 29) studies implicate the ventral prefrontal cortex in reward/punishment-based learning. Thus, our findings converge with evidence of ventral prefrontal pathology from postmortem studies (reviewed in reference 44) and imaging studies of attempted suicide in younger persons (45–47). Furthermore, the suicide attempters' intact performance on the Stockings of Cambridge test could indicate that forward planning and working memory associated with the dorsolateral prefrontal cortex (48) are relatively preserved.

Strengths, Limitations, and Future Directions

Detailed clinical characterization, the inclusion of suicide ideators, and the use of a control task add confidence in our findings. Many cognitive studies of suicide are difficult to interpret, since they often combine individuals with major depression, bipolar disorder (11), and schizophrenia (9). Unipolar depression is the most common antecedent of late-life suicide (49–51), and thus our study focused on elders with major depression. It is unclear whether our findings can be generalized to other psychiatric disorders, although in midlife the association between impaired decision making and attempted suicide appears to transcend diagnostic boundaries (34). Furthermore, we cannot assume that impaired reward/punishment-based probabilistic learning is unique to attempted suicide. While in our small sample it was not related to substance use, similar impairments have been described in pathological states characterized by impulsivity, such as pathological gambling (52) and behavioral toxicity of dopamine agonists (53, 54). Small group sizes and case-control design comprise the main limitations of this study. The lack of effect of depression on forward planning and working memory raises questions about the sensitivity of the Stockings of Cambridge test as a control test—a concern partly mitigated by the presence of robust test difficulty effects. Furthermore, suicide attempters paid less attention to their reinforcement history than did comparison subjects but did not differ significantly from depressed elders. This observation cannot rule out an alternative hypothesis that distortions in the time domain are associated with depression and not specifically with suicidal behavior. Finally, our reinforcement learning model is limited in the extent to which it captures human decision making. More complex formalisms, such as hidden Markov models (17) or models based on production rule learning (55), may be needed to characterize decision making in suicidal behavior.