Modulation of the Startle Response and Startle Laterality in Relatives of Schizophrenic Patients and in Subjects With Schizotypal Personality Disorder: Evidence of Inhibitory Deficits

Schizophrenia is a disorder characterized by deficits in attention, cognition, and information processing (1, 2). Through the use of psychophysiological and neuropsychological measures of information processing, these abnormalities have been quantified, allowing investigators to understand more about the phenomenology, pathophysiology, and genetics of schizophrenia (1). The study of vulnerability markers (i.e., endophenotypes) in nonschizophrenic relatives of patients with schizophrenia provides a means of assessing phenotypic traits of the schizophrenia spectrum that may not be clinically evident (3). Because relatives of schizophrenic patients may have some genetic susceptibility to schizophrenia but do not have the potential experimental confounds resulting from chronic illness, medication, and hospitalizations, this research can provide important information regarding specific trait-linked abnormalities that may reflect core deficits of patients with schizophrenia spectrum disorders.

Prepulse inhibition of the startle response is an operational measure of sensorimotor gating (4). Weak prestimuli presented at brief intervals (e.g., 100 msec) before a startle-eliciting stimulus reduce the magnitude of the blink reflex component of the startle response. Prepulse inhibition is a stable neurobiological marker with high reliability across repeated test sessions (5). Prepulse inhibition occurs in many species and across multiple sensory modalities (4, 6). Studies suggest that in infrahumans, forebrain circuitry regulates the contralateral facial effectors of prepulse inhibition. Thus, sequential projections from the limbic cortex through the striatum, pallidum, and pontine tegmentum to the nucleus reticularis pontine caudalis maintain an ipsilateral relationship, and efferents from the nucleus reticularis pontine caudalis cross to innervate the contralateral facial motor nucleus. The degree to which this nucleus reticularis pontine caudalis to contralateral facial motor nucleus projection crosses the midline appears to increase with phylogenetic advancement. In rats, nucleus reticularis pontine caudalis cells project bilaterally to innervate both sides of the contralateral facial motor nucleus, whereas this projection becomes predominantly contralateral in cats and to an even greater degree in higher-order species (7–10). Decrements in prepulse inhibition have been observed in individuals who are thought to have abnormalities within neural circuitry that modulates prepulse inhibition (11, 12). Patients with schizophrenia (4) and schizotypal personality disorder (13) have been shown to have less prepulse inhibition than comparison subjects. The findings of reduced prepulse inhibition in subjects with schizotypal personality disorder suggest that prepulse inhibition is a potentially useful neurobiological marker for intermediate phenotypes of schizophrenia spectrum disorders.

The aims of the current study were to assess prepulse inhibition of the startle response in nonschizophrenic relatives of patients with schizophrenia in order to determine whether these measures identify trait-linked deficits in sensorimotor gating. To our knowledge, this is the first report of prepulse inhibition in relatives of schizophrenic patients. First, we hypothesized that consistent with the findings in schizophrenic patients (4) and subjects with schizotypal personality disorder (13), relatives of schizophrenic patients would have reduced right-side prepulse inhibition of the startle response. A second goal of the study was to assess prepulse inhibition of the right eye-blink response compared to that of the left eye-blink response in subjects with schizotypal personality disorder and nonschizophrenic relatives of patients with schizophrenia to determine whether any lateralized deficits in prepulse inhibition were present. Such a finding would be consistent with other laterality findings of right versus left hemisphere dysfunction in patients with schizophrenia (14–16).

Method

Subjects

Twenty-five comparison subjects, 23 patients with schizophrenia, 34 relatives of the schizophrenic patients, and 11 subjects with schizotypal personality disorder participated in this study. Demographic data for subjects in each group are presented in Table 1. Subjects provided written informed consent after receiving an explanation of the study. Subjects with a history of major medical or neurological disorders or significant drug abuse were excluded. Additionally, urine toxicology tests were used to screen all subjects for current drug use. The comparison subjects were recruited through newspaper advertisements and had no history of axis I or II disorders as assessed by the Structured Clinical Interview for DSM-IV (SCID-I) and the Structured Clinical Interview for DSM-IV Personality Disorders (SCID-II) or any family history of schizophrenia in a first- or second-degree relative, determined through self report. Established interrater reliability for the SCID in our laboratory is 0.98 (17).

The patients with schizophrenia and their family members were recruited through inpatient and outpatient facilities at University of California, San Diego and the San Diego Alliance for the Mentally Ill. All subjects received the SCID or SCID Non-Patient Version to confirm the diagnosis of schizophrenia in the probands and to assess for axis I disorders in relatives. Four families had a confirmed history of schizophrenia in other family members in addition to the tested proband, while an additional four reported a possible history of schizophrenia that was not confirmed. Two probands from one family were included in the schizophrenic patient group. Nine of the patients with schizophrenia were receiving typical neuroleptic agents, 13 were receiving an atypical neuroleptic (including risperidone) either alone or in combination with a typical agent, and one patient was unmedicated. The schizophrenic patients had a mean Global Assessment of Functioning Scale score of 36.4 (SD=9.7) and a mean of 5.5 psychiatric hospitalizations (SD=9.9).

Axis I disorders among the relatives of the patients with schizophrenia included histories of major depression (N=9), bipolar disorder (N=1), anxiety disorder (N=2), alcohol dependence in remission (N=5), substance dependence in remission (N=3), and psychosis not otherwise specified (N=1). Five relatives were receiving psychotropic medications (one was receiving risperidone, 1.5 mg/day, plus an antidepressant for depression and anxiety symptoms; three were receiving antidepressants alone; one was receiving a mood stabilizer and an antidepressant for a reported bipolar disorder). The mean total score on the Schizotypal Personality Questionnaire (18) for the relatives (N=32) was 11.7 (SD=7.1).

Subjects with schizotypal personality disorder were recruited from outpatient facilities at the University of California, San Diego (N=3), and by newspaper advertisements (N=5) per our established methods (19). Additional subjects with schizotypal personality disorder were identified through screening of potential comparison subjects (N=3). All subjects with schizotypal personality disorder were assessed with the SCID-I and the SCID-II by one of the investigators (K.S.C.) to identify the diagnosis of schizotypal personality disorder. Five subjects with schizotypal personality disorder reported a history of psychiatric illness: major depression in remission (N=3), anxiety disorders (N=2), and alcohol dependence in remission (N=1). Four subjects with schizotypal personality disorder were receiving psychotropic medication: one was receiving a low-dose neuroleptic (perphenazine, 6 mg/day) plus an antidepressant for depression, one was receiving an antidepressant alone, and two were receiving mood stabilizers. Five reported a family history of paranoia, nervous breakdowns, suicide, or psychiatric hospitalizations, but there were no reported cases of schizophrenia. Three other subjects with schizotypal personality disorder reported a family history of depression. Because their relatives were not assessed, it was not possible to determine whether the subjects with schizotypal personality disorder had a family history of schizophrenia or other psychotic disorders. As a group, the subjects had a mean of 5.9 schizotypal personality disorder symptoms (SD=1.4), a mean Global Assessment of Functioning score of 58.0 (SD=15.2), and a mean total score on the Schizotypal Personality Questionnaire of 29.7 (SD=15.1).

Startle Testing

Subjects were first screened for hearing impairment by using auditory thresholds (exclusion threshold: >45 dB to 500 or 1000 Hz tones). Because of the potential impact of nicotine on prepulse inhibition (20, 21), subjects who were smokers were tested at least 30 minutes after their last cigarette to avoid both a severe nicotine withdrawal state or a maximal nicotine level. Subjects were seated in a lounge chair in a sound-attenuated room. Two small cup Ag/AgCl electrodes were placed below and at the outer canthus of each of eye and on the right and left mastoids; electrode resistances were less than 5 kΩ.

Electromyographic (EMG) activity was band-pass filtered (1–1000 Hz), and a 60-Hz notch filter was used to eliminate 60-Hz interference. A standard amplifier was used to direct recorded EMG activity to a computerized startle response monitoring system for digitization and analysis. The system recorded 250 1-msec readings starting at the onset of the startle stimulus.

Startle stimuli were presented binaurally through headphones. Each test began with a 5-minute acclimation period consisting of 70-dB[A] broadband noise that continued as background noise throughout the test. The startle paradigm used a pulse (a 115-dB[A], 40-msec noise burst) presented either alone or following a prepulse (an 86-dB[A], 20-msec noise burst that preceded the pulse by 30 or 120 msec). The paradigm began with one block of five pulse-alone stimuli. This block was followed by two blocks of 24 trials, which consisted of six trials each of the two prepulse conditions and 12 single-pulse stimuli presented in a fixed, pseudorandom order. The paradigm ended with one block of five more pulse-alone stimuli for a total of 58 trials. The intertrial interval averaged 15 seconds (range=8–22 seconds).

Initially, all trials were screened for errors according to our established criteria, and these trials were eliminated from further analysis (4). Startle data were then analyzed by trial type by using wave-form averaging as per established methods (5). After applying a 10-point smooth, the baseline magnitude was assessed by using the average of all points between 5 and 20 msec from stimulus onset of nonrejected trials. The magnitude and latency of the peak startle response (highest point relative to baseline between 30 and 100 msec) were determined for the three different trial types within each block.

The following startle measures were examined: 1) reactivity: the magnitude of response; 2) prepulse inhibition: the percentage change in startle magnitude to prepulse plus pulse versus pulse-alone trials; 3) prepulse asymmetry: the arithmetic difference in prepulse inhibition of the right eye-blink response minus prepulse inhibition of the left eye-blink response; 4) habituation: the decrement in the magnitude of the startle response across the test session; and 5) startle response latency: the time between stimulus onset and the peak magnitude of the startle response.

Statistics

Demographic data were analyzed by using an analysis of variance (ANOVA) design or chi-square. Startle measures were assessed by using a repeated measures ANOVA design and post hoc tests for significant interactions and main effects. Post hoc tests for non-hypothesis-driven multiple analyses used Bonferroni corrections. To illustrate the magnitude of group differences, effect sizes for the startle parameters of the schizophrenic probands, their relatives, and the subjects with schizotypal personality disorder versus comparison subjects were also calculated. All analyses were performed by using SPSS for Windows 9.0 statistical software.

Demographic variables including gender, age, and smoking status that have been shown to affect startle measures were used as covariates or grouping factors in the analysis of startle variables (20–29; J. Ellswanger and D. Braff, personal communication, 1999).

Results

Demographic Data

As seen in Table 1, the groups differed significantly in age (F=3.30, df=3, 89, p<0.05) and education (F=4.75, df=3, 89, p<0.005). Post hoc Scheffé analyses demonstrated significant (p<0.05) differences between the schizophrenic patients and comparison subjects in age and education. Because startle parameters such as startle magnitude have been shown to be significantly associated with age (24–28), age was used in all ANOVAs as a covariate. The groups did not differ in self-reported handedness but did differ significantly in gender ratio (Pearson χ²=12.28, df=3, p<0.01) and in the ratio of smokers versus nonsmokers (Pearson χ²=17.30, df=3, p<0.001). The significant difference in gender ratio was accounted for by the greater number of men than women in the schizophrenic proband group, whereas all other groups had a greater number of women. Similarly, the schizophrenic proband group included a greater number of smokers than nonsmokers, whereas the opposite was true in other groups. Because gender and nicotine have been shown to influence startle and prepulse inhibition measures (20, 21, 29, 30), separate analyses that used gender and smoking as grouping factors are presented despite small cell sizes. Although there was a lack of difference in handedness between groups, measures potentially affected by cerebral asymmetry were analyzed by using right-handed individuals because of the variability of cerebral organization in left-handed individuals (31, 32).

Startle Measures

Startle reactivity

Startle reactivity was examined across the test session by using data from pulse-alone startle trials in a block-by-eye design that included age as a covariate. The group effect was not significant, which suggests that the four groups did not significantly differ in startle reactivity. There was a significant block effect (F=32.02, df=3, 264, p<0.001) that reflected startle habituation (Figure 1), and the group-by-block interaction was not significant, which suggests that the rate of habituation did not differ between groups. There was a significant effect of age (F=7.64, df=1, 88, p<0.01). Pearson correlations showed a significant reduction of right eye-blink startle magnitude with increasing age (r=–0.31 to –0.36, df=91, p<0.005), but these correlations did not reach significance for the left eye blink. When gender was added as a variable, the main effect of gender on startle reactivity was not significant nor was there a significant age-by-gender interaction. The inclusion of smoking history revealed a significant effect of smoking on startle reactivity (F=5.22, df=1, 84, p<0.05) but no interaction effects. Estimated marginal means demonstrated that smokers had smaller startle response magnitudes relative to nonsmokers (smokers: mean=35.30 digital units, SD=30.97; nonsmokers: mean=66.50 digital units; SD=98.80).

Prepulse inhibition

To test the a priori hypothesis that the relatives of schizophrenic patients would have deficits in right-side prepulse inhibition, an ANOVA was performed for right-eye data that included the two interstimulus interval conditions, four groups, and age as a covariate. Because of the reduction in startle magnitude in the pulse-alone condition across the test session, only block 2 data were used to analyze prepulse plus pulse data, since small startle responses late in the session did not allow for the analysis of inhibition of the startle response in all subjects. There were significant effects for interstimulus interval (F=9.57, df=1, 88, p<0.005), group (F=3.66, df=3, 88, p<0.05), and age (F=8.56, df=1, 88, p<0.005) but no significant interactions. Post hoc analyses of the significant group effect that used estimated marginal means with least significant difference analyses revealed that both the nonschizophrenic relatives and the subjects with schizotypal personality disorder significantly differed (<0.05) from the comparison subjects, whereas there was not a significant difference between schizophrenic patients and comparison subjects. When prepulse inhibition data were assessed by using effect sizes (Table 2), the largest effects were in right eye-blink prepulse inhibition in the 30-msec interstimulus interval conditions. All three groups (schizophrenic probands, their relatives, and subjects with schizotypal personality disorder) had larger effect sizes (>0.80) than comparison subjects. Figure 2 shows the group distributions for the 30-msec interstimulus interval condition. When one standard deviation below the mean of the comparison subjects was used as a cutoff, 20% of the comparison subjects, 44% of the schizophrenic patients, 47% of the relatives, and 82% of the subjects with schizotypal personality disorder had prepulse inhibition values below the cutoff.

Post hoc analyses were performed to assess the effects of comorbidity, psychotropic medication use, and family history in the three subject groups. The 16 of 34 nonschizophrenic relatives with a past psychiatric history did not statistically differ from those without a psychiatric history on prepulse inhibition measures. Subjects with schizotypal personality disorder with a current or past comorbid axis I disorder (N=6) had better prepulse inhibition than those with no history of axis I disorder (N=5), but the difference did not reach significance (F=4.64, df=1, 9, p=0.06). To determine whether the relatives and subjects with schizotypal personality disorder who were receiving psychotropic medications differed from those who were not, each of the groups was divided according to psychotropic medication use and the analyses were repeated. There were no differences in prepulse inhibition between those relatives who were receiving a psychotropic medication (N=5) and those who were not (N=29), but the subjects with schizotypal personality disorder who were receiving psychotropic medication (N=4) performed better than those who were not (N=7) (F=6.18, df=1, 8, p<0.05). The subjects with schizotypal personality disorder with a family history of psychiatric illness (N=5) did not differ significantly differ from those without a family history of psychiatric illness (N=6) on prepulse inhibition. The 13 schizophrenic patients who were receiving an atypical neuroleptic were compared to the nine patients who were receiving a typical neuroleptic on prepulse inhibition, and there were no group differences.

Prepulse inhibition was also analyzed in a condition-by-eye design with age as a covariate (Figure 3). There were significant group (F=3.28, df=3, 88, p<0.05), condition (F=14.48, df=1, 88, p<0.001), and age (F=6.37, df=1, 88, p<0.05) effects but no interactions. Further analysis of the significant group effect with estimated marginal means and Bonferroni confidence-interval adjustments revealed reduced—but not statistically significant (p=0.12)—prepulse inhibition in relatives versus comparison subjects, whereas the difference between the schizophrenia spectrum disorder groups and comparison subjects did not reach significance. Pearson correlations revealed a significant correlation between age and prepulse inhibition in the 30-msec interstimulus interval condition for the right eye only (r=0.25, df=91, p<0.05). When gender was included as a grouping factor, there were no significant gender or interaction effects. When smoking history was also included as a grouping factor, there was no significant main effect of smoking, but a significant eye-by-smoking interaction was revealed (F=6.19, df=1, 84, p<0.05). When estimated marginal means were used, smokers as a group had greater prepulse inhibition of the right eye blink (smokers: mean=55.20, SD=19.93; nonsmokers: mean=41.00, SD=29.60), whereas there were no between-group differences on the left (smokers: mean=42.00, SD=39.31; nonsmokers: mean=41.70, SD=27.20).

Prepulse inhibition asymmetry

Prepulse inhibition asymmetry, presented in Figure 4, was analyzed by using self-reported right-handed individuals. There were no significant main or interaction effects, and the group effect fell short of significance (F=2.08, df=3, 74, p=0.11). All three study groups (schizophrenic probands, their relatives, and subjects with schizotypal personality disorder) differed from comparison subjects by small to moderate effect sizes (Table 2). When gender was added as a grouping factor, there was no significant main effect for gender nor were there interaction effects. The addition of smoking as a grouping factor revealed a significant main effect for smoking (F=6.09, df=1, 70, p<0.05) but no interaction effects. The main effect of smoking status was accounted for by greater asymmetry in smokers (estimated marginal mean=13.78, SD=28.49) than in nonsmokers (estimated marginal mean=–1.00, SD=21.92).

Startle habituation

Startle habituation was assessed between blocks 1 and 2, where the rate of habituation is greatest, and between blocks 1 and 4 in two separate analyses. There were no significant group main effects in either analysis, but there was a significant age effect for both (block 1 versus 2: F=3.95, df=1, 88, p<0.05; block 1 versus 4: F=8.75, df=1, 88, p<0.005). There were no significant interactions in either analysis. Significant positive Pearson correlations between age and habituation were evident only for block 1 versus block 4 (right eye: r=0.25, df=91, p<0.05; left eye: r=0.27, df=91, p<0.01). When gender was added as a grouping factor, there were no significant gender or interaction effects in either analysis. The addition of smoking history as a grouping factor did not produce a significant main effect for smoking or interaction effects in either analysis.

Startle latency

Startle latency, illustrated in Figure 5, was analyzed in a condition-by-eye design with age as a covariate. There were no significant group or age effects, but there was a significant condition effect (F=13.33, df=2, 176, p<0.001) due to latency facilitation in the 30-msec pulse-with-prepulse condition. There was also a significant age-by-condition interaction (F=6.13, df=2, 176, p<0.005). Pearson correlations between age and latency variables showed significant correlations between the latency for the single-pulse (r=0.25, df=91, p<0.05) and the 30-msec interstimulus interval condition (r=0.26, df=91, p<0.05) for the right eye blink but not the 120-msec condition or for the left eye blink. When gender was added as a grouping factor, there were no significant gender or interaction effects. The main effect for smoking history was not significant, but there was a significant group-by-condition-by-smoking interaction (F=3.38, df=6, 168, p<0.005). When the ANOVA was parsed into smokers and nonsmokers and the two conditions (30- and 120-msec interstimulus interval), there were no significant group or interaction effects remaining to clearly account for this finding.

Discussion

The present findings show for the first time that nonschizophrenic relatives of patients with schizophrenia have right-side prepulse inhibition deficits similar to those exhibited by schizophrenic patients and subjects with schizotypal personality disorder. Prepulse inhibition deficits were more prominent in the right eye blink in all three groups, and these groups tended to have less prepulse inhibition asymmetry than comparison subjects. The 30-msec interstimulus interval condition best separated groups in terms of prepulse inhibition performance, with 20% of the comparison subjects, 47% of the relatives, 82% of the subjects with schizotypal personality disorder, and 44% of the schizophrenic patients showing prepulse inhibition deficits (less than one standard deviation below the mean of the comparison subjects).

A secondary finding was that smokers within all groups had greater prepulse inhibition on the right than nonsmokers, an increase in the normal (right greater than left) prepulse inhibition asymmetry, and reduced startle reactivity. The greater proportion of smokers within the schizophrenic group may have accounted for the better prepulse inhibition in the 120-msec condition in the schizophrenic probands than in their relatives and subjects with schizotypal personality disorder, who had fewer smokers.

The three study groups (schizophrenic probands, their relatives, and subjects with schizotypal personality disorder) did not differ from the comparison group in startle reactivity, latency, or habituation. All groups clearly responded to the prepulse stimuli by demonstrating latency facilitation in the 30-msec condition. The lack of habituation differences in the three groups differs from previous reports, which showed less habituation in both schizophrenic patients (4) and subjects with schizotypal personality disorder (13) and may reflect differences in stimulus conditions across these studies. Previous studies (4, 13) did not use separate blocks of pulse-alone startle stimuli at the beginning and end of the session as in the current study and included additional prepulse plus pulse conditions plus tactile stimuli at the end of the session. Multiple pulse-alone startle stimuli at the beginning of the session in this study were specifically designed to produce rapid habituation, permitting analysis of prepulse inhibition during periods of relatively stable startle magnitude. As a consequence, this may have caused a “floor effect” in which all subjects were near maximal habituation levels and the magnitude of the differences between groups was not as large.

Consistent with previous reports (23–26, 28), significant age effects were noted in startle reactivity, latency, prepulse inhibition, and habituation. Older subjects were less reactive to startle stimuli and had longer startle response latencies, increased prepulse inhibition, and increased habituation. The finding of increased prepulse inhibition in older subjects differs from previous reports of no change in prepulse inhibition with age in comparison subjects (23; J. Ellswanger and D. Braff, personal communication, 1999) or reduced prepulse inhibition in older schizophrenic patients (24, 25). These differences may reflect interactions of age with a variety of factors, including clinical status, menopause, and medication history, which could not be examined with adequate power in this study.

Within this mixed population of schizophrenic probands, their relatives, subjects with schizotypal personality disorder, and comparison subjects, we did not observe significant gender effects in prepulse inhibition. The lack of significant gender effects in this study, relative to that reported in comparison subjects (22), may have reflected a complex effect due to the range of diagnoses in the present study and the fact that many of the women were in the menopausal age range, where gender differences are often less prominent (33).

Conclusions

This is the first report, to our knowledge, of sensorimotor gating deficits in nonschizophrenic relatives of patients with schizophrenia. Additionally, the present findings now replicate a previous report (13) of sensorimotor gating deficits in subjects with schizotypal personality disorder, who have phenomenological and familial (and perhaps genotypic) links to schizophrenic patients. Together, these data suggest that prepulse inhibition distinguishes an intermediate phenotype that is present in schizophrenia spectrum disorders. Larger sample sizes are required to assess heritability and cosegregation of prepulse inhibition within families of patients with schizophrenia and the potential impact of relatedness of individuals on statistical analyses. Prepulse inhibition has been shown to have high heritability in mouse models (34, 35), but this work has yet to be extended to humans.

Another potentially important finding in this study is the prepulse inhibition laterality data. The prepulse inhibition deficits of the schizophrenic patients, their relatives, and the subjects with schizotypal personality disorder were primarily of the right eye-blink response, which suggests that the inhibitory gating deficits of patients with schizophrenia spectrum disorders is differentially associated with left hemisphere modulation of prepulse inhibition. The findings of a right-side prepulse inhibition deficit is consistent with longstanding findings of left hemisphere dysfunction across a variety of measures in schizophrenia spectrum disorders (14, 15). The reduction of the normal prepulse inhibition asymmetry in schizophrenia spectrum disorders adds support to the theory that psychotic disorders are due to developmental anomalies affecting cerebral asymmetry (36–39). A history of smoking appears to be associated with the differential enhancement of prepulse inhibition on the right side, producing more prepulse inhibition asymmetry. Because smoking was most common among the patients with schizophrenia, reduced right eye-blink prepulse inhibition in schizophrenic versus comparison subjects (effect size=0.85) was likely underestimated in the present study. Nicotine and smoking have been found to play an important role in P50 event-related potential sensory gating (40). Although previous reports in humans (20, 21) and animals (28, 29, 41) have noted enhanced prepulse inhibition with nicotine, a laterality difference has not been reported. Further work is needed to explore the relationship between smoking, prepulse inhibition, and laterality of the observed effects.

TABLE 1

Enlarge table

TABLE 2

Enlarge table