Evidence for Differential Predictive Performance of the Prime Screen Between Black and White Help-Seeking Youths

Individuals at “clinical high risk” for psychosis are those experiencing recent attenuated psychotic syndromes or other indicators of susceptibility during adolescence or young adulthood, a key period of risk for first-episode psychosis (1). As only 25% of these individuals develop a formal psychotic illness in the years after identification (2), false-positive identification of psychosis risk syndromes limits the capacity of psychosis prevention efforts (3). Evidence suggests that recent trends toward drawing putatively high-risk research participants from the general, non–help-seeking population contributes to the high rates of false positives (4). In conjunction with the developing consensus that high-risk syndromes warrant clinical attention regardless of eventual psychosis (due to frequently high levels of distress and impairment; 5), these findings raise questions about the most appropriate ways to identify individuals on a path toward worsening prognosis. The use of brief, self-report screening instruments prior to clinical assessment referral may contribute to an efficient and cost-effective solution to this problem (6). Self-report screens can indicate one’s probability of meeting high-risk criteria once fully assessed (7) and have strong validity in the prediction of subsequent psychosis (8).

Given that normative experiences and item interpretation can vary as a function of factors related to race, ethnicity, and culture, validation of instruments designed to capture psychological and behavioral abnormalities requires close examination of an instrument’s performance across different racial and cultural backgrounds (9). Historically, many psychometric instruments lack sensitivity to important cultural factors (9–14), suggesting that the validity of psychosis risk instruments may differ between members of racial-ethnic majority and minority populations (10, 15). This problem can contribute to sociodemographic health disparities by limiting the benefits of screening, including early intervention, for members of racial-ethnic minority groups (16, 17).

Decades of research demonstrate that black individuals are more likely than white individuals to be misdiagnosed as having schizophrenia (18–20), further compounding what may be an actual underlying disparity in prevalence rates (21, 22) and quality of treatment (23). Given the importance of early detection and intervention in curbing the burden of serious psychopathology (24, 25), these findings highlight the need to develop screening tools that can signal emerging psychosis among black, help-seeking youths, who may be at risk of both an eventual misdiagnosis of schizophrenia and—paradoxically—the onset of an actual (not misdiagnosed) psychotic disorder.

This study aimed to determine whether the Prime Screen, a commonly used self-report prescreen for clinical high-risk criteria, performs equivalently across black and white help-seeking youths. Building from literature demonstrating limited cultural sensitivity of many psychometric instruments (9, 10, 14) and extending previous work suggesting a strong predictive relation between Prime Screen scores and clinical high-risk status, we examined whether the relation between the Prime Screen and clinical high-risk status was weaker among black participants relative to white participants. To address the possibility that differences in self-rated symptoms could be explained by group differences in levels of clinician-rated psychopathology, clinician bias, or disparities in socioeconomic status (26), we examined rates of high-risk diagnoses across racial groups, the magnitude of relations between Prime Screen scores and clinician-rated positive symptom severity, and whether family income accounted for any differential relation between Prime Screen scores and risk status. In exploratory analyses, we examined the specific Prime Screen items that may contribute to any observed racial differences.

Methods

Procedures

The study took place within the context of an ongoing longitudinal study of psychosis risk that began in 2010. Participants or their parents (if the child was younger than 18) spoke by phone with a trained researcher, who described study procedures and determined eligibility. Visits took place in a private room within university clinics. After providing informed consent, youths completed self-report measures alone while their parents (when present) were interviewed regarding the youths’ psychiatric history. Subsequently, participants completed psychiatric interviews with the researcher. The study was approved by the universities’ institutional review boards.

Participants

Individuals ages 12 to 25 were recruited from community clinics, hospitals, schools, and private practitioners in Baltimore. Additional inclusion criteria required only that participants were receiving mental health services at the time of enrollment. Participants were typically referred for mental health assessment and diagnosis because of suspected emerging psychosis or other psychiatric concerns (e.g., affective disorders). The participants could be divided into three categories: individuals at clinical high risk, a help-seeking control group made up of individuals with active mental health concerns but without clinical high risk or a diagnosable psychotic disorder (such as schizophrenia), and individuals with a diagnosable psychotic disorder. Given interest in the performance of the Prime Screen in predicting psychosis risk among black relative to white youths, participants with psychosis (N=26) and participants who were neither black nor white (N=27) were excluded from analyses.

Measures

Race.

Race was reported by participants or their parents by using a questionnaire concerning demographic characteristics derived from the National Institutes of Health’s definitions for racial and ethnic categories. The response item corresponding to black race was “Black or African American. A person having origins in any of the black racial groups of Africa. Terms such as ‘Haitian’ or ‘Negro’ can be used in addition to ‘black or African American.’” The item corresponding to white race was “White. A person having origins in any of the original peoples of Europe, the Middle East, or North Africa.” Racial subgroups (e.g., groups corresponding to specific African or European descent) were not identified.

Structured Interview for Psychosis-Risk Syndromes (SIPS).

The SIPS is a semistructured, gold-standard interview for identifying and rating the severity of clinical high-risk syndromes (27). Although no study to our knowledge has directly evaluated the cross-cultural performance of the SIPS, a recent comprehensive review of its reliability and validity across the 31 countries in which it has been used found no evidence of differential performance by culture (28). To meet criteria for a psychosis risk syndrome, participants must have experienced one or more attenuated positive psychotic symptoms at least weekly, an illness episode of psychotic-level intensity that was too brief to meet criteria for formal psychosis, or a recent functional decline of 30% or more in the context of schizotypal personality disorder (SPD) or a family history of psychosis. Given evidence that the degree of risk for transition to psychosis is comparable among adolescents with SPD and those meeting other SIPS criteria (e.g., 21% [29]), we included individuals with SPD but no family history or significant functional decline in the high-risk group. The SIPS items for unusual thought content, suspiciousness, grandiosity, perceptual abnormalities, and disorganized communication are rated on a 7-point scale, from 0 (absent) to 6 (severe), and ratings are summed to create a measure of positive symptom severity. All SIPS raters (nine white, one Asian) were certified following an official 2-day workshop and achieved excellent interrater agreement (intraclass correlation coefficient=.82 for positive symptoms; κ=1 for diagnosis). Raters were blind to participants’ Prime Screen scores.

Kiddie Schedule for Affective Disorders and Schizophrenia (K-SADS).

We used the K-SADS to characterize the sample clinically. The K-SADS is a well-validated, semistructured diagnostic interview used to identify DSM diagnoses among youths (30). K-SADS diagnoses are made on the basis of separate interviews with children and parents. Training included expert instruction, rating of audio-recordings, in vivo interview observation and co-rating, and supervised administration until diagnostic agreement with experienced raters was achieved for at least three participants and approval was given by the principal investigators.

The Prime Screen, revised.

The Prime Screen is a 12-item, self-report questionnaire developed by the SIPS authors as a brief way to estimate the probability of meeting clinical high-risk criteria (31). Items are rated on a 7-point scale, with 0 indicating definitely disagree; 1, somewhat disagree; 2, slightly disagree; 3, not sure; 4, slightly agree; 5, somewhat agree; and 6, definitely agree. Participants who endorse two or more items at the level of a 5 or 6 are considered to screen positive. The sum of positive symptoms is strongly correlated with the sum of SIPS-rated positive symptoms (7) and has been shown to predict subsequent transition to psychosis among those at clinical high risk (8). The average administration time of the instrument is 1 minute, 40 seconds, and the Flesch-Kincaid reading-level estimate is 6:8. Cronbach’s alpha for the present sample was .89.

Statistical Analyses

Preliminary analyses.

Prime Screen scores were computed by totaling the number of items endorsed at the level of 5 or 6 (hereafter referred to as Prime Screen “cutoff” scores), consistent with author recommendations and with previous validity studies (7). Primary analyses were also conducted by using the sum of raw Prime Screen scores; because the results were the same regardless of the scoring method, for simplicity the results obtained using raw scores are not reported here but are available upon request. Black and white participants were compared on demographic variables by using chi-square or t tests.

Primary analyses.

To examine whether the relation between Prime Screen scores and clinical high-risk status was weaker among black relative to white participants, a moderated logistic regression was performed in which dichotomous risk status was regressed on race, Prime Screen cutoff scores, and their cross product (race × Prime Screen cutoff score). In a second linear regression, the sum of SIPS positive symptoms was regressed on these same predictors. In the case of moderation, simple effects were computed to examine the effect of Prime Screen cutoff scores on the outcome variable (i.e., probability of meeting high-risk criteria, SIPS positive symptoms) separately among black and white participants (32). Regression analyses were then repeated controlling for family income and the demographic and clinical variables that differed significantly by race.

Exploratory analyses.

To explore whether specific Prime Screen items were differentially related to clinical high-risk status among black versus white participants, a 2×2 (race × risk status) analysis of covariance (ANCOVA) was conducted for each of the 12 raw Prime Screen item scores. ANCOVAs examined mean differences in scores for each item across risk groups within each race. Because of the exploratory nature of these analyses, no correction for multiple comparisons was applied. Finally, we computed sensitivity and specificity values for the entire sample and for the black and white groups separately.

Results

A total of 108 participants (clinical high-risk group, N=43; help-seeking control group, N=65) were included in the analyses, similar in size to several other psychosis-risk screening studies (6). Of the participants, 58 were black and 50 were white (Table 1). Because of incomplete research procedures, Prime Screen scores were missing for eight participants, the sum of SIPS positive symptoms scores was missing for 2, and family income data were missing for 11. Data were excluded pairwise per analysis. The continuous variables of interest displayed acceptable skew and kurtosis (i.e., <2 [33]) (Table 2). Black participants were on average younger than white participants (t=2.68, df=106, p=.009) and were less likely to have a mood disorder (N=41, 47%, versus N=34, 68%; χ²=5.03, df=1, p=.025). Age (r=–.26, p=.008) and mood disorder (t=−2.18, df=105, p=.032) were related to Prime Screen cutoff scores and were considered covariates. The race groups did not differ on any of the other demographic or clinical variables.

As demonstrated in Table 1, the race groups did not significantly differ on rates of high-risk diagnoses or on the severity of Prime Screen cutoff scores or attenuated positive symptoms. Results from a moderated logistic regression, however, revealed a significant interaction between race and Prime Screen cutoff scores in the predicted probability of meeting high-risk criteria (Table 3). Simple effects analyses suggested that higher Prime Screen cutoff scores significantly increased the probability of meeting these criteria for white but not black participants. The effect remained significant when the analyses controlled for household income, age, and mood disorder (b=–.51, Wald χ²=4.66, df=1, p=.031, Exp[B]=.60, 95% confidence interval=.38–.96; see Table S1 in the online supplement). When participants who met criteria for a formal psychotic disorder were included in the high-risk group, the pattern of findings remained the same (see Table S2 in the online supplement).

Table 4 shows means and standard errors of individual Prime Screen item scores, plus results of 2×2 (race × risk status) and within-race ANCOVAs comparing scores on each item across groups (controlling for family income, age, and mood disorder). These analyses sought to determine which Prime Screen items accounted for the differential response pattern among black and white participants, described above. Statistically significant race × risk status interactions were observed for six items (items 1, 2, 5, 6, 8, and 9). For black youths, mean differences between high-risk and help-seeking control groups were substantially smaller (items 1 and 5) or in the opposite direction (items 2, 6, 9, and 12) than was seen among white participants. For these latter items, black participants in the help-seeking control group scored numerically higher than black youths at risk.

Within-race contrasts explored risk group differences on Prime Screen items separately among black and white participants (Table 4). White participants in the control group consistently scored lower than white youths at high risk, whereas a mixed pattern of results was observed among black youths, with black participants in the control group frequently endorsing items at a level comparable to or even numerically greater than those at risk. In the combined sample, sensitivity and specificity of the Prime Screen were .43 and .90, respectively. Splitting by race, these values were .27 and .90, respectively, for the black group and .61 and .90, respectively, for the white group.

A linear regression predicting the sum of positive symptoms from race, Prime Screen cutoff scores, and their cross product revealed no significant interaction, suggesting that the relation between participant-rated Prime Screen scores and clinician-rated positive symptom severity (irrespective of one’s clinical high-risk status; Table 2) was roughly equal across black and white participants.

Discussion

We found that the Prime Screen, a frequently used self-report assessment of clinical high-risk criteria, did not reliably distinguish between black help-seeking youths who were at risk of psychosis and those who were not, even though it did distinguish these groups among white participants. The findings were not explained by differences in income, age, mood disorder, rates of clinical high-risk diagnosis, or clinician-rated symptom severity. Item-level analyses suggested a differential performance across race for most items, suggesting a relatively widespread versus item-specific effect.

A long history suggests that many psychometric instruments do not perform equivalently across cultures (9–14). Instruments may not measure the same constructs across racial-ethnic groups, may use language that conveys different meaning across these groups, or may concern constructs that are more familiar to some groups than others (34). Questionnaires may be inherently subject to certain of these limitations. The Prime Screen, for example, was designed to convey risk level mental experiences by adding contingency words (e.g., “I think that I have felt…” [italics added]), a convention that may have differentially influenced responses across race. Questionnaires also may restrict the opportunity to provide important contextual information associated with endorsements, such as the degree of associated distress or impairment. By contrast, diagnostic interviews allow clinicians to use age- and culturally appropriate language and to clarify the circumstances surrounding endorsements. The addition of a distress scale to the Prime Screen, as included in a similar measure (the Prodromal Questionnaire–Brief; PQ-B), may partly address this issue.

We found that Prime Screen scores among black participants in the help-seeking control group more closely resembled those of participants at clinical high risk (black or white) than those of white participants in the help-seeking control group. Notably, the frequency of high-risk diagnoses and the severity of clinician-rated positive symptoms did not differ between racial groups. These findings are important because they suggest that the black youths in our sample appear highly symptomatic when only their self-reported Prime Screen scores are considered. Following a structured interview administered by a trained diagnostician, however, it appears that black and white participants in this sample do not differ significantly in their clinical level of psychosis risk. Given the history of misdiagnosis of schizophrenia among black individuals, reduced access to health screening and high-quality treatment (16–19), and generally high levels of discrimination and risk factors for psychosis to which people of color are often exposed (22, 35, 36), these findings highlight the need to carefully consider the most appropriate referral and treatment options for black youths who, based on these and other findings, are at increased risk of inappropriate referral, diagnosis, and intervention.

An alternative explanation for our results is that Prime Screen ratings were a more accurate measure of psychosis risk than the SIPS among black participants in the help-seeking control group, but the SIPS clinicians did not accurately rate psychosis-risk symptoms, potentially because of limitations of the instrument or cultural differences between participants and the majority white clinicians. This possibility is unlikely, however, given that all clinician-measured indices of psychopathology among black participants were either equal to or lower than those of white participants, including SIPS-rated positive symptoms, rates of high-risk diagnosis, and DSM diagnoses; formal psychotic disorders are frequently overdiagnosed among black individuals, in contrast with the roughly equal rates of high-risk diagnoses we observed; and clinicians were blind to participants’ Prime Screen scores during assessment. Therefore, our results point to the screen as the primary source of inaccuracy in assessment.

Two general population studies recently found evidence of measurement invariance across multiple racial-ethnic groups for the PQ-B, another tool designed for psychosis-risk screening (37, 38). Although these results may appear to contrast with ours, a critical distinction between these studies and ours is that only our study assessed participants with both a screening instrument and the gold-standard SIPS. Notably, black and white participants in our pooled sample did not differ on their Prime Screen cutoff scores; only when the clinician-rated risk status was considered did a differential response pattern emerge between racial groups. Given that we observed such a pattern for nearly all Prime Screen items, our results suggest that this instrument may not capture the same constructs across racial or cultural populations. The field would benefit from studies incorporating measurement invariance analysis of multiple psychosis-risk screening instruments with direct comparisons against gold-standard assessments.

A strength of our study was its use of a clinical control group to assess the performance of the Prime Screen, a screening tool used in real-world clinical settings. Relative to “healthy controls,” control groups made up of help-seeking individuals are optimal comparators in studies like ours because they are more clinically representative of the population for which the instrument was designed (39, 40). Nonetheless, the participants in our sample at clinical high risk tended to have more DSM diagnoses than the help-seeking control participants, suggesting greater overall illness severity. Although specificity estimates of the Prime Screen were excellent and our main findings held after adjustment for racial differences in mood disorder, because black participants on average presented with fewer DSM diagnoses than white participants, it remains possible that general illness severity contributed to the differential performance of the Prime Screen.

With federal funding for clinical high-risk intervention programs, large-scale dissemination of screening tools is underway. Findings from this study may inform these efforts, but our relatively small sample may not generalize to larger programs with more inclusive recruitment strategies or a broader range of sociodemographic characteristics. Our requirement that participants had already contacted a mental health care provider, for example, likely distinguishes our sample from individuals whose initial psychosis-risk assessment may be their first lifetime contact with services; it is also possible that referral patterns were differentially distributed across clinical or racial groups in our study. Given that screening thresholds may vary by help-seeking status (41) and referral source (42), identifying interactions between idiographic factors such as these may advance early identification efforts.

It is important to consider that self-reported race is only a crude proxy for numerous cultural, historical, geographic, and socioeconomic factors (among many others [43]) that may influence a person’s mental health status or response to questionnaires. Community studies designed to carefully measure these factors would allow researchers to tease apart their relative influences on psychosis-risk screening in ways that our study could not; they may also have enhanced ability to detect influences on racial-ethnic cultural subgroups (e.g., specific Caribbean, African, or European descent). A valuable approach may be to develop a maximally and cross-culturally effective screening tool based on combinations of items from previously validated psychosis-risk questionnaires. Qualitative interviews with respondents of varying backgrounds may help to promote development of novel screening items.

Conclusions

Mental health screening is a critical juncture in pathways to care. The potentially inadequate performance of psychosis-risk screens among black youths may represent a rupture at this junction, further compounding racial disparities in access to accurate diagnosis and treatment. Greater attention to cultural and contextual influences on clinical assessment may foster more accurate diagnosis and early, targeted intervention.