Lack of Representation in Psychiatric Research: A Data-Driven Example From Scientific Articles Published in 2019 and 2020 in the American Journal of Psychiatry

Across assigned sex at birth and racial, ethnic, sexual (e.g., bisexual, gay, lesbian, queer), and gender (i.e., one’s internal sense of their gender) identities, there are important differences in prevalence and determinants of psychiatric illness, access to psychiatric care, and outcomes (e.g., 1–4). However, a growing body of research has shown a lack of representation and missing sociodemographic information regarding racial and ethnic identities in psychiatric research (5–9). While less research has examined representation of people with minoritized (i.e., experiencing discrimination, prejudice, and reduced power because of historical and current social hierarchies) sexual or gender identities, one review showed a near-absolute lack of description of these identities (10).

While previous reviews convey a clear message that lack of representation has been a persistent issue in psychiatric research, they have focused on specific disorders (e.g., alcohol use disorder) or on specific subpopulations (e.g., women), or have spanned decades. Research spanning psychiatric illnesses, examining multiple sociodemographic characteristics, and focused on recently published science will substantively add information about the state of the field today and provide a benchmark to track progress.

In this study, we analyzed published scientific articles in the American Journal of Psychiatry in 2019 and 2020 (capturing articles accepted for publication prior to the murder of George Floyd and subsequent calls for increased equity in science) to characterize representation across racial, ethnic, gender, and sexual identities. Inaccurate labels and categories, acknowledgment of representation as a limitation, and health inequities or disparities as a focus of the article were also assessed.

Methods

Given the lack of description of sociodemographic data of individual (human) samples, meta-analyses, animal studies, commentaries, editorials, reviews, and letters were excluded, resulting in the identification of 125 scientific articles. Two independent raters (S.L.P., P.P., C.E.V., and K.L.) reviewed all components of each article and supplementary material for information on sociodemographic variables (age, assigned sex at birth, and racial, ethnic, sexual, and gender identities) and description of these questions. Raters also searched the entire article for words (e.g., discrimination, disparities, inequities, race, sexual orientation, generalizability, representation) and topics (e.g., depression in Black adults) related to examination of health inequities and inclusion of lack of generalizability or representation as a limitation. The infrequent discrepancies (e.g., raters disagreed 2.4% [3/125] of the time about whether the article examined health inequity, and 19% [24/125] of the time about representation/generalizability being mentioned) were resolved by a third rater.

Raters coded the type (e.g., genetic, neuroimaging) and the location (e.g., United States, United Kingdom) of the study. Data are described separately for genetic and nongenetic studies, given previous research showing that genetic studies almost exclusively include participants of European ancestry (>78%) (11) and separately for studies conducted within and outside of the United States to allow for comparison with 2019 U.S. Census data population estimates (12). Studies conducted within the United States were assessed for over- or underrepresentation of participants with specific sociodemographic characteristics by comparing to local U.S. Census estimates for articles where a specific location or region was provided (N=19 for race/ethnicity; N=25 for sex/gender). Articles for which a location could not be determined were tested against national U.S. Census estimates. Z-tests of the difference of proportions using the prop.test function in R were run to assess whether the probability of the numbers of female individuals/women and people from specific racial and ethnic groups were higher or lower in the published articles compared to the U.S. Census data (13, 14). The z-tests provide a 95% confidence interval and chi-square with a p value.

Results

Representation Across Racial Groups

While 90% (113/125) of studies reported the age of their participants, only 43% (54/125) provided information about race or ethnicity. Across these 54 studies (Table 1), 94.6% of participants were described as White or Caucasian. When only nongenetic studies were examined (N=39), 71.4% of participants were described as White or Caucasian; the second most prevalent group was described as “other” or “nonwhite” (10.6%), followed by African American, Black, or African (7.6%), and Hispanic or Latino (5.3%). The remaining racial groups comprised <1% of the participants. Table 2 shows the summary results of over- and underrepresentation across race and ethnicity when each study is compared to local or national U.S. Census estimates. Asian/Asian American and Hispanic/Latino participants were underrepresented in 78% of articles. Biracial/multiracial participants were underrepresented in 49% of articles. Black and White participants were underrepresented in 29% and 37% of the articles, respectively.

Representation Across Sex/Gender

One hundred and five studies (84%) reported on the sex/gender of their participants. Sixty-four studies only presented the sample size for one sex/gender. In these instances, we imputed the other number by assuming the authors restricted the options to binary (male/female, man/woman) responses. Following imputation, 50.8% of the total sample of participants was described as female/women/girls. While this distribution was similar for nongenetic studies, in U.S. genetic samples, only 24.7% are described as female/women/girls (Table 1). Table 2 shows that female participants/women/girls were underrepresented in 47% of articles and overrepresented in 25% of articles.

Inaccurate or Missing Information

One study described the question used to ask participants about their racial or ethnic identity and 43% (26/60) of articles used nonexistent racial categories such as “nonwhite,” “other,” and/or “mixed.” Additionally, no study provided a description of the question asked to establish sex or gender identity, precluding our ability to know if the articles were referring to participants’ assigned sex at birth or gender identity (given this lack of clarity, we used “sex/gender” in the present study). One study presented the sexual identity of participants. Only 25% (31/125) of articles identified lack of representation or generalizability concerns based on sample composition as a study limitation.

Prevalence of Health Equity Research

Two of the 125 studies examined inequities/disparities or psychiatric illness within minoritized populations.

Discussion

We examined representation across a recent 2-year period (2019–2020) in a high-impact psychiatry journal. We found that minoritized racial and ethnic groups, especially Asian American and Latino/Hispanic participants, and female participants/women were underrepresented in these articles. Our results also showed that descriptions of sociodemographic characteristics were frequently missing, use of inaccurate labels was prevalent, and acknowledgment of lack of representation as a limitation was uncommon.

To increase diversity within research samples, investigators can connect with community-engaged researchers at their universities and read the existing literature (e.g., 15) to learn about best practices for partnering with minoritized populations. Researchers should also review the sociodemographic characteristics of participants in an ongoing fashion, like other enrollment criteria, and adjust as needed to ensure adequate representation based on local population estimates. Additionally, it is imperative to collect genetic data from ancestrally diverse populations and interpret these data in culturally informed ways. There are published recommendations to guide researchers (16).

Our results also found missing or inaccurate language about race and ethnicity in the articles. First, less than half of the articles provided the racial composition of the sample, limiting our ability to even evaluate representation. Further, within the studies that provided descriptors for racial identities, minoritized racial and ethnic groups may be being homogenized into an “other/nonwhite” category, as this group was overrepresented compared to U.S. Census data in 49% of the articles (and underrepresented in 5%). This is an example of “white centering” in research (17, 18), which assumes similarity of all minoritized racial and ethnic groups into a single group in relation to White participants. Additionally, “Caucasian,” a label with a racist history (19), was used as a racial category in 45% of articles.

Despite the Institute of Medicine stating that assessing and reporting sexual and gender identities is important (20), only one article reported the sexual identity of participants, no studies clarified if they were reporting on assigned sex or gender identity, and descriptions of samples defaulted to an assumed gender binary. Researchers should review how they are assessing sociodemographic characteristics to ensure that accurate and affirming questions are being used to collect information on sexual and gender identities (as well as racial and ethnic identities). For the 1.2 million people in the United States who have a minoritized gender identity (21), binary response options (e.g., only man or woman) are often inaccurate and hurtful. The PhenXToolkit (22) provides examples of how to assess sociodemographic characteristics.

Despite lack of representation or missing data related to sociodemographic characteristics, there is significant value in utilizing existing data for secondary data analyses. However, when interpreting results and writing manuscripts, directly acknowledging these limitations and framing conclusions accordingly is crucial. This will spur further innovative research and increase rigor and reproducibility. Authors are also called upon to be thoughtful about conducting and interpreting subgroup analyses. It is imperative to place results within a sociohistorical context that recognizes structural inequities experienced by minoritized individuals and communities and to define race, ethnicity, and gender as social constructs. Additionally, if scientists more clearly and accurately describe the sample characteristics and questions used, meta-analyses can be utilized to examine determinants of health inequities by pooling across samples that were not initially powered to examine processes within and across subpopulations.

In addition to individual investigators, scientific journals and funding agencies (e.g., the National Institutes of Health [NIH]) can foster investment in recruiting representative samples and describing sociodemographic characteristics. For example, the Journal states on its website, “The methods section should provide a comprehensive description of the nature of the study group.” However, in the articles reviewed here, this guideline was largely not followed, indicating that expanded guidelines and provision of concrete requirements for characterizing sample sociodemographic characteristics (e.g., sample description must include racial and ethnic identities, gender identities, and age) may be beneficial. Resources can also be added to journal and funding agency websites. For example, the American Psychological Association has published guidelines that are available (23, 24) for reducing bias in scientific language, and there are published recommendations on how to report racial and ethnic identities in medical and scientific journals (25). Journals and funding agencies could also include specialty reviewers to provide critiques focused on inaccurate language, interpretation of results, and representation, ensuring that conclusions are framed within this context. This will also educate scientists and improve future research.

The lack of research (<2% of articles) on health inequities/disparities in 2019 and 2020 in a top-tier psychiatry journal is another urgent issue that can be remedied by journals and funding agencies. Our findings dovetail with a recent review of over 26,000 articles published in top-tier psychology journals that found that around 5% of the articles reviewed focused on outcomes for minoritized racial groups (26). Individuals from minoritized identities are more likely to conduct health equity research, which is subject to publication bias (27), subsequently contributing to NIH funding inequities (28), further hampering the careers of scientists already facing discrimination (28). As NIH pushes to fund more health equity science and scientists call for NIH to address long-standing funding inequities (29), journals could set benchmarks (and track progress over time) for inclusion of health equity research.

While the present study utilized a large sample of 125 recently published articles, several limitations are worth noting. First, we focused on articles only from 2019 and 2020 and one journal. We chose these years because they include recent research that was likely accepted for publication prior to the murder of George Floyd, which occurred in May 2020. We examined one high-impact journal to allow for in-depth review of each article for multiple sociodemographic characteristics. This approach also allows our findings to serve as a “baseline” for the Journal to track progress over time and provides an example that other journals can follow, as these issues are not specific to this journal. We also did not know the specific location of all U.S. studies, which diminished the specificity of the representation results for these studies.

Despite these limitations, the present study significantly advances understanding about representation and scientific language in psychiatric research. Further research that integrates the identities of authors, reviewers, and editorial board members and examines change over time in comparison to this “baseline” is needed. We hope that these results and conclusions encourage action and further discussion across levels of the scientific community (e.g., researchers, community members, institutions, journals, funding agencies).