0
Articles   |    
DSM-5 Field Trials in the United States and Canada, Part III: Development and Reliability Testing of a Cross-Cutting Symptom Assessment for DSM-5
William E. Narrow, M.D., M.P.H.; Diana E. Clarke, Ph.D., M.Sc.; S. Janet Kuramoto, Ph.D., M.H.S.; Helena C. Kraemer, Ph.D.; David J. Kupfer, M.D.; Lisa Greiner, M.S.S.A.; Darrel A. Regier, M.D., M.P.H.
Am J Psychiatry 2013;170:71-82. 10.1176/appi.ajp.2012.12071000
View Author and Article Information

All authors report no financial relationships with commercial interests.

This study was funded by the American Psychiatric Association.

From the American Psychiatric Association, Division of Research and American Psychiatric Institute for Research and Education, Arlington, Va.; the Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Md.; the Stanford University School of Medicine, Palo Alto, Calif.; and the University of Pittsburgh Medical Center, Pittsburgh, Pa.

Presented in part at the 165th annual meeting of the American Psychiatric Association, Philadelphia, May 5–9, 2012, and the New Clinical Drug Evaluation Unit Annual Meeting, Phoenix, June 2012.

Address correspondence to Dr. Narrow (wnarrow@psych.org).

Copyright © 2013 by the American Psychiatric Association

Received July 30, 2012; Revised August 31, 2012; Accepted September 04, 2012.

Abstract

Objective  The authors sought to document, in adult and pediatric patient populations, the development, descriptive statistics, and test-retest reliability of cross-cutting symptom measures proposed for inclusion in DSM-5.

Method  Data were collected as part of the multisite DSM-5 Field Trials in large academic settings. There were seven sites focusing on adult patients and four sites focusing on child and adolescent patients. Cross-cutting symptom measures were self-completed by the patient or an informant before the test and the retest interviews, which were conducted from 4 hours to 2 weeks apart. Clinician-report measures were completed during or after the clinical diagnostic interviews. Informants included adult patients, child patients age 11 and older, parents of all child patients age 6 and older, and legal guardians for adult patients unable to self-complete the measures. Study patients were sampled in a stratified design, and sampling weights were used in data analyses. The mean scores and standard deviations were computed and pooled across adult and child sites. Reliabilities were reported as pooled intraclass correlation coefficients (ICCs) with 95% confidence intervals.

Results  In adults, test-retest reliabilities of the cross-cutting symptom items generally were good to excellent. At the child and adolescent sites, parents were also reliable reporters of their children’s symptoms, with few exceptions. Reliabilities were not as uniformly good for child respondents, and ICCs for several items fell into the questionable range in this age group. Clinicians rated psychosis with good reliability in adult patients but were less reliable in assessing clinical domains related to psychosis in children and to suicide in all age groups.

Conclusions  These results show promising test-retest reliability results for this group of assessments, many of which are newly developed or have not been previously tested in psychiatric populations.

Abstract Teaser
Figures in this Article

Your Session has timed out. Please sign back in to continue.
Sign In Your Session has timed out. Please sign back in to continue.
Sign In to Access Full Content
 
Username
Password
Sign in via Athens (What is this?)
Athens is a service for single sign-on which enables access to all of an institution's subscriptions on- or off-site.
Not a subscriber?

Subscribe Now/Learn More

PsychiatryOnline subscription options offer access to the DSM-IV-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.

Need more help? PsychiatryOnline Customer Service may be reached by emailing PsychiatryOnline@psych.org or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).

Anchor for Jump
TABLE 1.DSM-5 Dimensional Cross-Cutting Symptom Assessment for Adult Patients
Table Footer Note

a “During the past TWO (2) WEEKS, how much have you been bothered by the following problems….” Questions assessing items of anger, mania, anxiety (items 2 and 3), somatic distress, sleep, memory, dissociation, and personality were developed by DSM-5 work groups or study groups. Depression items taken from PHQ-2 (adapted) (16); anxiety item 1 taken from GAD-7 (adapted) (17); suicide item taken from P4 Suicide Screener (18); psychosis items from MINI (adapted) (19); repetitive thoughts/behavior items from Florida Obsessive-Compulsive Inventory (FOCI) (adapted) (20); substance use items from NIDA Quick Screen V1.0 (adapted) (21).

Table Footer Note

b PROMIS SF: Patient Reported Outcomes Measurement Information System Short Form, v1.0 (22, 23); PHQ-SSS: Patient Health Questionnaire Somatic Symptom Short-Form (unpublished 2010 instrument by K. Kroenke, adapted from the PHQ-15 [25]); FOCI: Florida Obsessive Compulsive Inventory (adapted) (20); NIDA-modified ASSIST: National Institute on Drug Abuse-Modified Alcohol, Smoking and Substance Involvement Screening Test (adapted) (21).

Anchor for Jump
TABLE 2.DSM-5 Dimensional Cross-Cutting Symptom Assessment for Parents
Table Footer Note

a “During the past TWO (2) WEEKS, how much (or how often) has your child…”; for the substance use and suicide items, the question began “In the last 2 weeks has he/she…” Questions assessing items of anger, mania, anxiety, somatic distress, psychosis, sleep, repetitive thoughts/behaviors, and attention were developed by DSM-5 work groups or study groups. Depression items taken from PHQ-2 (adapted) (16); irritability item taken from Affective Reactivity Index (adapted) (26); substance use items taken from NIDA Quick Screen V1.0 (adapted) (21); suicide items taken from Suicide Rating Scale for Teens (D. Shaffer and M. Gallagher, unpublished 2010 scale).

Table Footer Note

b PROMIS: Patient Reported Outcomes Measurement Information System Parent Proxy Bank v1.0 (22, 23); PROMIS SF: Patient Reported Outcomes Measurement Information System Short Form, v1.0 (22, 23); PHQ-SSS: Patient Health Questionnaire Somatic Symptom Short-Form (unpublished 2010 instrument by K. Kroenke, adapted from the PHQ-15 [25]); SNAP-IV: Swanson, Nolan, and Pelham Scale, version IV (adapted) (27); NIDA-modified ASSIST: National Institute on Drug Abuse-Modified Alcohol, Smoking and Substance Involvement Screening Test (adapted) (21).

Anchor for Jump
TABLE 3.DSM-5 Dimensional Cross-Cutting Symptom Assessment for Children
Table Footer Note

a “During the past TWO (2) WEEKS, how much (or how often) have you…”; for the substance use and suicide items, the question began “In the last 2 weeks have you…” Questions assessing items of anger, mania, anxiety, somatic distress, psychosis, sleep, attention and repetitive thoughts/behaviors were developed by DSM-5 work groups or study groups. Depression items taken from PHQ-2 (adapted) (16); irritability item taken from Affective Reactivity Index (adapted) (26); substance use items taken from NIDA Quick Screen V1.0 (adapted) (21); suicide items taken from Suicide Rating Scale for Teens (D. Shaffer and M. Gallagher, unpublished 2010 scale).

Table Footer Note

b PROMIS: Patient Reported Outcomes Measurement Information System Pediatric Bank v1.0 (22, 23); PROMIS SF: Patient-Reported Outcomes Measurement Information System Short Form, v1.0 (22, 23); PHQ-SSS: Patient Health Questionnaire Somatic Symptom Short-Form (unpublished 2010 instrument by K. Kroenke, adapted from the PHQ-15 [25]); FOCI: Florida Obsessive Compulsive Inventory (adapted) (20); NIDA-modified ASSIST: National Institute on Drug Abuse-Modified Alcohol, Smoking and Substance Involvement Screening Test (adapted) (21).

Anchor for Jump
TABLE 4.Test-Retest Reliability of Adult Self-Rated DSM-5 Cross-Cutting Symptom Measures, Level 1
Table Footer Note

a Pooled intraclass correlation coefficient (ICC) for a stratified sample with 95% confidence interval. For all items except psychosis 1, repetitive thoughts, and personality 1 there was a nonoverlapping 95% confidence interval for at least one of the seven adult sites, so the pooled ICC must be interpreted with caution.

Anchor for Jump
TABLE 5.Test-Retest Reliability of Parent- and Child-Rated DSM-5 Cross-Cutting Symptom Measures, Level 1
Table Footer Note

a Pooled intraclass correlation coefficient (ICC) for a stratified sample with 95% confidence interval.

Table Footer Note

b There is a nonoverlapping 95% confidence interval for at least one of the four child sites, so the pooled ICC must be interpreted with caution.

Table Footer Note

c Reliability estimates were not included in the pooled estimates for these items at the following sites because standard errors for the ICC estimates were greater than 0.1: Baystate (child respondents): mania 1, substance use 1, substance use 3, substance use 4, and suicide 1; Colorado (child respondents): substance use 1, substance use 3, substance use 4, and suicide 1; Columbia (parent respondents, child <11): substance use 3; (child respondents): psychosis 1, attention, substance use 1, substance use 2, substance use 3, substance use 4, suicide 1, and suicide 2; Stanford (parent respondents, child <11): substance use 3; (child respondents): substance use 3 and substance use 4.

Table Footer Note

d Reliability estimates could not be computed for these items at the following sites because all responses were identical within the site: Baystate (parent respondents, child <11): substance use 1, substance use 2, and substance use 3; (child respondents): substance use 2 and suicide 2; Colorado (parent respondents, child <11): substance use 1, substance use 2, and substance use 3; Columbia (parent respondents, child <11): substance use 1, substance use 2, and substance use 4; Stanford (parent respondents, child <11): substance use 1, substance use 2, substance use 4, and suicide 2.

Table Footer Note

e Estimated using intraclass kappa for dichotomous variables since the item responses were Yes/No for child respondents.

Anchor for Jump
TABLE 6.Test-Retest Reliability of Adult Self-Rated DSM-5 Cross-Cutting Symptom Measures, Levels 1 and 2 Combined
Table Footer Note

a Pooled intraclass correlation coefficients (ICCs) for a stratified sample with 95% confidence intervals. Pooled ICCs for all items need to be interpreted with caution because the confidence intervals for at least one site did not overlap with the others.

Anchor for Jump
TABLE 7.Test-Retest Reliability of Parent- and Child-Rated DSM-5 Cross-Cutting Symptom Measures, Levels 1 and 2 Combined
Table Footer Note

a Pooled intraclass correlation coefficients for a stratified sample with 95% confidence intervals; n/a indicates that the item was not assessed in that patient group in the field trials.

Table Footer Note

b There is a nonoverlapping 95% CI for at least one of the four child sites so the pooled ICC needs to be interpreted with caution.

Table Footer Note

c Reliability estimates could not be computed for these items at the following sites because all responses were identical within the site: Baystate  (parent respondents, child <11): substance use 1, substance use 2, and substance use 3; Colorado  (parent respondents, child <11): substance use 1, substance use 2, and substance use 3; Columbia : (parent respondents, child <11): substance use 1, substance use 2, and substance use 4; (parent respondents, child 11+): substance use 2, substance use 4, and suicide ideation; (child respondents): substance use 4; Stanford : (parent respondents, child <11): substance use 1, substance use 2, substance use 4, and suicide attempt; (parent respondents, child 11+): substance use 3.

Table Footer Note

d Reliability estimates were not included in the pooled estimates for these items at the following sites because standard errors for the ICC estimates were greater than 0.1: Baystate : (parent respondents, child 11+): substance use 3 and substance use 4; Columbia : (parent respondents, child <11): substance use 3; (parent respondents, child 11+): substance use 1 and substance use 3; Stanford : (parent respondents, child <11): substance use 3.

Anchor for Jump
TABLE 8.Test-Retest Reliability of the Clinician-Rated DSM-5 Cross-Cutting Symptom Measures
Table Footer Note

a Pooled intraclass correlation coefficients for a stratified sample with 95% confidence intervals.

Table Footer Note

b The 95% CI for the intraclass correlation coefficients for at least one site did not overlap with the others, hence the pooled ICC needs to be interpreted with caution.

Table Footer Note

c Individual site ICC estimates with SE greater than 0.1. (i.e., length of 95% CI greater than 0.5) for a dimensional measure were not included in the pooled estimates. These included psychosis ratings at the Stanford, Columbia, and Colorado sites.

+

References

Clark  LA;  Watson  D;  Reynolds  S:  Diagnosis and classification of psychopathology: challenges to the current system and future directions.  Annu Rev Psychol   1995; 46:121–153
[CrossRef] | [PubMed]
 
Helzer  JE;  Kraemer  HC;  Krueger  RF;  Wittchen  H-U;  Sirovatka  PJ;  Regier  DA:  Dimensional Approaches in Diagnostic Classification: Refining the Research Agenda for DSM-V .  Arlington, Va,  American Psychiatric Association,  2008
 
Hyman  SE:  Neuroscience, genetics, and the future of psychiatric diagnosis.  Psychopathology   2002; 35:139–144
[CrossRef] | [PubMed]
 
Morris  SE;  Cuthbert  BN:  Research Domain Criteria: cognitive systems, neural circuits, and dimensions of behavior.  Dialogues Clin Neurosci   2012; 14:29–37
[PubMed]
 
Löwe  B;  Spitzer  RL;  Williams  JB;  Mussell  M;  Schellberg  D;  Kroenke  K:  Depression, anxiety and somatization in primary care: syndrome overlap and functional impairment.  Gen Hosp Psychiatry   2008; 30:191–199
[CrossRef] | [PubMed]
 
Fava  M;  Rush  AJ;  Alpert  JE;  Balasubramani  GK;  Wisniewski  SR;  Carmin  CN;  Biggs  MM;  Zisook  S;  Leuchter  A;  Howland  R;  Warden  D;  Trivedi  MH:  Difference in treatment outcome in outpatients with anxious versus nonanxious depression: a STAR*D report.  Am J Psychiatry   2008; 165:342–351
[CrossRef] | [PubMed]
 
Conley  RR;  Ascher-Svanum  H;  Zhu  B;  Faries  DE;  Kinon  BJ:  The burden of depressive symptoms in the long-term treatment of patients with schizophrenia.  Schizophr Res   2007; 90:186–197
[CrossRef] | [PubMed]
 
Szelenberger  W;  Soldatos  C:  Sleep disorders in psychiatric practice.  World Psychiatry   2005; 4:186–190
[PubMed]
 
Mojtabai  R;  Olfson  M:  National trends in psychotropic medication polypharmacy in office-based psychiatry.  Arch Gen Psychiatry   2010; 67:26–36
[CrossRef] | [PubMed]
 
Berlin  RM;  Litovitz  GL;  Diaz  MA;  Ahmed  SW:  Sleep disorders on a psychiatric consultation service.  Am J Psychiatry   1984; 141:582–584
[PubMed]
 
Wilk  JE;  West  JC;  Narrow  WE;  Marcus  S;  Rubio-Stipec  M;  Rae  DS;  Pincus  HA;  Regier  DA:  Comorbidity patterns in routine psychiatric practice: is there evidence of underdetection and underdiagnosis? Compr Psychiatry   2006; 47:258–264
[CrossRef] | [PubMed]
 
Trivedi  MH:  Tools and strategies for ongoing assessment of depression: a measurement-based approach to remission.  J Clin Psychiatry   2009; 70(suppl 6):26–31
[CrossRef] | [PubMed]
 
Valenstein  M;  Adler  DA;  Berlant  J;  Dixon  LB;  Dulit  RA;  Goldman  B;  Hackman  A;  Oslin  DW;  Siris  SG;  Sonis  WA:  Implementing standardized assessments in clinical care: now’s the time.  Psychiatr Serv   2009; 60:1372–1375
[CrossRef] | [PubMed]
 
Clarke  DE;  Narrow  WE;  Regier  DA;  Kuramoto  SJ;  Kupfer  DJ;  Kuhl  EA;  Greiner  L;  Kraemer  HC:  DSM-5 Field Trials in the United States and Canada, part I: study design, sampling strategy, implementation, and analytic approaches.  Am J Psychiatry   2013; 170:43–58
 
Harris  PA;  Taylor  R;  Thielke  R;  Payne  J;  Gonzalez  N;  Conde  JG:  Research Electronic Data Capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support.  J Biomed Inform   2009; 42:377–381
[CrossRef] | [PubMed]
 
Kroenke  K;  Spitzer  RL;  Williams  JB:  The Patient Health Questionnaire-2: validity of a two-item depression screener.  Med Care   2003; 41:1284–1292
[CrossRef] | [PubMed]
 
Spitzer  RL;  Kroenke  K;  Williams  JBW;  Löwe  B:  A brief measure for assessing generalized anxiety disorder: the GAD-7.  Arch Intern Med   2006; 166:1092–1097
[CrossRef] | [PubMed]
 
Dube  P;  Kurt  K;  Bair  MJ;  Theobald  D;  Williams  LS:  The P4 screener: evaluation of a brief measure for assessing potential suicide risk in 2 randomized effectiveness trials of primary care and oncology patients.  Prim Care Companion J Clin Psychiatry   2010; 12:e1–e8
 
Sheehan  DV;  Lecrubier  Y;  Sheehan  KH;  Amorim  P;  Janavs  J;  Weiller  E;  Hergueta  T;  Baker  R;  Dunbar  GC:  The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10.  J Clin Psychiatry   1998; 59(suppl 20):22–33
[PubMed]
 
Storch  EA;  Kaufman  DA;  Bagner  D;  Merlo  LJ;  Shapira  NA;  Geffken  GR;  Murphy  TK;  Goodman  WK:  Florida Obsessive-Compulsive Inventory: development, reliability, and validity.  J Clin Psychol   2007; 63:851–859
[CrossRef] | [PubMed]
 
National Institute on Drug Abuse: NIDA Quick Screen V 1.0. http://www.nida.nih.gov/nidamed/screening/nmassist.pdf, accessed 7/20/12
 
Cella  D;  Yount  S;  Rothrock  N;  Gershon  R;  Cook  K;  Reeve  B;  Ader  D;  Fries  JF;  Bruce  B;  Rose  M; PROMIS Cooperative Group:  The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years.  Med Care   2007; 45(Suppl 1):S3–S11
[CrossRef] | [PubMed]
 
Cella D, Gershon R, Bass M, Rothrock N: Assessment Center, http://www.assessmentcenter.net, accessed 7/20/12
 
Altman  EG;  Hedeker  D;  Peterson  JL;  Davis  JM:  The Altman Self-Rating Mania Scale.  Biol Psychiatry   1997; 42:948–955
[CrossRef] | [PubMed]
 
Kroenke  K;  Spitzer  RL;  Williams  JBW:  The PHQ-15: validity of a new measure for evaluating the severity of somatic symptoms.  Psychosom Med   2002; 64:258–266
[PubMed]
 
Stringaris  A;  Goodman  R;  Ferdinando  S;  Razdan  V;  Muhrer  E;  Leibenluft  E;  Brotman  MA: The Affective Reactivity Index: a concise irritability scale for clinical and research settings. J Child Psychol Psychiatr (Epub ahead of print, May 10, 2012)
 
Swanson  JM:  School-Based Assessments and Interventions for ADD Students .  Irvine, Calif,  KC Publishing,  1992
 
Shrout  PE;  Fleiss  JL:  Intraclass correlations: uses in assessing rater reliability.  Psychol Bull   1979; 86:420–428
[CrossRef] | [PubMed]
 
Kraemer  HC;  Kupfer  DJ;  Clarke  DE;  Narrow  WE;  Regier  DA:  DSM-5: how reliable is reliable enough? Am J Psychiatry   2012; 169:13–15
[CrossRef] | [PubMed]
 
Ustün  TB;  Chatterji  S;  Kostanjsek  N;  Rehm  J;  Kennedy  C;  Epping-Jordan  J;  Saxena  S;  von Korff  M;  Pull  C; WHO/NIH Joint Project:  Developing the World Health Organization Disability Assessment Schedule 2.0.  Bull World Health Organ   2010; 88:815–823
[CrossRef] | [PubMed]
 
Krueger  RF;  Derringer  J;  Markon  KE;  Watson  D;  Skodol  AE:  Initial construction of a maladaptive personality trait model and inventory for DSM-5.  Psychol Med   2012; 42:1879–1890
[CrossRef] | [PubMed]
 
Dawson  NV:  Physician judgment in clinical settings: methodological influences and cognitive performance.  Clin Chem   1993; 39:1468–1478, discussion 1478–1480
[PubMed]
 
Kendler  KS:  Toward a philosophical structure for psychiatry.  Am J Psychiatry   2005; 162:433–440
[CrossRef] | [PubMed]
 
References Container
+
+

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.
Submit a Comments
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of APA editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe



Related Content
Articles
Books
Manual of Clinical Psychopharmacology, 7th Edition > Chapter 2.  >
Manual of Clinical Psychopharmacology, 7th Edition > Chapter 1.  >
Manual of Clinical Psychopharmacology, 7th Edition > Chapter 8.  >
Manual of Clinical Psychopharmacology, 7th Edition > Chapter 12.  >
Manual of Clinical Psychopharmacology, 7th Edition > Chapter 12.  >
Topic Collections
Psychiatric News
APA Guidelines
PubMed Articles