Diagnosis of Comorbid Psychiatric Disorders in Substance Users Assessed With the Psychiatric Research Interview for Substance and Mental Disorders for DSM-IV

Psychiatric and substance use disorders co-occur frequently in the clinical (1) and general population (2 – 4) . Understanding the relationship between substance use disorders and psychiatric disorders is necessary to clarify the etiology of the disorders and to improve treatment, but diagnostic issues have hindered this process. The diagnosis of psychiatric disorders among substance abusers is complicated by the resemblance of intoxication and withdrawal effects to the symptoms of psychiatric disorders. The challenge has been to design measures to differentiate three conditions: 1) expected intoxication and withdrawal symptoms, 2) psychiatric disorders occurring during periods of active substance use, and 3) psychiatric disorders that are clearly independent from substance use because they are temporally distinct from periods of substance use. Before publication of DSM-IV, specific criteria for making these differentiations in substance abusers did not exist, resulting in reliability (5 – 8) and validity (9 – 12) problems and low levels of agreement between study groups, even when the same measures were used (13) .

DSM-IV substantially clarified the guidelines for differentiating independent and substance-induced disorders. Among individuals with histories of substance use disorders, primary (independent) DSM-IV psychiatric disorders are diagnosed if the full psychiatric syndrome is established before substance use or if it occurs or persists more than 4 weeks after the cessation of acute intoxication or withdrawal. A psychiatric episode occurring when substance use was insufficient to produce persistent intoxication or withdrawal would be diagnosed as primary (independent) according to DSM-IV. In contrast, substance-induced psychiatric disorders are defined as those that occur during periods of heavy substance use (or remitting shortly thereafter) and that have symptoms that exceed the expected effects of intoxication or withdrawal listed in DSM-IV. DSM-IV was published in 1994. To our knowledge, no study has yet addressed the reliability of the DSM-IV system of differentiating between intoxication/withdrawal symptoms, disorders whose symptoms exceed intoxication/withdrawal effects, and psychiatric disorders that are temporally independent from periods of substance use.

To provide a diagnostic instrument that was reliable and valid for assessment of psychiatric disorders in substance abusers, the Psychiatric Research Interview for Substance and Mental Disorders (PRISM) (14) was developed. The pre-DSM-IV PRISM included standardized guidelines and probes to differentiate “organic” and “nonorganic” syndromes and was administered by experienced clinical interviewers. A test-retest reliability study (14) involving 172 dual-diagnosis or substance abuse patients showed good to excellent reliability for many diagnoses, including substance use disorders, primary affective disorders, eating disorders, some anxiety disorders, and psychotic symptoms (analogous to the Structured Clinical Interview for DSM-IV-TR Non-Patient [SCID-IV-TR-NP] category). Publication of DSM-IV and investigator feedback indicated the need to incorporate the new DSM-IV criteria (including the criteria for substance-induced disorders), to shorten and simplify the interview, and to add assessment of specific psychotic disorders. This led to development of the PRISM-IV (unpublished 2003 manuscript of Hasin et al.). The development process integrated refinement and editing of probes and structure with piloting and small test-retest studies among substance abuse patients. The results of this process were then subjected to a new, larger test-retest reliability study, reported here.

Method

Procedures

Subjects were recruited from New York City treatment facilities. All subjects were engaged in ongoing outpatient treatment or were inpatients hospitalized for as long as 1 month before study participation. Clinic staff screened patients for eligibility and referred eligible patients to a research coordinator who explained the study and obtained written informed consent. Eligibility criteria included age 18–65 years; use of alcohol, heroin, or cocaine in the prior 30 days (or 30 days prior to hospitalization, among inpatients); and sufficiently stable medical and psychiatric status to participate, including resolution of any acute withdrawal symptoms. The retest interviewer was blind to the results of the first interview. Mean time between interviews was 5.2 days (SD=5.1, range=0–36); 95% of retests occurred within 14 days. Subject fees were $35–$50 per interview, depending on the site. The Institutional Review Board of New York State Psychiatric Institute approved all procedures.

Subjects

Of 342 eligible patients, 285 (83.33%) completed a test and a retest interview. Of these 285 patients, 54.04% (N=154) were male. About half (51.93%, N=148) were white, 31.58% (N=90) were African American, 12.28% (N=35) were Hispanic, and 2.21% (N=6) were of other ethnicities. The subjects’ mean age was 36.28 years (SD=8.77), almost one-half (47.02%, N=134) were unemployed, 74.74% (N=213) had at least 12 years of education, and 13.70% (N=39) were married or cohabiting. Treatment settings included outpatient substance abuse/dual-diagnosis (43.2%, N=123), outpatient mental health (35.4%, N=101), and inpatient rehabilitation or dual-diagnosis (21.4%, N=61) settings. When we compared the subjects who completed only one interview (N=17) to those who completed both interviews (N=285), no significant differences were found for age, race, gender, employment, marital status, education, or a lifetime diagnosis of substance abuse/dependence or major depressive disorder (primary or substance-induced). No information was available for those who did not participate in at least one interview, but the good response rate (83%) precluded substantial bias.

Measure

The PRISM-IV is a semistructured interview. Initial PRISM-IV probes are structured and asked as written; follow-up probes are also provided to ascertain more information. Unlike fully structured interviews where rereading the written probe is the only authorized clarification, the PRISM-IV also allows some unstructured interviewer follow-up probing. Interviewers are trained to use the follow-up probes provided or to add their own probes when necessary, using guidelines in the interview.

The PRISM-IV covers the following diagnoses for current and lifetime time frames: abuse and dependence (by substance), major depressive disorder, dysthymia, cyclothymia, mania, hypomania, schizophrenia, schizophreniform disorder, schizoaffective disorder, mood disorder with psychotic features, delusional disorder, brief psychotic disorder, psychotic disorder due to a medical condition, psychotic disorder not otherwise specified, specific phobia, social phobia, panic disorder, agoraphobia, generalized anxiety disorder, obsessive-compulsive disorder, posttraumatic stress disorder (PTSD), anorexia, bulimia, antisocial personality disorder, and borderline personality disorder. Psychiatric syndromes due to a medical condition are noted. Diagnostic modules can be selected to suit specific research needs. Substance use disorders that are unimportant to the research question can be omitted.

The PRISM-IV also provides continuous measures, including age at onset of disorder, and severity measures based on symptom counts for some disorders, such as major depressive disorder and substance dependence. Onset of substance dependence is established on the basis of the time at which the subject first met the full criteria, i.e., experienced three or more symptoms within the same year. If onset occurred in the last 12 months, onset is identified as having occurred in the month when three or more symptoms first co-occurred. Onset of other psychiatric disorders is the age when the full criteria were initially met (e.g., onset of major depressive disorder is the age when the subject had five or more symptoms and impairment or distress). Each module begins with low-threshold screening questions and skip-outs.

In the PRISM-IV, the sections on substance use precede other diagnostic sections so that the interviewer ascertains the substance use history before assessing primary and substance-induced psychiatric episodes. Periods of heavy use or chronic intoxication (4 or more days/week for a month, or binge use) and extended periods of abstinence are identified to assist later comorbidity assessment. All four abuse and seven dependence criteria are rated for each substance used to avoid underdiagnosing dependence (15) .

Primary disorders are diagnosed when symptoms persist at least 4 weeks in the absence of heavy substance use or when symptoms precede the onset of heavy use. For a PRISM-IV substance-induced diagnosis, 1) a primary episode must be ruled out and 2) the symptoms must exceed the expected effects of intoxication or withdrawal. The PRISM-IV provides guidelines that specify these DSM-IV intoxication and withdrawal effects. Because specific diagnostic criteria are necessary to obtain a reliable diagnosis (16) , the PRISM-IV also addresses the lack of duration and threshold criteria for substance-induced disorders in DSM-IV by assigning substance-induced diagnoses only when an episode meets the duration and symptom requirements for a corresponding primary DSM-IV diagnoses. For example, substance-induced major depressive disorder must have a duration of at least 2 weeks, and five of the nine depression symptoms, including depressed mood or anhedonia, must be present.

Depression is common among substance abusers (17 , 18) , but diagnosing major depressive disorder in substance abusers has been problematic (19 – 21) . Therefore, when primary major depressive disorder has been ruled out by the history, the PRISM-IV depression module addresses substance-induced major depressive disorder, giving special attention to depressive symptoms also listed as DSM-IV intoxication/withdrawal effects. To assess these symptoms, the subject’s own substance-using, nondepressed experience is used as a reference period (commonly a period of substance use preceding the onset of depressed mood). Symptoms that do not change in presence or intensity between nondepressed and depressed periods of consistently heavy substance use (e.g., insomnia in a patient with alcoholism) are not counted as symptoms of substance-induced major depressive disorder. However, during periods of heavy substance use, symptoms that begin or substantially increase in intensity with the onset of depressed mood are counted toward a diagnosis of substance-induced major depressive disorder.

Partial remission is available as a time-frame option for some PRISM-IV categories. Relative to current disorders, the PRISM-IV default for partial remission is to consider it as past, although this definition can be adjusted to the needs of individual studies. Specifically, patients with past alcohol/drug dependence and one or two current symptoms are considered to be in partial remission. A major depressive episode that persists at a subthreshold level (i.e., fewer than five symptoms) for at least 2 months is considered to be in partial remission. Patients with psychotic disorders who experienced an active phase are assessed for current residual symptoms. If functioning remains markedly below the functioning level prior to onset of psychosis or if the subject continues to be dependent on social services, the residual phase is coded as “current.” For mood disorder with psychotic features, (nonbizarre) delusional disorder, and brief psychotic episode, there is no distinction between active and residual symptoms. If any psychotic symptoms are present within the last 2 months, the disorder is considered “current”; otherwise it is past. For anxiety and eating disorders, “current” is defined as within the last 2 months, and partial remission is not assessed. Antisocial personality disorder and borderline personality disorder are considered current if full criteria were ever met and any symptoms are present in the last 12 months. The diagnostic programs include three time frames: past, within the past 12 months, and within the past 2 months.

Interviewers and Supervision

Interviewers had master’s degrees in psychology and at least 1 year of clinical experience. Training included reading the PRISM-IV training manual, didactic sessions and role playing, and observation of interviewers. Weekly staff meetings were held to discuss diagnostic questions and prevent interviewer drift. Interviews were audiotaped if patients consented, and 81 randomly selected tapes were reviewed by a supervisor.

Statistical Analysis

Reliability for diagnoses was assessed with kappa (22) . Reliability for continuous variables was assessed with the intraclass correlation coefficient (ICC) (22 – 24) . Lifetime kappas represent reliability for diagnoses any time in the patient’s lifetime, including last 12 months. Current kappas exclude cases in partial remission except where otherwise noted.

Coefficients ≥0.75 indicate excellent reliability; 0.40–0.74, fair to good reliability; and ≤0.39, poor reliability (22) . Low prevalence (<0.05) is sometimes but not always associated with poor reliability. Reliability for low-prevalence conditions is presented as qualitative information.

Results

Substance Use Disorders

For most categories of DSM-IV substance dependence, reliability was good to excellent, including alcohol, cocaine, heroin, cannabis, and sedatives dependence ( Table 1 ). For other, low-prevalence substance dependence disorders, kappas were fair. For substance abuse diagnosed as required in DSM-IV (i.e., excluded if dependence was present), reliability ranged from poor to fair, as has been found elsewhere (14 , 25–28) . Abuse assessed independently of dependence showed substantially better reliability, indicating that the abuse criteria themselves were reliable. Reliability of dependence severity (number of criteria, range=0–7) was very good to excellent, as was reliability of age at onset of dependence ( Table 2 ).

Enlarge table

Enlarge table

Mood Disorders

Reliability was excellent for current primary (independent) major depressive disorder ( Table 3 ) and good for current substance-induced major depressive disorder, despite the low prevalence of the latter disorder. Reliability was very good for lifetime primary (independent) and substance-induced major depressive disorder. An either/or category indicating primary (independent) or substance-induced major depressive disorder had excellent lifetime and very good current reliability. Depression severity (number of symptoms, range=0–9) showed very good reliability (ICC=0.71). Reliability of dysthymia was poor, consistent with findings in other substance abuse studies (8) . This poor reliability was due to the difficulty of rating diffuse, mild mood symptoms in substance abusers and to the hierarchical relationship of dysthymia to major depressive disorder. The number of patients with a manic episode (primary or substance-induced) was very low; only a combined category (primary or substance-induced manic episode) had acceptable reliability.

Enlarge table

Anxiety Disorders

With some exceptions, the reliability of current and lifetime primary anxiety disorders was fair to good. The kappa for any lifetime primary anxiety disorder was 0.56 (SE=0.06) and for the either/or category of lifetime primary and substance-induced anxiety disorder was 0.55 (SE=0.06). However, reliability of individual substance-induced anxiety disorders (panic or generalized anxiety disorder) was poor, as was the reliability of any lifetime substance-induced anxiety disorder (kappa=0.33). The kappa of 0.58 for lifetime PTSD was acceptable.

Psychotic Disorders

Consistent with DSM-IV, hallucinations that the individual realizes are substance-induced when experienced are coded as intoxication/withdrawal symptoms and not counted toward a diagnosis of an independent psychotic disorder. Although psychotic disorders were rare, reliability was surprisingly robust ( Table 3 ). Reliability of any current independent psychotic disorder was excellent, as was the reliability of any substance-induced psychotic disorder, schizophrenia, and “substance-induced schizophrenia,” a condition meeting all the criteria for DSM-IV schizophrenia except that all active phases of the disorder occurred entirely within periods of heavy substance use. Brief psychotic disorder (primary and substance-induced) had lower but acceptable kappas. No cases of schizoaffective disorder, schizophreniform disorder, or delusional disorders were found in the study subjects.

Personality Disorders and Eating Disorders

For antisocial personality disorder, the PRISM-IV showed good reliability (kappa=0.69), with the main disagreement in the conduct disorder element, accounting for 76.7% of the discrepant antisocial personality disorder cases (also noted elsewhere [29] ). However, very good to excellent reliability was found for severity of conduct disorder ( Table 2 ) and adult antisocial symptoms. Reliability for borderline personality disorder and number of borderline personality disorder symptoms (the measure of severity) was good (kappa=0.67, ICC=0.66). Consistent with previous results (14) , anorexia and bulimia were rare, but the diagnoses of these disorders were highly reliable.

Including Partially Remitted Cases Among Those Considered Current

Expanding the current time frame to include episodes in partial remission did not substantially affect reliability. Reliability of current plus partially remitted dependence (e.g., three or more past symptoms, one or two current symptoms) was slightly lower than current dependence across drug categories (e.g., alcohol, kappa=0.74; cocaine, kappa=0.86; heroin, kappa=0.92). Similarly, minor differences emerged in reliability for primary and substance-induced major depressive disorder when current and partially remitted cases were combined (primary, kappa=0.71; substance-induced, kappa=0.63).

Administration Time

The mean administration time (based on all interviews except the first five done by each interviewer) was 2.03 hours (SD=0.91). We divided time (hours) into quartiles and examined the characteristics of each quartile. The mean administration times in the briefest, the second, the third, and the lengthiest quartiles were 1.05 hours (SD=0.23), 1.64 hours (SD=0.17), 2.18 hours (SD=0.18), and 3.26 hours (SD=0.75), respectively. The mean numbers of substance use sections covered in the briefest, second, third, and lengthiest quartiles were 3.19 (SD=1.80), 3.67 (SD=1.94), 4.04 (SD=2.25), and 4.93 (SD=2.41), respectively. The mean numbers of psychiatric sections covered because patients passed screening in the briefest, second, third, and lengthiest quartiles were 3.79 (SD=1.33), 4.83 (SD=1.82), 4.87 (SD=1.75), and 5.24 (SD=1.91), respectively. The mean numbers of lifetime psychiatric diagnoses made (other than abuse or dependence) in the briefest, second, third, and lengthiest quartiles were 0.61 (SD=0.90), 1.39 (SD=1.29), 1.79 (SD=1.43), and 2.69 (SD=1.64), respectively. Complicated substance use and psychiatric histories have previously been shown to lengthen administration times (30 , 31) .

Discussion

We conducted a test-retest study of DSM-IV diagnoses in substance-abusing patients by using the PRISM-IV, a diagnostic instrument designed to improve reliability in such samples. In developing the instrument, we used fundamental principles of psychometrics, including the need for clear criteria and guidelines for rating symptoms and syndromes. The study had a rigorous test-retest design and a large demographically and clinically varied sample. The results indicate that many DSM-IV disorders can be diagnosed reliably with the PRISM-IV in substance abusers, including substance dependence, primary and substance-induced major depressive disorder, primary and substance-induced psychotic disorder, some primary anxiety disorders, antisocial personality disorder, and borderline personality disorder.

Although reliability for substance dependence disorders was largely very good to excellent, reliability for DSM-IV substance abuse was lower, as was found previously (14 , 25–28) . However, as in earlier studies (7 , 32) , abuse reliability improved considerably when abuse was not ruled out by dependence. That the improvement in the reliability is not simply due to increased prevalence is suggested by the improvement in reliability of current hallucinogen abuse despite its remaining a low-prevalence condition. This finding indicates that abuse criteria are reliable but that their hierarchical relationship to dependence in DSM-IV reduces their reliability, a potential consideration for DSM-V.

The reliability of some DSM-IV anxiety disorders and dysthymia was lower than that of the other disorders. Although part of the reason might be low prevalence, the good to excellent reliability values obtained for some low-prevalence disorders in this study suggest that prevalence is not a complete explanation. The structure of the PRISM-IV interview did not allow assessment of the reliability of anxiety disorders or dysthymia independently of the occurrence of major depressive disorder. However, some of the unreliability of these anxiety disorders and dysthymia appeared to occur because of inconsistencies in reporting major depressive disorder in the first and second interview, which led to inconsistent skipping out of some of the anxiety and dysthymia sections. The influence of diagnostic hierarchies on reliability (also found for substance abuse, albeit in a different form) suggests the need for attention in DSM-V, and future studies should address this issue in more detail. Validation studies of anxiety disorders and dysthymia in substance abusers suggest problems that DSM-IV does not appear to have solved (33 , 34) , and no test-retest reliability information on these disorders in substance abusers has been published. Considerable interest exists in the substantive relationships of substance use disorders and anxiety disorders, but the results reported here suggest that reliability problems may impede progress in this area. More methodological information on the performance of DSM-IV primary and substance-induced anxiety categories in substance-abusing samples is needed, either to show that other assessment methods work better or to indicate that the diagnostic criteria themselves need improvement.

Reliability is an important characteristic of a measure, since an unreliable measure cannot be valid. The present study did not address PRISM-IV validity, but such information is emerging from other sources. PRISM-IV and SCID were compared to longitudinal, expert, all data (LEAD)/psychiatrist diagnoses in Spanish dual-diagnosis patients (35) . For major depressive disorder, substance-induced psychosis, anxiety disorders, antisocial personality disorder, and borderline personality disorders, agreement between the PRISM-IV and LEAD diagnoses was substantially higher (mean kappa=0.69) than agreement between the SCID and LEAD diagnoses (mean kappa=0.36). Another study validated the PRISM-IV differentiation between DSM-IV primary and substance-induced psychotic disorders in 287 emergency room patients (36) . Further, PRISM-IV data on the distinction between primary and substance-induced disorders are emerging as an important source of information on psychiatric disorders in substance abusers (21 , 37 , 38) . Given increasing use of the DSM-IV version of the PRISM, information on its reliability among substance abusers is important in interpreting results of these studies.

In this study, the PRISM-IV was tested in substance-abusing subjects, who have traditionally presented challenges to diagnostic accuracy due to complex, overlapping symptoms. Thus, the study served two purposes, one broader and the other more specific. The broader purpose was to address the reliability of the DSM-IV distinction between primary and substance-induced disorders. To our knowledge, this study is the first to do so. The results showed that many substance-induced psychiatric disorders could be evaluated with acceptable reliability. When reliability of primary and substance-induced disorders differed, reliability was usually lower for the substance-induced disorder, almost certainly reflecting the difficulty of differentiating these disorders from the expected effects of intoxication and withdrawal associated with specific substances. However, if we had attempted to make the substance-induced diagnoses without explicit duration and symptom requirements (following DSM-IV exactly), reliability would almost certainly have been far lower, and we wished to provide a method to assess these disorders with good reliability. The necessity of specific diagnostic criteria in achieving good reliability has been a fundamental principle underlying psychiatric diagnosis since DSM-III (16) . There is no reason to think that substance-induced disorders differ in this regard, a consideration that may be useful for DSM-V.

The second purpose of the study was to present the reliability of a specific instrument that may be used when study samples include many substance abusers. Demonstrating reliability of the PRISM-IV was important, because it differed considerably from the DSM-III-R version, including the addition of many diagnoses, adjustments for DSM-IV substance-induced disorders, and overall reduction in the number of items, complexity, and training time. These changes could all have reduced reliability and hence were important reasons to conduct a new reliability study. The study showed that many diagnostic categories have good to excellent reliability when evaluated with the PRISM-IV and better reliability than with the pre-DSM-IV instruments tested in substance-abusing samples (5 , 7 , 8 , 32) . Other instruments in widespread use, including the Composite International Diagnostic Interview and the NIMH Diagnostic Interview Schedule IV have no published reliability information in substance-abusing samples. The importance of good reliability has been illustrated in recent reviews indicating that studies in which rigorous methods are used were much more likely to find significant effects of psychiatric comorbidity on the follow-up status of patients with substance use disorders (1) and on treatment effects in clinical trials (39) .