0
Get Alert
Please Wait... Processing your request... Please Wait.
You must sign in to sign-up for alerts.

Please confirm that your email address is correct, so you can successfully receive this alert.

1
Article   |    
Diagnostic Reliability of Telepsychiatry in American Indian Veterans
Jay H. Shore, M.D., M.P.H.; Daniel Savin, M.D.; Heather Orton, M.S.; Jan Beals, Ph.D.; Spero M. Manson, Ph.D.
Am J Psychiatry 2007;164:115-118. doi:10.1176/appi.ajp.164.1.115

Abstract

Objective: This study examined the reliability of the Structured Clinical Interview for DSM-III-R (SCID) in the administration of psychiatric assessments by real-time videoconferencing compared to face-to-face assessment within a rural American Indian community. Method: The SCID was administered to 53 male American Indian veterans who were randomly assigned over two separate occasions by different interviewers to face-to-face and real-time interactive videoconferencing within 2 weeks. Comparisons were made with prevalences, the McNemar test, and the kappa statistic. Results: With the exception of past-year substance dependence and abuse/dependence combined, there were no significant differences between face-to-face and videoconference administration. The majority of kappas calculated (76%) indicated a good or fair level of agreement. Externalizing disorders tended to elicit greater concordance than internalizing disorders. Conclusions: Overall, SCID assessment by live interactive videoconferencing did not differ significantly from face-to-face assessment in this population. Videoconferencing is a viable vehicle for clinical and research purposes.

Abstract Teaser
Figures in this Article

Mental illness is among the most important health concerns facing American Indian communities today. Telepsychiatry, in the form of live interactive videoconferencing, has the potential to address many of the challenges of clinical service and research among rural American Indians (1). Although they have been encouraging, the handful of articles on American Indian telepsychiatry is limited to case reports and program descriptions (2–4). To our knowledge, no well-controlled studies exist regarding the impact of videoconferencing on psychiatric assessment and diagnosis in this special population. Several investigators have addressed the reliability, validity, and feasibility of administrating structured clinical interviews, including the Structured Clinical Interview for DSM (SCID), by means of interactive videoconferencing in other patient populations (5). The study most relevant to this report examined the reliability of SCID diagnoses of major depression, bipolar disorder, panic disorder, and alcohol dependence among 15 veterans assessed face-to-face and by means of videoconferencing (6).

Panic disorder had the greatest concordance; major depressive disorder, the least. Although these results were promising, this study was limited by its small size, its exclusion of posttraumatic stress disorder (PTSD) and other diagnoses, and its failure to consider how racial status may have affected diagnostic concordance.

Two large-scale studies have demonstrated that the SCID can be administered reliably and validly among American Indians, thus setting the stage for determining whether teleconferencing holds similar promise (7, 8).

The purpose of this study was to examine the reliability of the SCID in the administration of psychiatric assessments by means of real-time videoconferencing between an academic medical center and a rural American Indian community. We used an experimental design to randomly assign American Indian military veterans to both face-to-face and real-time interactive video conditions of SCID assessment on two separate occasions within a 2-week period. We hypothesized that diagnoses obtained through videoconferencing would agree with those acquired through face-to-face interviews.

This study was conducted among a rural Northern Plains American Indian reservation population that had participated in a previous study known as the American Indian Vietnam Veterans Project (8). When the present study was initiated, veterans from the previous study represented the best sample available of American Indians with a known prevalence of lifetime psychiatric disorders, including PTSD (8). The substantial prevalence of well-characterized cases of mental illness was critical to testing the reliability of the SCID in this manner. Bias due to respondents’ prior familiarity with the SCID was minimized by the 8 years or more that had elapsed since the conclusion of the American Indian Vietnam Veterans Project.

A videoconferencing link (an integrated services digital network line, 1¼ T, 384K, with full remote capacity) was established between the American Indian Vietnam Veterans Project and a private room in the community’s Tribal Veterans Center. Colorado Multiple Institutional Review Board approval, Health Insurance Portability and Accountability Act consent, and tribal approvals were obtained before the project’s initiation, as well as written informed consent from each veteran. A power analysis indicated that 50 respondents tested twice would provide 80% power to detect the difference between kappas of 0.6 and 0.8 (9). A list of available participants was generated from the American Indian Vietnam Veterans Project sample.

In order to achieve adequate power, 20 additional Vietnam-era veterans were identified and recruited from the community. Of the 70 veterans approached, 60 (86%) agreed to participate, of whom 53 (76%) completed both interviews. Thirty-eight (72%) were drawn from the American Indian Vietnam Veterans Project sample. The participants were administered a version of the SCID for DSM-III-R that was modified for the American Indian Vietnam Veterans Project and further adapted in the American Indian Services Utilization, Psychiatric Epidemiology, Risk and Protective Factors Project (7). As in the American Indian Vietnam Veterans Project and the latter studies, several modules were eliminated from the SCID because of the low rate of the disorders in question (Table 1). Minor modifications of the SCID were made in the American Indian Vietnam Veterans Project to render some questions more culturally relevant. Two American Indian Vietnam Veterans Project psychiatrists were selected and trained to administer the SCID. Both had extensive experience working with American Indians and PTSD in cross-cultural settings. The interviewers were trained to administer the SCID in the same manner as in the American Indian Vietnam Veterans Project and the American Indian Services Utilization, Psychiatric Epidemiology, Risk and Protective Factors Project. Videotapes created specifically for the latter projects were used to assess interrater reliability, with an average kappa >0.80 obtained by both interviewers. Once enrolled, the participants were randomly assigned to either of two conditions: 1) a real-time interactive videoconference followed by a face-to-face interview or 2) a face-to-face interview followed by a real-time interactive videoconference. The paired SCIDs were to be completed within 2 weeks of each other (average time between interviews, 10.8 days). The face-to-face interviews were conducted at a private office in the community. For the real-time interactive videoconferencing, the participants were seen at the Tribal Veterans Program; the interviewer was located in Denver. Interviews took between approximately 80 and 90 minutes to complete in both conditions. No one else was present during the interviews.

The interviews were conducted in 2003, spanning 12 months, in four cohorts of approximately 10–15 participants each. Completed SCIDs were keyed into a computer; the resulting data were transformed into an SPSS file (10), cleaned, and organized in system files for analyses. In order to determine if the interview sequence had an effect on the diagnosis of disorders, a paired T test for dependent samples was performed; no significant difference was found. The prevalence of each diagnosis was calculated for both assessment modalities. In order to compare the proportion of participants with a particular diagnosis from the face-to-face session to the proportion from the videoconference session, McNemar’s test for dependent data was used (11). Interrater reliability between the two administration modalities was assessed with the kappa statistic, which accounts for agreement that occurs by chance. Cutoff values suggested by Byrt (12) were used to classify kappa estimates as excellent (>0.92), good/very good (>0.6), and fair (>0.4). One limitation of the kappa statistic is that it tends to be underestimated when the true prevalence in the population of interest is low (13). Because the prevalence of several of the diagnoses was low in this population, the percent agreement was also calculated (13). An emerging pattern led to further examination of the kappas by diagnoses that can be classified as internalizing or externalizing disorders. Grouping diagnoses in these terms was informed by an analysis from the National Comorbidity Survey (14); kappa statistics were calculated for past month.

The sample was relatively homogenous. All subjects were American Indian male Vietnam-theater and -era veterans from a Northern Plains tribe. The mean and median age was 54 years (range=46–71). Ninety percent had been previously married, and 32% were currently married. Half of the sample (53%) had completed some form of higher education, and 30% had completed high school/the graduate educational development (GED) test or trade school. Table 1 presents the prevalence and reliabilities by diagnosis for the past month, the past year, and lifetime by modality. With the exception of past-year substance dependence and abuse/dependence combined, there were no significant differences in prevalence between the face-to-face and live interactive videoconferencing modalities. Percent agreement between modalities was greater than 80%, except for lifetime drug abuse (76%), lifetime substance abuse (72%), and lifetime major depressive disorder (66%). The majority (76%) of kappa statistics were classified as good agreement or fair agreement (greater than 0.6) (12). Externalizing disorders elicited a higher kappa (0.73) than internalizing disorders (0.53).

This is the first study, to our knowledge, to examine the diagnostic reliability of administering a structured clinical interview by means of videoconferencing to American Indians. We believe that it is also the largest reliability study of the administration of the SCID by these means and the first to examine the reliability for PTSD, substance abuse and dependence, generalized anxiety disorder, and antisocial personality disorder. Overall, with some notable exceptions, SCID assessments by videoconferencing did not differ significantly from those obtained in face-to-face interviews.

Our data indicate that, with the exception of PTSD, SCID assessments by videoconferencing with this population may more easily detect externalizing disorders. The criteria for externalizing disorders are more behavioral and less subjective by nature, possibly rendering these symptoms easier to elicit through this modality. It may be more difficult to engage individuals with internalizing disorders through videoconferencing and thus identify relevant symptoms. PTSD possibly represents a unique internalizing diagnosis, one in which the real-time interactive videoconferencing actually enhances symptom reporting (3).

This study has several limitations. The 2-week interval between interviews could have introduced symptom changes affecting the reliability of current diagnoses. The low prevalence of certain disorders precluded meaningful conclusions about diagnostic reliability. The high prevalence and comorbidity of most conditions may have complicated the diagnosis of any specific disorder. The ethnic homogeneity limits our ability to generalize these findings to other populations.

Despite its limitations, this study represents an important first step toward understanding the diagnostic reliability of administering important clinical tools such as the SCID among American Indian veterans. This is especially timely because clinical work along these lines is already in process (4). The next steps should include determining the impact of cultural factors on this technology and identifying additional clinical tools for telepsychiatric application and adaptations needed in clinical and research protocols to improve the reliability of telepsychiatric assessment.

The capacity of this technology must be better understood in order to bend it to our clinical and research needs, with American Indians as well as other rural minority populations to fulfill its promise of increasing both access and quality of care.

+Received Nov. 2, 2005; revision received Feb. 1, 2006; accepted April 7, 2006. From the American Indian and Alaska Native Programs, University of Colorado Health Sciences Center. Address correspondence and reprint requests to Dr. Shore, American Indian and Alaska Native Programs, University of Colorado Health Sciences Center, Nighthorse Campbell Native Health Building, Mail Stop F800, P.O. Box 6508, Aurora, CO 80045-0508; jay.shore@uchsc.edu (e-mail). All authors report no competing interests.Supported by grants 1F32-MH-63527-01A1 (to Dr. Shore, principal investigator) and P60-MD-000507 (to Dr. Manson, principal investigator).

1.Dixon Mar Y (ed): Promises to Keep: Public Health Policy for American Indians and Alaska Natives in the 21st Century. Washington, DC, American Public Health Association, 2001
 
2.Shore JH, Manson SM: A developmental model for rural telepsychiatry. Psychiatr Serv 2005; 56:976–980
 
3.Shore JH, Manson SM: The American Indian veteran and posttraumatic stress disorder: a telehealth assessment and formulation. Cult Med Psychiatry 2004; 28:231–243
 
4.Shore JH, Manson SM: Telepsychiatric care of American Indian veterans with post-traumatic stress disorder: bridging gaps in geography, organizations, and culture. Telemed J E-Health 2004; 10(suppl 2):64–69
 
5.Monnier J, Knapp RG, Frueh BC: Recent advances in telepsychiatry: an updated review. Psychiatr Serv 2003; 54:1604–1609
 
6.Ruskin PE, Reed S, Kumar R, Kling MA, Siegel E, Rosen M, Hauser P: Reliability and acceptability of psychiatric diagnosis via telecommunication and audiovisual technology. Psychiatr Serv 1998; 49:1086–1088
 
7.Beals J, Novins DK, Spicer P, Orton HD, Mitchell CM, Baron AE, Manson SM, Big Crow CK, Buchwald D, Chambers B, Christensen ML, Dillard DA, DuBray K, Espinoza PA, Flemming CM, Frederick AW, Gurley D, Jervis LL, Jim SM, Kaufman CE, Keane EM, Klein SA, Lee D, McNutly MC, Middlebrook DL, Moore LA, Nez TD, Norton IM, Randall CJ, Sam A, Shore JH, Simpson SG, Yazzie LL, the American Indian Services Utilization, Psychiatric Epidemiology, Risk and Protective Factors Project Team: Challenges in operationalizing the DSM-IV clinical significance criterion. Arch Gen Psychiatry 2004; 61:1197–1207
 
8.National Center for Post-Traumatic Stress Disorder and the National Center for American Indian and Alaska Native Mental Health Research: Matsunaga Vietnam Veterans Project. White River Junction, Vt, 1996
 
9.Hintze JL: Power Analysis and Sample Size (PASS) for Windows User’s Guide. Kaysville, Utah, NCSS, 2000
 
10.Statistical Package for the Social Sciences. Chicago, SPSS, 2001
 
11.Rosner B: Fundamentals of Biostatistics. Pacific Grove, Calif, Duxbury, 2000
 
12.Byrt T: How good is agreement? Epidemiology 1996; 7:561
 
13.Szklo M, Nieto F: Epidemiology: Beyond the Basics. Gaithersburg, Md, Aspen, 2000
 
14.Krueger RF: The structure of common mental disorders. Arch Gen Psychiatry 1999; 56:921–926
 
+

References

1.Dixon Mar Y (ed): Promises to Keep: Public Health Policy for American Indians and Alaska Natives in the 21st Century. Washington, DC, American Public Health Association, 2001
 
2.Shore JH, Manson SM: A developmental model for rural telepsychiatry. Psychiatr Serv 2005; 56:976–980
 
3.Shore JH, Manson SM: The American Indian veteran and posttraumatic stress disorder: a telehealth assessment and formulation. Cult Med Psychiatry 2004; 28:231–243
 
4.Shore JH, Manson SM: Telepsychiatric care of American Indian veterans with post-traumatic stress disorder: bridging gaps in geography, organizations, and culture. Telemed J E-Health 2004; 10(suppl 2):64–69
 
5.Monnier J, Knapp RG, Frueh BC: Recent advances in telepsychiatry: an updated review. Psychiatr Serv 2003; 54:1604–1609
 
6.Ruskin PE, Reed S, Kumar R, Kling MA, Siegel E, Rosen M, Hauser P: Reliability and acceptability of psychiatric diagnosis via telecommunication and audiovisual technology. Psychiatr Serv 1998; 49:1086–1088
 
7.Beals J, Novins DK, Spicer P, Orton HD, Mitchell CM, Baron AE, Manson SM, Big Crow CK, Buchwald D, Chambers B, Christensen ML, Dillard DA, DuBray K, Espinoza PA, Flemming CM, Frederick AW, Gurley D, Jervis LL, Jim SM, Kaufman CE, Keane EM, Klein SA, Lee D, McNutly MC, Middlebrook DL, Moore LA, Nez TD, Norton IM, Randall CJ, Sam A, Shore JH, Simpson SG, Yazzie LL, the American Indian Services Utilization, Psychiatric Epidemiology, Risk and Protective Factors Project Team: Challenges in operationalizing the DSM-IV clinical significance criterion. Arch Gen Psychiatry 2004; 61:1197–1207
 
8.National Center for Post-Traumatic Stress Disorder and the National Center for American Indian and Alaska Native Mental Health Research: Matsunaga Vietnam Veterans Project. White River Junction, Vt, 1996
 
9.Hintze JL: Power Analysis and Sample Size (PASS) for Windows User’s Guide. Kaysville, Utah, NCSS, 2000
 
10.Statistical Package for the Social Sciences. Chicago, SPSS, 2001
 
11.Rosner B: Fundamentals of Biostatistics. Pacific Grove, Calif, Duxbury, 2000
 
12.Byrt T: How good is agreement? Epidemiology 1996; 7:561
 
13.Szklo M, Nieto F: Epidemiology: Beyond the Basics. Gaithersburg, Md, Aspen, 2000
 
14.Krueger RF: The structure of common mental disorders. Arch Gen Psychiatry 1999; 56:921–926
 
+
+

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.
Submit a Comments
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of APA editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe



Web of Science® Times Cited: 38

Related Content
Articles
Books
The American Psychiatric Publishing Textbook of Substance Abuse Treatment, 4th Edition > Chapter 45.  >
Dulcan's Textbook of Child and Adolescent Psychiatry > Chapter 44.  >
Dulcan's Textbook of Child and Adolescent Psychiatry > Chapter 44.  >
The American Psychiatric Publishing Textbook of Substance Abuse Treatment, 4th Edition > Chapter 45.  >
The American Psychiatric Publishing Textbook of Substance Abuse Treatment, 4th Edition > Chapter 42.  >
Topic Collections
Psychiatric News
PubMed Articles