Thirteen experienced supervising psychiatrists independently rated the
psychotherapy interviews of six residents, using a detailed assessment
instrument. Although interrater agreement was significantly greater than
would be expected by chance, the extent of agreement was uniformly low.
Agreement was significantly poorer for judgments about technique skills
than about communication skills and for second-year than for first-year
residents. Other factors, such as the faculty rank and training of the
rater, did not covary significantly with observed ratings. The findings
suggest that rating methods which purport to assess psychotherapy skills
should undergo careful methodological evaluation and testing before they
are accepted as valid assessment techniques.Abstract Teaser