OBJECTIVE: Very few studies have quantified the level of agreement among alternative diagnostic procedures that use a common set of fixed operational criteria. The authors examined the procedural validity of four independent methods of assigning DSM-III-R diagnoses of psychotic disorders. METHOD: The research was conducted as a satellite study to the DSM-IV Field Trial for Schizophrenia and Related Psychotic Disorders. The setting was the National Health and Medical Research Council Schizophrenia Research Unit's Early Psychosis Prevention and Intervention Centre, which focuses on first-episode psychosis. Consecutively admitted patients (N = 50) were assessed by independent raters who used four different procedures to determine a DSM-III-R diagnosis. These procedures were 1) the diagnostic instrument developed for the DSM-IV field trial, 2) the Royal Park Multidiagnostic Instrument for Psychosis, 3) the Munich Diagnostic Checklists, and 4) a consensus DSM-III-R diagnosis assigned by a team of clinician researchers who were expert in the use of diagnostic criteria. RESULTS: Concordance between pairs of diagnostic procedures was only moderate. Corresponding levels of percent agreement, however, ranged from 66% to 76%, with converse misclassification rates of 24%-34% (assuming one procedure to be "correct"). CONCLUSIONS: These findings have significant research and clinical implications. Despite the introduction of operationally defined diagnoses, there remained an appreciable level of differential classification or misclassification arising from variability in the method of assigning the diagnostic criteria rather than the criteria themselves. Such misclassification may impede neurobiological research and have harmful clinical effects on patients with first-episode psychosis.