OBJECTIVE: This study examined the validity of four psychiatric utilization management criteria sets: the 1992 and 1993 InterQual Intensity, Severity, and Discharge (ISD) criteria, the Managed Care Appropriateness Protocol, and an instrument developed by the Department of Veterans Affairs (VA). METHOD: The appropriateness of acute care admission and continued stay for 70 randomly selected VA psychiatric inpatients was retrospectively assessed with each criteria set. The sensitivity and specificity of each instrument were evaluated by comparing its assessments to the consensus of judgments of a panel of expert psychiatrists who reviewed the same cases. Sensitivity was defined as the proportion of admissions or continued-stay days the panel found appropriate for acute care that the criteria judged to be appropriate for acute care; specificity was the proportion the panel found inappropriate for acute care that the criteria judged to be inappropriate for acute care. RESULTS: For admissions, there were only minor differences in the validity of the four criteria sets as assessed by agreement with the panel's judgments. For each of 4 continued-stay days studied, either the sensitivity or specificity of the 1993 InterQual ISD criteria was below 0.30. The specificity of the 1992 InterQual ISD criteria was below 0.60 for 2 days. In contrast, for the Managed Care Appropriateness Protocol, sensitivity was 0.73-0.93 and specificity was 0.78-0.88 over the 4 days. CONCLUSIONS: The findings raise major concerns about the validity of the widely used InterQual ISD psychiatry criteria, suggest that the Managed Care Appropriateness Protocol may be a useful tool for psychiatric utilization management, strongly underline the need to validate all criteria used to assess medical care, and support the appropriateness of the procedures used to perform these assessments.