In the past few decades, about 1,000 controlled studies have examined the effects of mental health programs aimed at preventing mental health problems at school (1, 2), substance use and abuse at school (3), work-related stress (4), distress among caregivers for the elderly (5, 6), depression (7, 8), aggression and behavior problems in children (9), child abuse (10–12), and several other conditions. This considerable body of research has shown that some prevention programs in mental health are capable of strengthening protective factors, such as social skills, problem-solving skills, stress-management skills, pro-social behavior, and social support; that these programs can reduce the consequences of risk factors, psychiatric symptoms, and substance use; and that they may have positive economic effects.
Despite this large body of research, few studies have examined whether these prevention programs are actually capable of reducing the incidence of new cases of mental disorders defined according to diagnostic criteria, although this research question can easily be regarded as one of the most important ones, both from a public health perspective and from a scientific point of view.
One major reason why the prevention of new cases of mental disorders has hardly been examined is that very large numbers of subjects are needed to yield sufficient statistical power to be able to show significant effects on incidence. This "power problem" is related to the lack of understanding of the exact pathways leading to mental disorders and to the very low specificity of most known risk factors. This low specificity implies that most subjects who are exposed to the risk factor do not develop the disorder and that one such risk factor by itself is not sufficient to produce the disorder (13).
In this article, I will explore this problem of statistical power in different types of prevention research and will discuss possible solutions and research directions for solving this problem. An influential report of the Institute of Medicine (14) on prevention of mental disorders distinguishes three types of prevention of mental disorders. Universal prevention is aimed at the general population or parts of the general population, regardless of whether they have a higher than average risk of developing a disorder. The best-known examples of universal prevention include school programs and mass media campaigns. Selective prevention is aimed at high-risk groups, who have not yet developed a mental disorder. Indicated prevention is aimed at individuals who have some symptoms of a mental disorder but do not meet diagnostic criteria. Universal, selective, and indicated prevention are aimed at subjects who do not yet have a mental disorder.
The next three sections will address the problem of statistical power for studies examining each of these three types of prevention. Some solutions to the observed problems will be discussed after them.
The problem of statistical power is most prominent in studies examining universal prevention programs for mental disorders, and this is especially true for disorders with low incidence rates. For example, the incidence rate of anorexia nervosa, a severe and often fatal mental disorder, is estimated to be about 6.3 cases in 100,000 subjects per year (15). A simple power calculation shows that if we were to have a universal intervention that could reduce this incidence by one-third, we would need about 1.5 million subjects in an experimental condition plus another 1.5 million subjects in a control condition to be able to show that this intervention has indeed reduced the incidence (if we assume statistical power to be 0.80 and alpha 0.05, one-sided). In absolute numbers, a prevention study involving 1,565,749 subjects would result in a reduction from 95 new cases of anorexia nervosa in the control group to 63 new cases in the experimental group. Of course, we could drastically reduce the number of needed subjects for a prevention study by concentrating on women, as anorexia nervosa in men is almost nonexistent, or by assuming that a preventive program has effects that reach beyond 1 year. But, whatever we do, the number of needed subjects is enormous and adds up to hundreds of thousands. By way of comparison, a review of school programs aimed at the prevention of eating disorders included six studies, with a total of 1,905 subjects (16). These programs may show effects on knowledge about dieting, on skills, or on eating habits, but a study giving evidence of a reduction in the incidence of eating disorders because of a universal prevention program is clearly not feasible.
Research examining universal prevention of mental disorders with higher incidence rates, such as mood disorders, anxiety disorders, and substance use disorders, has basically the same power problem as prevention of low-incidence disorders, except that the numbers of subjects needed to show a significant reduction are somewhat smaller. F1 shows how many subjects are needed in studies of prevention programs to be able to show a reduction of 15% in the new cases of a mental disorder and the numbers needed to show a reduction of 30%. For example, the incidence of major depression, one of the most common mental disorders, was recently estimated to be 1.7% per year (17). In order to be able to show that a universal prevention program has reduced this to 1.4% (a reduction of 15%), both the experimental group and the control group need to consist of 30,211 subjects. This is considerably smaller than the numbers needed in research for prevention of anorexia nervosa, but it is still very large. For example, the National Comorbidity Survey, one of the most recent and influential epidemiological studies of mental disorders in the United States, consisted of 8,098 subjects (18).
Accordingly, the sheer number of subjects needed to examine the effects of universal prevention programs on the incidence of mental disorders makes this type of research almost impossible to realize. And this is complicated by the ethical problem of imposing an intervention on a population of whom only a fraction will develop the disorder. This does not mean, of course, that universal prevention has no effects on the incidence of mental disorders, but it does mean that it is almost impossible to demonstrate these possible effects in scientific research.
There are more problems in universal prevention that complicate this research even further. One problem is related to the fact that prevention programs typically try to reduce the influence of existing risk factors for mental disorders (14). This is true for universal, selective, and indicated prevention. A large body of research has shown that no single risk factor for mental disorders explains more than 15% of the onsets (19). If a universal intervention is 100% effective in eliminating one risk factor or all consequences of one risk factor, the incidence of new cases of mental disorders cannot be reduced by more than 15%. And an effectiveness of 50% (which would be more realistic) instead of 100% would again dramatically increase the number of subjects needed per experimental condition.
Other problems are that many important risk factors—such as hereditary conditions, life events, gender, and age—cannot be influenced and that the ability of prevention programs to influence risk factors is limited. It may be possible to influence some risk factors, such as socioeconomic status or education level, but this would need political or social actions instead of health programs. Therefore, prevention programs necessarily focus on psychological and psychosocial risk factors and on the consequences of risk factors that cannot be influenced themselves.
Research examining the effects of selective prevention programs is also hampered by power problems. As indicated earlier, these problems can be traced back to the fact that the exact causes of mental disorders are not yet known. Most theoretical models that try to explain the development of mental disorders are based on the assumption that these disorders are caused by a complex interplay of physical, social, and psychological factors. Mental disorders are never caused by one of these factors alone, as no single risk factor explains more than about 15% of the onsets (19). And which combination of factors causes the disorder in a specific individual at a specific time is not known. Moreover, the specificity of most risk factors is low, implying that most subjects who are exposed to the risk factor do not develop the disorder and that one such risk factor by itself is not sufficient to produce the disorder (13).
For example, in a study of subjects who lost their partners, it was found that about 16% met criteria for major depression 1 year after the death, compared to 4% of comparable subjects who did not lose their partners (20). Suppose we could develop an intervention that is capable of reducing this incidence rate by 30%. In this case we would need 671 subjects in each experimental condition of an effect study in order to show this result. This would mean that 671 subjects would be submitted to an intervention in order to reduce the number of new depressive cases from 107 to 75.
Although the feasibility of such a study is better than for a study of universal prevention, the numbers needed for such a trial are still considerable and the costs would be very high. As the presence of a mental disorder has to be determined with a personal diagnostic interview, a study of about 1,350 subjects with repeated measurements would be expensive and would need a strong commitment from financers and researchers.
Only a few randomized trials have examined the effects of selective prevention (for subjects belonging to high-risk groups without meeting criteria for current mental disorders) on the incidence of mental disorder. One of these is a study on the prevention of depression in ethnic minorities with chronic illness by Muñoz and colleagues (21, 22). Unfortunately, the statistical power in this study was too small to show significant effects on the incidence of mental disorder.
In another study, the effects of a preventive intervention for pregnant women on the incidence of postpartum depression were examined (23). It was found that 33% of the subjects in the control group (N=18) developed a postpartum depression, compared to none in the experimental group (N=17). The power problem clearly does not exist in a trial with such a high incidence rate and such large contrast between the experimental and control groups. Other studies in this area, however, did not show such positive outcomes (24). Several more studies have examined the effects of debriefing on the prevention of posttraumatic stress disorder. In a meta-analysis of these studies (25), it was concluded that there is no evidence that debriefing prevents the incidence of posttraumatic stress disorder and that it possibly even has negative results.
Three major randomized, controlled studies of indicated prevention programs (aimed at subjects who have symptoms of a mental disorder but do not meet criteria for the full-blown disorder) that examined the effects of prevention on the incidence of mental disorders are summarized for illustrative purposes in t1. These studies do not give a comprehensive overview of studies examining the effects of prevention on the incidence of new cases of mental disorders. Other pilot studies have examined the effects of preventive interventions on psychosis (29) and generalized anxiety disorder (30). But the three studies in t1 are among the best trials in the field and can illustrate the power problem well. Two of the three studies are aimed at the prevention of depression, and one targets anxiety disorders. In all three studies a significant reduction of the incidence of new cases of mental disorders was found.
These studies have clearly solved the power problem. Where the incidence of mental disorders is never higher than 15% in high-risk groups, the incidence in subjects who already have some symptoms of the disorder without meeting full criteria can be much higher. The control group in the 2001 study by Clarke and colleagues (26) had an incidence of major depressive disorder of 28.8%, and the intervention reduced this rate to 9.3%. In this study, adolescent children of depressed parents, who themselves had depressive symptoms but no major depressive disorder, received a psychoeducational cognitive behavior training program for mood management to prevent the onset of major depression. The study by Dadds and colleagues (28) even showed an incidence rate of 54% in the control group and 16% in the experimental group (at 6-month follow-up). In this study, primary school children with subclinical anxiety symptoms, but no anxiety disorder, received a cognitive behavior intervention to prevent the onset of an anxiety disorder.
These studies illustrate that relatively small numbers of study subjects are needed when the incidence rate in the target population is high enough. Such studies use a relatively simple method of identifying a high-risk group. They screen a population for the presence of symptoms of a mental disorder. Potential participants who score high on the screening test and meet diagnostic criteria for the mental disorder are excluded from the study, and those who do not meet diagnostic criteria enter the trial. They are randomly assigned to the prevention program or to a control condition. If the incidence in this population is high enough, the number of subjects needed to yield sufficient power is relatively small. And if the intervention is very successful in reducing the incidence, such as in the study by Dadds and colleagues (28), the number of needed subjects can be reduced to less than 50 for the total study group.
There are several ways to increase the statistical power in prevention studies. I will discuss the most important ones.
Focus on Populations With High Incidence Rates of Mental Disorders
The identification of high-incidence groups can be realized in several ways.
Focus on indicated prevention
The studies described in t1, which represent indicated prevention programs, have shown that populations who have symptoms of a mental disorder but do not meet diagnostic criteria can have high incidence rates. Many other studies, especially in the area of depression, have shown that subjects with symptoms of mental disorders have high incidence rates (31, 32), although in some studies the incidence rate remains 10% or lower (33, 34). It is possible that in prevention studies in which a complete population is screened, the only subjects who agree to participate are those whose problems are more serious and whose risk of developing a full-blown disorder is greater than that of subjects who decline participation in the study.
Focus on high-risk groups with multiple risk factors
Another way of identifying a high-incidence target group was demonstrated in the study by Clarke and colleagues (26). In this study subjects with symptoms of depression were selected from a high-risk group (children of depressed parents). The incidence rate in a high-risk group is usually low, but combining membership in such a high-risk group with the presence of symptoms of the disorder probably increases the incidence rate dramatically. It may also be possible to define target groups with high incidence rates by combining risk factors, such as recently widowed women with low social support or sons of alcoholic fathers who are unemployed and get divorced.
Focus on target groups with multiple disorders
Focusing the prevention program on several mental disorders could increase the incidence rate in the target population. For example, in a recent large epidemiological study among Dutch adults (17), the 1-year incidence of major depression was 1.7%, while the incidence of any DSM-III-R disorder was 5.6%, which is more than three times as high. Several prevention programs can easily focus on more than one disorder. For example, many prevention programs at school focus on generic life skills, such as coping skills, social skills, and cognitive skills. These life skills may affect the incidence of depression, anxiety disorder, and substance use disorder at the same time. Possibly, such programs could affect the incidence of internalizing problems, while other programs could affect the incidence of externalizing problems. Indicated preventive interventions aimed at the prevention of depression could be combined with the prevention of anxiety disorders because the rate of comorbid depression and anxiety is high and because behavioral approaches for prevention could use comparable methods based on cognitive and behavioral psychotherapies. The interventions described by Clarke and colleagues (26, 27) and by Dadds and colleagues (28) used several of these strategies.
Strengthen the Effects of Prevention Programs
The second way to reduce the problem of statistical power in prevention studies is to increase the effects of the intervention. Basically, in order to optimize the effect of a program, the goals and design of the program should be based on a theoretical framework, it should focus on the risk factors and protective factors that are known to be related to the disorder, and it should meet the needs of the target population. It is, for example, important to conduct good risk estimation studies before the evaluation of an intervention. Currently, there is often little evidence that the risk factor an intervention is targeting is indeed a causal factor in the process leading to a disorder. And how multiple risk factors work together in the causal path toward a disorder is even less examined. But it is this type of research that is necessary for the development of effective prevention programs.
As illustrated before, universal interventions alone can hardly be expected to result in a significant reduction of new cases of mental disorder, even when the intervention has been designed well. But combining universal interventions with selective and indicated interventions may result in larger effects than can be realized with each of these interventions alone.
Use Cumulative Meta-Analyses
The third way to increase statistical power is to make more efficient use of available trials in the prevention field. The cumulative meta-analysis is an effective way to realize this (35). In a cumulative meta-analysis, the results of each new study are integrated into the integrated results of all earlier studies. This reduces the numbers needed for a new trial considerably, as the numbers included in earlier studies would be used for this newly designed trial. For example, almost 100 relatively well-designed randomized trials have examined the effects of school-based prevention programs on substance use (3). If only a handful of these would have used diagnostic measures to examine the incidence of substance use disorder, a new trial could build on these data through cumulative meta-analysis and would need only a limited number of subjects in order to examine the effects on incidence. On the other hand, cumulative meta-analysis is possible only when multiple high-quality studies examining comparable interventions for comparable target study groups are available. This is currently not the case.
Use Other Ways to Increase Power in Prevention Studies
Other ways to increase power include extending the follow-up times (although this needs powerful interventions with strong effects) and using survival methods rather than fixed incidence counts. Improving the reliability of diagnoses would also increase the power of studies.
The lack of statistical power and the related large numbers of subjects needed for prevention studies are important reasons why so few studies have examined the effects of preventions programs on the incidence of mental disorders. This "power problem" makes it almost impossible to design feasible studies examining the effects of universal prevention on the incidence of mental disorders. Studies examining the effects of selective prevention on new cases are feasible but very expensive because of the high number of needed subjects. Indicated prevention can without any doubt be examined best, without the staggering numbers of subjects needed for universal and, to a lesser extent, selective prevention. Several studies examining indicated prevention have shown that this type of research is possible and feasible.
The indicated prevention studies have shown that well-designed interventions are capable of reducing the incidence of mental disorders. This is an important finding, from both scientific and public health perspectives. Many researchers assume that prevention of mental disorders is not possible because the exact causes and pathways that lead to these disorders are not known (36). The indicated prevention studies show that a reduction of the incidence of mental disorders actually is possible. However, critics could argue that subjects with subclinical symptoms of anxiety, depression, or substance use disorders are in fact in the prodromal phase of the disorder (37) and that the indicated intervention is in fact an early intervention, keeping the disorder from developing further instead of actually preventing it. It would therefore be interesting from a scientific point of view to examine whether selective prevention can also reduce incidence. Unfortunately, there is currently insufficient scientific evidence available to answer the question of whether real prevention (universal or selective) of mental disorders is possible.
A major challenge for future researchers in this area is to identify target populations with high incidence rates of mental disorders. Epidemiological research is needed for the identification of these populations. Much research in psychiatric epidemiology defines risk status only in terms of relative risk or odds ratio. Although these measures are informative in illustrating the strengths of risk factors, they are not sufficient for the design of prevention programs. As illustrated in this article, the incidence rate in a high-risk group has to be clear, but in order to compare the importance of different risk factors, other measures are also needed, such as attributable risk (37) and the numbers needed to be treated.
In this article, I have described the power problem of prevention studies, using the prospective, randomized prevention trial as the gold standard. Because of the enormous study group sizes needed for trials examining universal prevention and, to a lesser extent, selective prevention, the use of other designs, such as time-series designs and case-control studies, may be considered in some cases.
This article should not be interpreted as a recommendation to discard universal and selective prevention and focus on indicated prevention and treatment. Universal prevention may be effective in informing individuals about risk factors for mental disorders, reducing stigma related to mental disorder, or stimulating awareness of mental problems in general. Selective prevention programs have been shown to strengthen protective factors and reduce risk factors, and they may reduce the incidence of mental disorders. And combining universal, selective, and indicated prevention programs may result in larger effects than can be realized with each of these interventions alone.
The power problem described in this article is also a relative one. With nonpsychiatric conditions, huge study groups have been studied in prevention research, as was shown in a review of studies of newborn screening for cystic fibrosis (38). In this review two well-designed trials in this area were found, with a total of 1,124,483 included neonates. So there is no doubt that it is partly because of stigma that insufficient resources are available for this type of mental health research.
Five of the "top ten" causes of disability have been identified as mental disorders, and unipolar depression is expected to be the second leading cause of disability worldwide in 2020 (39). Reducing the burden of mental disorders by preventing the onset of new cases is possible, as was shown by the indicated prevention studies described in this article . The further development and study of preventive interventions in this area constitute major challenges for prevention science.
Received May 6, 2002; revision received Oct. 15, 2002; accepted Dec. 17, 2002. From the Trimbos Institute, Netherlands Institute of Mental Health and Addiction. Address reprint requests to Dr. Cuijpers, Trimbos Institute, P.O. Box 725, 3500 AS Utrecht, Netherlands; firstname.lastname@example.org (e-mail).
Numbers of Subjects Needed for Experimental and Control Conditions to Show That a Prevention Program Has Reduced the Incidence of a Mental Disorder by 15% or 30%a
aStatistical power=0.80, alpha=0.05. The numbers of needed subjects were calculated with Stata software, release 7 (Stata Corp., College Station, Tex., 2001). For the 15% reduction, the commands were sampsi I 0.85I, power(0.80) onesided, where I=incidence. For the 30% reduction, the commands were sampsi I 0.70I, power(0.80) onesided, where I=incidence.