Standardizing Measures in Four Domains of Employment Outcomes for Individual Placement and Support

The individual placement and support (IPS) model of supported employment is a systematic approach to helping people with severe mental illness achieve competitive employment (1). Reviews have concluded that IPS is effective in helping clients obtain competitive employment (2–9). But these reviews have been hampered by the lack of standardization in measures of competitive employment outcomes. Consequently, these reviews have been more tentative in drawing conclusions about the influence of IPS on many crucial dimensions of employment outcome. Specifically, although virtually all studies report outcomes for job acquisition—that is, whether a participant attained competitive employment at any time during the follow-up period—studies have been far less consistent in measuring job tenure, hours worked, and employment earnings. Consequently, cumulative knowledge about IPS outcomes has been limited with respect to these other dimensions of employment.

This lack of standardization in employment measures has contributed to uncertainty regarding the scope of effectiveness of IPS. One widely held view is that IPS is effective in helping clients to obtain jobs but not in helping them to keep jobs (10–15). Evaluation of this assertion has been hampered by the lack of consensus about how to measure job retention.

Competitive employment is multifaceted. However, we posit four conceptually distinct outcome domains, each of which can be assessed by operationally defined measures: job acquisition (employed at any time and rapidity of time to first job), job duration (weeks worked at any job and weeks worked at longest-held job), hours worked per week (hours worked per week and percentage of clients working at least 20 hours a week), and total hours and wages. The distinction between getting and keeping a job is widely recognized (16). The distinction between duration and hours worked per week recognizes that many clients work very part-time and others work close to full-time, regardless of the length of their employment. All these measures are positive indicators of successful employment.

In this study we examined two hypotheses. First, IPS improves both the rate of obtaining employment (job acquisition) and the amount of work (job duration, hours worked per week, and total hours and wages). Second, employment outcomes within domains are strongly correlated, whereas outcomes across domains are relatively weakly related.

Methods

Overview

We used archival data from four independent studies. Institutional review boards at local sites and participating universities approved the original projects. In addition, the data reanalyses were approved by the Indiana University-Purdue University Indianapolis Institutional Review Board.

Sample and procedures

The sample consisted of study participants from four randomized controlled trials in which IPS was compared with other vocational services (17–20). The follow-up period in these studies varied from 18 to 24 months; however, for the analyses reported here, we standardized the follow-up period to the first 18 months after enrollment. All four studies compared a newly established IPS program to one or more well-established vocational programs. Each study ensured fidelity to the IPS model through intensive training and consultation and ongoing monitoring with the IPS Fidelity Scale (21).

All study participants were clients enrolled in public mental health programs. They were unemployed adults who met each state's criteria for severe mental illness. Other common eligibility criteria included desire for competitive work, ability and willingness to give informed consent, and absence of significant general medical conditions precluding employment. The studies used similar protocols to track employment outcomes. Despite similar research methods, the four studies differed in geographic location and control group interventions (Table 1).

Measures

Competitive employment was defined as employment in integrated work settings in the open job market at prevailing wages with supervision provided by personnel employed by the business. We examined eight competitive employment outcomes: employment rate (that is, employed at any time), total weeks worked, job tenure in the longest-held job (defined as weeks worked on the longest-held competitive job), total hours worked, average hours per week worked, total wages (defined as total earnings from competitive employment), days to first job (defined as the number of days from entry into the IPS program or the alternative program to first competitive job), and ever working ≥20 hours per week (defined as working at least 20 hours a week at some time during follow-up). The measure of days to first job is a negative indicator of successful employment; that is, the longer the duration, the poorer the outcome. Percentages of participants employed at any time and ever working ≥20 hours per week are dichotomous measures; the others are continuous measures. Wage data were adjusted to December 2010 dollars according to the Consumer Price Index for All Urban Consumers. Finally, we determined the number of competitive jobs held during the follow-up period.

Statistical analyses

We combined IPS participants from the four studies into one composite sample and participants in the alternative programs into another sample (total IPS sample and total control sample). Within each group, we also examined the worker subsample—that is, participants who were competitively employed at any time during follow-up.

We first compared the 18-month competitive employment outcomes between the total IPS and control samples and then the same variables in the worker subsamples. We assessed employment rate (percentage employed at any time) for the total sample and days to first job for the worker subsamples only. We log-transformed positively skewed continuous variables before conducting statistical tests. We conducted t tests for continuous variables and chi square tests for dichotomous variables, and we computed effect sizes (d) (22) for differences between the IPS and control samples on continuous vocational outcomes. Effect sizes for the two dichotomous measures were estimated by using an arcsine transformation (23). Finally, we examined Pearson correlations between employment outcomes within the IPS and control groups for both the total sample and the worker subsample.

Results

Characteristics of the total sample—IPS participants and participants in the control sample—have been reported in detail elsewhere (24). The mean±SD age for the total sample was 39.5±9.0 years; 55% (N=372) were men; 39% were white (N=265), 46% were African American (N=311), and 13% were Latino (N=85). Thirty-four percent (N=229) had less than a high school education, and 81% (N=546) were receiving Social Security disability benefits. In terms of diagnosis, 63% (N=429) had a schizophrenia-spectrum disorder, and 34% (N=229) had a diagnosis of a mood disorder. With a few very minor exceptions, the IPS and control samples were similar on baseline characteristics.

The worker subsample consisted of 216 IPS participants and 91 participants in the control sample. Comparisons on baseline demographic and clinical characteristics in the worker subsample yielded two significant differences. Workers in the control sample had worked more weeks in paid community jobs during the past five years than IPS workers (93.12±83.59 compared with 61.12±69.46; t=3.21, df=305, p<.01, d=‒.43) and had received less in Social Security benefits during the past month ($402.84±$356.38 compared with $527.20±$343.31; t=‒2.85, df=301, p<.01, d=‒.36).

As shown in Table 2, in the total sample, IPS participants had significantly better outcomes than the participants in the control sample on all seven competitive employment measures, with effect sizes ranging from .51 to .96. Analyses of the worker subsamples revealed three significant differences favoring IPS over the control programs. Working clients in IPS started their first jobs sooner, averaged more weeks worked during follow-up, and had longer job tenure at their longest-held job compared with workers in the control programs. Among IPS workers, 105 (49%) held one job, 62 (29%) held two jobs, and 49 (23%) held more than two jobs, whereas among workers in the control sample, 58 (64%) held one job, 18 (20%) held two jobs, and 15 (17%) held more than two jobs (χ²=5.90, df=2, N=307, p=.05). The differences in the worker subsamples on other competitive employment outcomes were not significant. Overall, the large differences between the IPS and control samples prompted us to examine correlations between employment outcomes for each group separately.

Table 3 shows the correlations between the competitive employment outcomes in the total IPS and control samples. Most of the correlations were large, statistically significant, and in the predicted direction. The strongest associations were between total weeks worked and job tenure in the longest-held job and between total hours and total wages. The pattern of results was similar for the IPS sample and the control sample. Table 4 shows the correlations between the outcomes for the worker subsamples. The majority of these correlations were also significant.

Discussion

Our analyses strongly supported the first hypothesis: IPS outperformed control programs in all domains of employment outcome: acquisition, duration, hours worked per week, and total hours and wages. Even the analyses restricted to the worker subsamples showed significant advantages for IPS on three of seven outcome measures. The latter findings are remarkable because the comparisons between IPS workers and workers in the control programs were nonequivalent because of adverse selection, with significantly better work histories for control program participants. Thus IPS was superior to the other vocational models on all domains of competitive employment outcome, not just job acquisition.

Our analyses only partially supported the second hypothesis: relationships within vocational domains were generally strong, while those between domains were mixed. Within domains, measures were mostly highly correlated because of closely related operational definitions. Many of the cross-domain correlations were also strong, such as between the job duration domain and the domain of total hours and wages. In short-term follow-up studies, strong correlations are expected between measures such as total weeks worked and job tenure in the longest-held job, but the expected degree of association is less obvious in longer-term follow-up studies.

One important finding not previously reported in the literature is that the simple (and widely reported) measure of job acquisition (competitive employment rate) is a decent proxy for other employment measures assessing job duration, hours worked per week, and total hours and total wages, with correlations ranging from .37 to .79. In terms of interpreting research reports, the competitive employment rate statistic (that is, the percentage of participants who gain competitive job during follow-up) has been rightly criticized as a crude indicator. But the findings reported here suggest that although employment rate is an imperfect measure, it is useful as a general-purpose measure of employment outcome. This finding is practical and useful because the employment rate is often the easiest outcome to obtain and the sole measure used for quality improvement purposes (25). At a practice level, this also means that one critically important element in the process of helping clients achieve their goal of working is getting that first job. This by no means obviates the importance of job matching, ongoing support, and other elements of the employment process, but certainly job acquisition is a necessary condition for long-term employment.

This study also demonstrated that the principle of rapid job search is consistent with job duration, as suggested by significant negative correlations between days to first job and total weeks worked and job tenure in longest-held job. The findings contradict the assumption made by one early theoretical formulation of supported employment, the choose-get-keep model, which theorized that clients with severe mental illness required an extended period of career planning before starting the job search (26). According to this theory, the preliminary period of career planning would reap later benefits by facilitating longer job tenure once a client found a job suiting his or her career aspirations. This model (and theory) has been disconfirmed (27). The findings of this study further support the view that job search should not be delayed by skills training or other preparatory activities. No current evidence indicates that delaying the job search enhances job tenure. Moreover, a larger proportion of IPS sample than the control sample held multiple jobs during follow-up, suggesting a commitment by IPS programs to providing greater opportunities for clients to become attached to the work force. The findings validate the IPS principle of giving second chances to clients whose initial job ends.

Also illuminating were findings in the worker samples regarding measures that were at best weakly correlated. For example, correlations between the two job duration measures (job tenure and tenure in longest-held job) and hours worked per week were small, ranging from .16 to ‒.16, suggesting statistical independence between these two domains. We interpret these findings as suggesting that individuals' preference for working a certain number of hours per week—or their capacity to do so—may differ from their willingness or ability to stay in a particular job over time. This lack of association between job duration and hours worked per week suggests that people work in the amounts that meet their goals to feel productive, contributory, and in control, not necessarily to maximize income. For some, this is five hours per week; for others, it is 20 hours or more per week. In this study, the decision about the number of hours to work was not always limited by fears of losing benefits because many individuals worked well below that level. This interpretation accords with self-reports in long-term follow-up studies (28).

Generally, relationships between measures within employment domains are very strong because the measures often address closely related concepts, especially in short-term follow-up studies. For example, in studies with short follow-up periods, the longest-held job largely determines the total duration of employment, and hours of work determine wages, because most clients work in entry-level jobs with a relatively narrow range of wages.

For economic analyses of employment outcomes that combine findings across studies, the strong association between hours worked and total earnings suggests a convenient analytic strategy. Specifically, economists could aggregate findings across multiple studies by using the proxy measure of hours worked and subsequently converting the results to their approximate dollar equivalent. This technique may permit researchers to combine findings across studies conducted in different time periods and in different communities (or nations) and may permit inferences about earnings from employment that might circumvent some imputation challenges related to discounting, cost of living, minimum wage, and the like. Estimation strategies that minimize the assumptions, especially additive assumptions, may be both more transparent and more conservative.

We have no way of estimating how many part-time workers would work full-time if benefits regulations were to change. Some have speculated that clients receiving Social Security Disability Insurance perceive an “earnings cliff” that may result in a loss of benefits if they exceed substantial gainful activity for more than an allotted number of months (29). If this theory is correct, clients should avoid this eventuality by working up to, but not above, the earnings limit. In practice, however, relatively few clients seem to respond this way (30).

Finally, this study suggests that many employment measures are moderately to highly correlated, but it does not obviate the need to define a standard core battery of measures for future vocational studies (31). Such a battery is the single best way to ensure comparability across studies and to permit syntheses in meta-analyses.

We have proposed a conceptual framework for classifying employment measures. The fourfold framework is a beginning. A comprehensive framework would include many other domains. In addition, within each domain are many possible measures that were not examined in this study. For example, within the duration domain, rather than measuring weeks worked, an even more standardized measure of duration would be percentage of time worked (weeks worked divided by total weeks of follow-up). In this study, weeks worked and percentage of time worked yielded statistically equivalent results, but in analyses comparing data sets with variable follow-up periods, only the percentage measure is directly comparable (4). The Rehabilitation Services Administration defines “successful closure” for clients within its system by using a measure that fits in the duration domain (32). Another important measure of duration found in the IPS literature is the “steady worker” concept, defined as working at least 50% of the follow-up period (33). Vocational researchers would benefit by incorporating not only economic concepts related to productivity, which includes both quantity and quality of work, but also components of lost productivity, such as absenteeism and presenteeism (34). Vocational research should also routinely track clients working above substantial gainful activity.

Limitations of the findings reported here include the methodological shortcomings found in the original studies, which are related to relatively short follow-up, missing data, reliability of measurement, and other factors. As in other studies of IPS, we have focused exclusively on competitive employment. Future research might also examine noncompetitive employment outcomes. Type of occupation, as measured by the Standard Occupational Classification system (www.bls.gov/soc), should also be examined. Measures of job quality (35) are needed, as are measures of job satisfaction and other aspects of workers' experience.

Measuring job tenure in randomized controlled trials has been problematic because many participants are employed at the end of follow-up (that is, many job tenure periods are right-censored). Thus the literature consistently underestimates job tenure. The time between study enrollment and first day employed also limits job tenure because participants often spend the first few months of the study involved in engagement, job development, interviewing, and other activities. The optimal solution, of course, is to conduct long-term follow-up studies. In the absence of long-term follow-up, we propose weeks worked during follow-up as the single most valid measure of job duration (33). Weeks worked ignores the issue of number of jobs held, consistent with the IPS principle of helping clients obtain a new job if an initial job is unsatisfactory or ends for any reason.

Finally, these analyses did not include measures of variability between IPS teams and between employment specialists. The job strategies pursued by various IPS teams may differ, yielding higher competitive employment rates but jobs of shorter duration for clients of some teams (36).

Conclusions

We examined longitudinal competitive employment outcomes in a large database of clients with severe mental illness who were enrolled in IPS supported employment and other vocational services. Our analyses showed that IPS improves competitive employment outcomes in four domains: job acquisition, job duration, hours worked per week, and total hours and wages. Our findings also suggest that measures within these domains are strongly related, whereas measures are variably related between these domains. The correlational patterns suggest analytic and measurement strategies for use in future studies, especially in meta-analytic reviews in which proxy measures may be required.

Ideally, a unified conceptual framework that encompasses employment measures used in diverse fields of inquiry, including vocational rehabilitation, occupational therapy, longitudinal research, and economics, as well as measures used by federal agencies, such as the Department of Labor, the Rehabilitation Services Administration, and the Social Security Administration, would provide a common vocabulary and an opportunity to make meaningful comparisons across disciplines.

Figures and Tables