Improving the U.S. Workforce System by
Transforming Its Performance Measurement System
Randall W. EbertsW.E. Upjohn Institute for Employment Research November 2013
A well designed and executed system of performance measures and targets is an integral part of an effective workforce system. It is the means by which the workforce system tracks its results, communicates expectations among its key partners, allocates and aligns resources among its federal, state, and local entities, holds administrators accountable for performance, and drives continuous improvement. For nearly 30 years, the Employment and Training Administration of the U.S. Department of Labor has incorporated an outcomes-based performance system into many of its programs. Today, 15 federal workforce programs, serving nearly 20 million people annually, are subject to performance measures and targets. Three common measures are used: the percentages of participants finding a job and retaining a job, and the earnings levels of those employed.
However, an effective performance measurement system should and can go further than merely reporting the employment outcomes of participants to higher levels of government. Information within a performance measurement system can also flow in the other direction—to the customers. Using the same administrative data as used to record the common measures, a performance measurement system can provide important information to customers that can help job seekers find meaningful work and businesses find qualified workers. When structured in a longitudinal format, administrative data combined with UI wage records and educational outcomes can be a powerful tool. It can offer valuable insights into an individual’s future
employment prospects and what services may work best for them. Adding other “real-time” tools can provide job seekers and employment services staff with information about other occupations for which a person may be qualified, current job postings, and even what qualifications
businesses most prefer.
Therefore, an effective performance measurement system is only one component of an
effective, intelligent information system in which valuable information flows from customers to
providers and administrators and back to customers. Part of the information system should record and report the outcomes of participants so that local service managers and workforce investment
area administrators can manage the referral and delivery of services more effectiveness. This information can also flow to higher levels of government—state and federal levels—to better inform policy and hold the system accountable for meeting various targets.
Presently, the workforce system has many of the basic elements of an intelligent information system, but the various parts have not been well integrated into a system that can be used simultaneously by customers, managers, and decision makers. The current performance system of common measures and targets flows only in one direction from the customer and manager to the agencies to which they report. Because of reporting lags, that information is not very useful to managers or to customers, and there is little effort to make that information available to front-line staff. Furthermore, until recently, performance measures or targets have not taken into account differences in local labor market conditions or personal participant characteristics, which have significant effects on the ability of participants to find jobs. WIA’s predecessor, JTPA, made these adjustments using a statistical methodology, but this was abandoned when WIA was enacted in 1998. Only in 2009 were federal targets once again adjusted for these factors, and this approach has not yet been widely adopted at the state or local levels. Another element is the construction of longitudinal files to track progress along the continuum from education to employment. USDOL has recently launched an initiative for states to construct longitudinal data files linking K–12, post-secondary, and employment information. Such files would provide the opportunity to track the progress of students through the
educational continuum and into work and the effectiveness of educational institutions in moving students toward a meaningful career. Few states have completed this project, and it is uncertain at this time how these data will be made available to users.
Integrating this information into an accessible and coherent system would go a long way in providing current and relevant information customers, managers, and decision makers need to make more informed decisions. The purpose of this paper is to describe the merits of such an information/performance measurement system and the attempts that have been made to develop and implement components of it. The paper will first describe and critique the existing performance measurement system. Next, it will describe the various components, which if combined systematically, could lead to an effective intelligent system. The components are described within three major subheadings related to the three types of customers—job seekers, businesses, and workforce program administrators. More could be added to this list, such as decision makers and subcontractors, which include community colleges as examples of training and basic education providers. However, the three included in the paper are sufficient to illustrate how their needs should and can be met. Furthermore, some of the components discussed, such as the Frontline Decision Support System (FDSS), are applicable to more than one type of
customer. The paper concludes with an overall recommendation. In contemplating a more intelligent information system that can be an effective decision-making tool, the following enhancements are considered. The first necessary enhancement is that the information must be easily accessible in a meaningful format. The information must make sense to customers. The second enhancement is to use the collective experience of customers, as embodied in the full set of information, to guide decisions. This requires statistical algorithms that can help customize the data to address the employment prospects and service effectiveness for customers with specific characteristics.
Current WIA Performance Measurement System
The Workforce Investment Act (WIA) legislation requires a comprehensive
performance accountability system for the three WIA programs—Adult, Dislocated Worker, and Youth. The Government Performance and Results Act (GPRA) requires that all federal
programs set performance targets and establish performance tracking systems. While each of the 15 federal workforce programs has established performance measures and sets annual targets, only the WIA programs will be discussed here. The purpose of the accountability system is to optimize the return on investment of federal workforce development funds by assessing the effectiveness of states and local Workforce Investment Areas (LWIAs) in achieving continuous improvement of workforce investment activities funded under the WIA. The U.S. Department of Labor (USDOL) has established common measures of performance for each of the three
programs.1 For the two adult programs, the measures focus on employment outcomes—the
entered employment rate, employment retention rate, and earnings levels. For the Youth
program, the measures relate to educational attainment—placement in employment or education, attainment of a degree or certificate, and literacy and numeracy gains. WIA is a partnership among the federal and state governments and local entities, and these performance measures are common across all three levels. Targets for these measures are set annually for each entity. USDOL sets national targets when submitting the President submits the annual budget to
Congress. USDOL negotiates with each state in setting its performance targets, and the states set targets for each of their LWIBs. Targets may vary from year to year and across states and
LWIBs. Failure to meet the performance targets may result in financial sanctions.
The WIA system of performance measurement is similar in many ways to the system of its predecessor, the Job Training Partnership Act (JTPA). Both used employment and educational outcome measures, and both set targets annually. The primary difference was the way in which targets were set. Both JTPA and WIA were required to take into account the effect of differences in labor market conditions and personal characteristics in achieving outcomes. JTPA adopted a regression-adjusted approach to account for these factors, whereas WIA the left this adjustment to the negotiation process.
The performance measurement systems of both programs have received considerable scrutiny from researchers. In particular, James Heckman and his graduate students at the time the research was conducted initiated an extensive study of how performance standards and
incentives influence the behavior of program administration and staff and contribute to program
outcomes or unintended consequences.2 They found three main lessons from their body of
research.
First, agencies respond to incentives. The concern among policy makers and
administrators with the performance system was that the financial incentives were so low that administrators disregarded them and thus the performance measures have no effect on
performance. The researchers find, however, that “low-powered cash incentives may, in fact, be
1
Training and Employment Guidance Letter (TEGL) 17-05, issued February 17, 2006. 2
James J. Heckman, Carolyn J. Heinrich, Pascal Courty, Gerald Marschke, and Jeffrey Smith, Editors; The Performance of Performance Standards, Kalamazoo, MI: W.E. Upjohn Institute for Employment Research, 2011.
high-powered because of the value of the budgetary awards in establishing the reputation of bureaucrats and the recognition that comes with them” (Heckman et. al. 2011, 306). However, they caution that bureaucrats may learn over time the weaknesses of the system and how the weaknesses can be exploited to their advantage. They recommend that the incentive system and performance measures be reviewed regularly and redesigned when deemed necessary to achieve the desired outcomes.
Another dimension of incentives, which complement the overall performance measures, is performance based contracting. Under both JTPA and WIA, local entities (Service Delivery Areas under JTPA and LWIAs under WIA) subcontract with local organizations, such as community colleges and nonprofits, to provide services. Many of these contracts are
performance based. Heinrich (2000), in a detailed study on an Illinois Service Delivery Area under JTPA, found that that inclusion of performance incentives in service contracts has a very strong positive effect on realized wages and employment at termination and up to four quarters after leaving the program. Based on this result and that of others (Dickinson 1988; Spaulding 2001), one can conclude that performance-based contracts yield higher performance on the
rewarded dimension.3 As previously mentioned, one has to ensure that incentives are properly
aligned with desired outcomes.
Second, current program outcomes do not predict long-run outcomes. This conclusion came primarily from studies of the JTPA system. JTPA, unlike WIA to date, was subject to a random assignment evaluation, which allowed researchers to examine the long-run outcomes of the program and compare these to the performance measures. The researchers concluded that the performance measures to not promote the key objective of JTPA of long-run gains in earnings and employment for participants. They found no correlation between the short-run outcomes from the performance system and the long-run outcomes from the evaluation. This finding raises the broader issue of whether these short-run performance measures are related to the value added of the program, regardless of whether the objective of the program is long-run or short-run outcomes. They attribute some of the disconnect between performance measures and value added to the dynamics of performance measurement system as bureaucrats learn more about the incentive system.
Third, the cream-skimming problem is overstated. There has been serious concern that local administrators of the workforce system were “gaming” the system by enrolling into their programs participants with high abilities to find employment without the services provided by the program, at the expenses of those who truly needed assistance. The researchers found little evidence that this had occurred in the JTPA programs. One possibility for the low incidence of cream skimming could be the methodology used to adjust for factors that lead to such behavior. As previously mentioned, JTPA used a regression-adjusting approach to adjust targets for factors such as the personal characteristics of participants that affect their ability to find employment.
3Dickinson, Katherine, Richard West, Deborah Kogan, David Drury, Marlene Franks, Laura Schlichtmann, and Mary Vencill. 1988. Evaluation of the Effects of JTPA Performance Standards on Clients, Services and Costs. National Commission for Employment Policy Research Report 88-16. Washington, DC: National Commission for Employment Policy.
Spaulding, Shayne. 2001. Performance-Based Contracting under the Job Training Partnership Act. Master’s Thesis, John’s Hopkins University, Baltimore, MD.
By adjusting targets upward when a local program has a higher percentage of participants with more favorable characteristics, the performance standards are raised for those trying to game the system by enrolling those who are more likely to find employment because of their own
capabilities. The reliance of WIA performance measures on negotiations to adjust for outside factors rather than a quantifiable and transparent system adopted by JTPA led adjustment led Barnow and Smith (2004) to conclude that WIA took a step backward by taking the performance
system farther away from evidence than it was under JTPA.4 The more accurately the
performance system can be adjusted for such factors, the closer the system moves toward an indicator of the value added of the program. The advantage of adopting such an adjustment methodology for WIA will be discussed under the section of workforce administrators.
Providing Information to Customers
The paper now turns to describing the information customers need to make more informed decisions and consequently for the workforce system to be more effective. Under each section, various programs, initiatives, or pilots that could enhance this information are described. Job Seekers
In order for job seekers to find jobs for which their qualifications match the job requirements posted by businesses, they need information about job prospects, assessments of the qualifications required by businesses, and the efficacy of services that could help them qualify for job postings. For those who require training to improve their skills, job seekers also need to know what skills will be in demand when they complete their training and where to receive their training. This information is not always readily available, particularly for those without much experience searching for jobs and for those with little education or skills. One Stop Career Centers offer some information. They provide job postings, projections of demand for various occupations and the likely skills required to qualify for such jobs. However, this information is limited and in many cases out of date. Businesses are not required to post job openings with the State employment service or the One-Stop Career Centers, so the job postings include only a subset, and in most states, a small subset of the job openings that actually exist at any one time. Demand forecasts of occupations and skills are dated, and do not accurately reflect the current and future demand for occupations and skills.
Value of Information about Training Outcomes
Job seekers need to know which training providers are most effective. This is
particularly important for those receiving training through Individual Training Accounts (ITAs). Job seekers receiving training vouchers are free to use them at most providers. However, to make effective choices of training providers and of the type of training to pursue, job seekers need to know the employment outcomes of those who have completed the training. The federal
government requires training providers who receive WIA funding to post specific outcome information, such as the percentage graduating from the program and more importantly the
4
Burt S. Barnow and Jeffrey A. Smith, “Performance Management Systems of U.S. Job Training Programs; Lessons from the Job Training Partnership Act,” Public Finance and Management, 4(3), 247–87, 2004.
percentage finding employment and completing the training. At present, some states use UI wage records to provide employment outcomes for training graduates.
A recent follow-up study of the random assignment evaluation of Individual Training Accounts (ITAs) conducted by Mathematica suggests the importance of targeted information in making training decisions. Findings infer that customers and society would benefit markedly from intensive counseling, along with higher potential ITA awards, compared with less information and direction from counselors and fixed awards. Estimates from the benefit-cost analysis indicate that society would benefit by about $46,600 per ITA customer by switching to what is referred to as a Structured Choice Model, while the benefit for customers would be about $41,000. The government also benefits from this switch, by about $5,000, because increased taxes more than offset the higher costs of larger ITA awards and somewhat more intensive
counseling.5 However, the study states that the evaluation design was unable to completely
disentangle the effects of more intensive counselor guidance from more customized and generous ITA awards. The evaluators concluded that it is impossible from their study to
determine whether the increased benefits would have occurred without the intensive counseling. The Structured Choice approach, whether due to more intensive counseling or more generous awards is associated with training that provides skills better matched to the jobs available in the chosen occupation. Results show that Structured Choice customers were significantly more likely than Guided Choice customers to have been employed in the occupation for which they trained.
Workforce Data Quality Initiative
States are beginning to develop data systems that can form the foundation for a system that provides such information to job seekers. The Workforce Data Quality Initiative, a federally funded collaboration between the U.S. Departments of Education and Labor, is a competitive program that provides funds for states to pull together educational, workforce administrative data, and UI wage records to provide a longitudinal history of each workers education and employment history. The initiative provides the information necessary to compute the
completion and employment rates of training providers. It also provides the basis for tracking the continuum of educational and employment services, and workforce outcomes. However, this initiative is still in the development stage with 26 states participating in rounds one and two. According to USDOL, high quality and consistent data that is available from service providers about services offered, and how well their customers benefited as they enter or re-enter the labor
market, are integral to informed consumer choices.6 At this time, states are expected to use this
data analysis to create materials on state workforce performance to share with workforce system stakeholders and the public. As demonstrated with two pilots mentioned below, a greater potential benefit is to incorporate this data in systems that bring individual participant to customers and front-line staff.
5Irma Perez-Johnson, Quinn Moore, Robert Santillano, Improving the Effectiveness of Individual Training Accounts:
Long-Term Findings from an Experimental Evaluation of Three Service Delivery Models, Final Report, Mathematica, October 2011.
Frontline Decision Support System
A pilot project funded by the U.S. Department and Labor and developed by the Upjohn Institute offers some guidance for how such a system would look. Referred to as the Frontline Decision Support System (FDSS), its purpose was to better information customers’ job search efforts and to improve the effectiveness of employment services in getting people into jobs. FDSS offered a set of tools that provided jobseekers and One-stop Career Center staff with customized information about employment prospects and the effectiveness of services. The pilot was implemented in Georgia, and was a joint effort of the Employment and Training
Administration of the U.S. Department of Labor, the Georgia Department of Labor, and the
Upjohn Institute.7
Using dislocated workers as an example, FDSS offers a systematic sequence of steps they can use to move through the reemployment process, beginning with understanding their likelihood of returning to work in the same industry, proceeding to exploring job prospects in occupations that require similar skills and aptitudes, then to accessing information about the earnings and growth of jobs in particular occupations within their local labor market, and ending with an understanding of which reemployment and training services work best for them, if none of the previous steps leads to a job. The tools are based on statistical relationships between a customer’s employment outcomes, personal characteristics, and other factors that may affect his or her outcomes, all of which are available from administrative files already collected by the various agencies. The statistical algorithms provide an evidence-based approach to determining which services are most effective for specific individuals.
By using administrative data that captures the experience of all customers who have participated recently in the state’s workforce system, this evidence-based approach offers a more comprehensive “collective” experience of what works and what doesn’t than relying on the narrower experience of individual caseworkers. In addition, FDSS incorporates local labor market information and data about job requirements and available openings, so that most information pertinent to a person’s job search is available in a comprehensive and systematic framework. Implementation of such a system also helps to develop a culture of management by evidence within the workforce development system. The system can be enhanced by recently developed internet search engines that can “spider” the internet in search for job postings and job seeker resumes. Several companies have developed sophisticated approaches of gathering this information with minimal duplication of job postings. The result is a comprehensive database of “real-time” job opportunities, which can be used by job seekers to determine the demand for jobs with specific skill requirements. One-Stop Career Center administrators can use this information to identify the demand for various occupations and for various skill sets. This information is helpful in guiding participants into the appropriate training programs.
Barnow and Smith (2004), in a critique of the performance management system in the federal workforce system, recommend using FDSS as the center-piece for a redesign of the
7
Randall W. Eberts, Christopher J. O’Leary, Kelly J. DeRango, “A Frontline Decision Support System for One-Stop Centers,” in Randall W. Eberts, Christopher J. O’Leary, and Stephen A. Wandner (eds.) Targeting Employment Services, W.E. Upjohn Institute for Employment Research, 2002, 337–380.
performance system. In what they describe as an “ideal” performance system, “randomization would be directly incorporated in the normal operations of the WIA program…through a system similar in spirit to the Frontline Decision Support System” (p. 276). They contend that such randomization need not exclude persons from any intensive services, but only assign a modest fraction to low-intensity services, i.e., the core services under WIA. The randomization would then be used, in conjunction with outcome data already collected, to produce experimental impact estimates that would serve as the performance measures. However, one of the drawbacks with randomization is sample size. A relatively large sample, typically larger than the inflow of participants into local workforce programs, would be required. Because of the need for large samples, this approach would be most applicable for state-level performance incentives, which is not the level at which contracts are administered and services delivered. Furthermore, for
purposes of informing management decisions, the effect of individual or bundle of services is more useful than the overall effect of the program. To use randomization to estimate service-level effects would require even larger sample sizes.
Another approach of estimating the effects of programs and services is to construct a counterfactual by using propensity scoring techniques. While considered not as reliable in estimating net impacts as randomization, it is considered a viable alternative and has been used extensively in program evaluations, most recently in evaluating the net impact of WIA programs
(Hollenbeck 2009; Heinrich et al. 2009).8 It has several advantages over randomization. One is
the need for a smaller sample size. The second is that one does not need to exclude participants from any services. With randomization, a control group must be constructed by randomly excluding individuals from services. With propensity scoring, the control group is constructed from observationally similar individuals who were not enrolled in any services. This is the major difference between randomization and propensity scoring and one of the latter’s drawbacks because individuals may not have enrolled for reasons that are unobserved and thus could bias the net impact estimates. However, by finding individuals who are similar in observed
characteristics helps to control for these unobserved attributes. The studies mentioned earlier have used as control group members those who participate in the Wagner-Peyser Employment Service. Another advantage is that propensity scoring methodologies can be “built in” to a performance system and can be refreshed periodically as new data are entered into the system. While not completing automatic and self-functioning, it does require a minimal amount of intervention during the updating phases.
Worker Profiling and Reemployment Services
The development and implementation of FDSS was based in part on the success of two prior U.S. Department of Labor initiatives. The first was a national program, signed into law in 1994, that required each state to identify UI claimants who are most likely to exhaust their Unemployment Insurance (UI) benefits before finding employment and refer them as quickly as possible into reemployment programs. The purpose of the Worker Profiling and Reemployment services (WPRS) system is to encourage UI beneficiaries to use reemployment services
intensively at the beginning of their unemployment spell rather than toward the end, when they face the prospect of their benefits being cut off. The identification procedure uses statistical
8
Peter Mueser, Carolyn Heinrich,Kenneth Troske, Workforce Investment Act Non-Experimental Net Impact Evaluation, U.S. Department of Labor, Final Report, ETAOP 2009-10, 2009.
analyses, similar to some of the algorithms used in FDSS. Independent evaluations show that
WPRS reduces the use of UI benefits.9
Targeting Services to Welfare-to-Work Participants
The second initiative is similar to a hybrid approach, incorporating a randomization element and a propensity scoring methodology into a performance management system. In 2002, the U.S. Department of Labor funded a pilot project developed and implemented by the Upjohn Institute that used a statistical means to refer welfare recipients to service providers. The idea was to target services that best meet the needs of customers. In that pilot, welfare-to-work participants were referred to one of three service providers based on a statistical algorithm that used administrative data to determine which provider offered services that were shown to be most effective for customers with specific characteristics and employment backgrounds. Each provider offered different services and different approaches to delivering services. A random assignment evaluation of the pilot showed that targeting services in this way increased the
90-day employment retention rate of participants yielding a benefit-cost ratio of over 3.10
Neither the welfare-to-work pilot nor FDSS are currently in operation. However, their original concepts and approaches are still embodied in some of the principles adopted by the U.S. Department of Labor. For instance, in using funding under the American Recovery and Reinvestment Act, the U.S. Department of labor has encouraged states and local workforce areas to:
• Target the use of funds on services that most efficiently and effectively assist dislocated
workers;
• Integrate the implementation of Dislocated Worker services with reemployment services
and UI programs;
• Integrate data-driven counseling and assessment into service strategies;
• Provide easy and seamless access to all programs regardless of their point of entry.11
Some of these suggestions enhance existing systems, such as targeting UI claimants who are likely to exhaust their benefits and directing them to services, and encourages the
development of more integrated information systems, such as FDSS. Businesses
Businesses are also customers of the workforce system. Increasingly businesses are complaining they cannot find qualified workers. This mismatch may stem from a variety of reasons: lack of specific occupational skills, low wage offers, lack of soft skills, and more
9Katherine P. Dickinson, Paul T. Decker, and Suzanne D. Kruetzer, “Evaluation of WPRS Systems,” in Randall W. Eberts, Christopher J. O’Leary, and Stephen A. Wandner (eds.) Targeting Employment Services, W.E. Upjohn Institute for Employment Research, 2002, 61–82.
10Randall W. Eberts, “Using Statistical Assessment Tools to Target Services to Work First Participants,” in Randall W. Eberts, Christopher J. O’Leary, and Stephen A. Wandner (eds.) Targeting Employment Services, W.E. Upjohn Institute for Employment Research, 2002, 221–244.
11
U.S. Department of Labor, Employment and Training Administration, Training and Employment Guidance Letter No. 13-08, March 6, 2009, p. 2.
stringent hiring requirements. Businesses are looking to the workforce system to help identify, assess, and train workers to meet their specific requirements. Many local workforce investment areas are working closely with businesses to identify needs and assess the qualifications of prospective workers. Businesses and local workforce investment areas are partnering with community colleges to develop curricula and train workers to meet their specific needs.
However, such planning requires “real-time” information and future projections of the demand by businesses for specific skill sets. Traditional methods of gathering this information through surveys yields outdated information. State and local workforce agencies are starting to turn to other means of collecting and assessing information on the skills required by businesses. Real-time Labor Demand Information
An alternative approach is the same “spidering” method as described above. While not a statistically valid survey, web-based information is timely and comprehensive in that all job postings on the internet can be searched and compiled. The information can also be reported by individual businesses and broken out into highly detailed occupational categories. The detailed occupational categories can be identified with specific skill sets, which in turn can inform community colleges and local workforce investment boards about the skills gap and the specific skills that training programs need to provide. Information on the flow of participants into training programs and those graduating from those programs can offer businesses important insights into the availability of workers with the skills qualifications businesses need. Information on the skill needs of businesses, the training providers that offer such training, and the success of recent graduates helps workers determine the skills and occupations that are in greatest demand. Therefore, completing the loop between the demand for and supply of workers with specific skills and incorporating this information into a system easily accessing by both employers and prospective workers can help improve the match between employers and employees. Regional Sector Alliances
Another approach is to work directly with employers. These initiatives typically focus on a handful of local sectors, which have been identified as important to the future growth of the region. Workforce agencies partner with economic development agencies, community colleges, and businesses to identify the workforce needs of businesses within these sectors, specifically the skills required currently and into the future. Training providers then work closely with their partners, particularly businesses, to develop curricula that will offer the skill needs projected by businesses in the near term. Workforce agencies refer participants of their programs into the training programs that will prepare them to qualify for future job openings.
This demand-driven, sectoral approach has been embraced by many local areas around the country. This approach is seen as a more direct way of gathering information about business needs, with the added bonus of directly involving them in the training process. Some local workforce agencies see business as their primary customer and focus on recruiting and screening job seekers to meet the specific needs of local businesses. The spidering approach complements the sectoral approach by offering a more comprehensive data base than can typically be obtained through direct contacts with employers, especially within larger regions. However, such
information has limited value unless the information gathered from direct interaction among the various partners is captured within a system that all can access.
Several states have initiated programs that engage businesses and form partnerships through this sectoral approach, including the Michigan Regional Skills Alliance and the
California Regional Workforce Preparation and Economic Development Act.12 The U.S.
Department of Labor followed their lead and funded WIRED (Workforce Innovation in Regional Economic Development). It supported the development of a regional, integrated approach to workforce and economic development and education. The ultimate goal of WIRED was to expand employment and advancement opportunities for workers and catalyze the creation of high-skill and high-wage opportunities. WIRED consisted of three generations of regional
collaborations, totally 39 regions.13
Providing Information to Workforce Administrators
The workforce performance measures are intended to hold the system accountable for outcomes and incentivize administrators and providers to continuously improve the system. For performance measures and targets to incentivize continuous improvement, inform policy decisions, and hold the workforce system accountable, the system must be transparent and objective and the measures must focus on the effectiveness of the services and hold
administrators accountable for factors they can control. In addition, incentives need to be of sufficient size to affect the behavior of administrators and must be aligned with desired outcomes.
The current system does not meet these requirements. First, while the current
performance measures focus on getting people into a decent paying job, which is the goal of the workforce system according to ETA’s strategic plan and the desired outcome of the participant, the measures do not reflect the effectiveness, or value-added, of the workforce system. Without understanding the contribution of the services provided, it is impossible to use these measures to guide management decisions.
Statistical Approaches to Adjusting Performance Measures
Many factors lead to a person’s employment. Evidence shows that factors “outside the control” of administrators generally influence employment more than the re-employment services offered by the workforce system. The broader set of factors includes a person’s past employment history, personal attributes, and education, to name a few, and also job opportunities in the local labor market. The common measures, as currently constructed and reported, do not provide much, if any, information on the contribution of the workforce system. Furthermore, they can invoke perverse incentives. Essentially allowing administrators to take credit (or blame)
12
Eberts, Randall, and Kevin Hollenbeck. 2009. "Michigan Regional Skills Alliances: A Statewide Initiative to Address Local Workforce Needs." In Designing Local Skills Strategies, edited by Francesca Froy, Sylvain Giguère, and Andrea-Rosaline Hofer. Paris: OECD, pp. 129–153.
13Nurturing America's Growth in the Global Marketplace Through Talent Development: an Interim Report on the Evaluation of Generations II and III of WIRED, Nancy Hewat, Kevin Hollenbeck; submitted by Public Policy Associates and Upjohn Institute; issued as U.S. Dept. of Labor Employment and Training Administration. ETA Occasional Paper 2009-19 (2009).
for factors outside their control opens the possibility for “cream skimming” and other ways of gaming the system. The ETA has recognized the need to adjust performance targets but has been inconsistent in its approach. Over the past 30 years, ETA has fluctuated from using a rigorous, statistical, evidence-based approach to make the appropriate adjustments to relying on
negotiations between the federal and state governments and most recently to depending upon a hybrid approach.
Recently, the U.S. Department of Labor established a target-setting procedure that is objective, transparent, and more reflective of the value-added of the workforce system. Beginning with PY2009, USDOL/ETA adopted a regression-adjusted approach for setting national targets for these three programs and other federal workforce development programs. Prior to that time, national targets were based on past performance and the desire to encourage continuous improvement in the workforce programs. The continuous improvement approach typically increased target levels from year to year without a systematic way of accounting for changes in economic conditions or the ability to meet previous targets. Furthermore, the use of negotiations to set targets at the state and local levels did not always provide an objective and transparent approach. The onset of the recent recession drew into question this practice, and the Department and OMB sought to develop a target-setting methodology that would take into account changes in labor market conditions on program outcomes. USDOL/ETA extended this procedure of setting targets to the state level as a two-year pilot, but did not continue the practice afterward.
The methodology adopted by USDOL/ETA addresses the requirements of stated above in the following ways. First, to meet the requirement that the performance measurement system reflects, to the extent possible, the value-added of the workforce system, the methodology in essence subtracts factors outside the control of local administrators from the performance targets. These outside factors have little to do with the contribution of the workforce system to
participant outcomes. They include local economic conditions and the personal characteristics of customers. By purging the performance targets of these factors, the targets are more reflective of
the contribution of the workforce system.14
Second, to meet the requirement that the performance system is objective, quantifiable and measurable, the methodology uses a common data set and statistical means to determine the importance (as quantified in weights) of each customer characteristic and of the unemployment rates on performance outcomes. The standard data set is the WIASRD (Workforce Investment Act Standard Reporting Database), which is the compilation of the administrative data of the WIA workforce programs and UI wage records. It is a longitudinal file of each individual participant that links together their prior work history, WIA services received, and post-service employment outcomes. It is similar to the data compiled for FDSS, as previously described. The methodology also includes the local economic conditions of states and LWIAs as measured by local unemployment rates, which are also obtained from an objective data source, in this case the Bureau of Labor Statistics. The weights associated with each factor are determined by using regression analysis, which estimates the effect of each personal characteristic and the unemployment rates on the performance outcomes. Using common databases and standard
14
Randall W. Eberts and Wei-Jang Huang, “Description of the Methodology for Setting State and WIB PY2011 WIA Performance Targets,” Final Report to the U.S. Department of Labor, ETA, November 2011.
statistical techniques, such as regression analysis, allows for a transparent and objective system, which can be easily scrutinized.
Value-Added Performance Improvement System
Even with the regression-adjusted approach to setting targets, the information is not current enough to be an effective management tool. Because of the use of administrative data, particularly the UI wage records, information regarding participant employment outcomes are not available for at least a year after they exit the program. In many cases, these outcomes may be reflective of individuals who entered the workforce program at least two years prior to when their employment outcomes are available. The long lag makes it difficult for administrators to base management decisions on these measures or to use them for continuous improvement.
One state attempted to address this issue by using regression techniques to forecast likely employment outcomes of participants based on the outcomes of past participants. The State of Michigan, with technical assistance from the Upjohn Institute, developed the Value-Added Performance Improvement System (VAPIS) to aid local workforce area administrators in making better management decisions. The system was similar to the regression-adjusted targets, described previously, except that instead of adjusting the targets the methodology adjusted the common measures for factors outside the control of administrators. In this way, the performance measures themselves reflected to a greater extent the value added of the workforce system. In addition, VAPIS forecasted the possible outcomes of participants currently receiving services, so that local administrators could get some idea of how their current decisions may affect future
outcomes. Michigan provided VAPIS to local workforce administrators for several years.15
Measures of Business Satisfaction and Needs
As previously mentioned, many state and local workforce entities are including businesses as their customer. However, the common measures currently adopted by
USDOL/ETA do not include any direct measure to reflect their use of the system or how they may benefit from using the system. Obviously, the fact that a workforce system participant found a job is beneficial to an employer, because they found it advantageous to hire that person.
However, the current common measures do not record whether an employer used the workforce system to find workers, whether the workers they hired came through the workforce system, or whether they retained the worker after so many months of being initially employed. Many of these measures can be added to the performance measurement system by more fully utilizing the data available from UI wage records. Others may need to be derived from surveys.
The Commonwealth of Virginia and the State of Washington have looked at alternative and additional performance indicators. Of particularly interest is a measure they have constructed
15
Timothy Bartik, Randall W. Eberts, and Wei-Jang Huang, “Recent Advances in Performance Measurement of Federal Workforce Programs,” in Douglas J. Besharov and Phoebe H. Cottingham (eds.) The Workforce Investment Act: Implementation Experiences and Evaluation Findings, W. E. Upjohn Institute for Employment Research 2011, 233–276.
Timothy J. Bartik, Randall W. Eberts, and Kenneth J. Kline, “Estimating a Performance Standards Adjustment Model for Workforce Programs that Provides Timely Feedback and Uses Data from Only One State,” W. E. Upjohn Institute Working Paper 09-144, 2009.
to record the use by employers of WIA services. It is a measure of repeat employer customers and is calculated as the percentage of employers served by WIA who return to the same program
for service within one year.16 More specifically, an employer was categorized as satisfied if they
hired someone who had exited from a program in the first quarter of the fiscal year, and then hired another individual from the program before the fiscal year was over. The denominator for this indicator was the number of employers who hired someone in the first quarter of the fiscal year. For example, Hollenbeck and Huang (2008) calculated the measure for the WIA adult programs in Virginia and found 52 percent of employers who hired someone from one of the two programs hired at least one more worker from the same program within the year. Of course, this is contingent upon the number of times an employer hires during the year, but it can be
normalized by a state or industry average.
Currently, employers are concerned with finding qualified workers to fill their job openings. Business satisfaction with the WIA programs would depend upon the programs’ ability to place individuals with appropriate skills with employers. The measure adopted by Virginia assumes that employers are repeat customers because the programs have provided them with applicants with the appropriate skills. As mentioned in the previous section on business as customers, the question for workforce administrators is how does this measure help them guide participants into the skills that are in demand. Moreover, how does this measure help training providers determine the appropriate curriculum and the appropriate capacity to meet employers’ demands? A more precise measure would actually measure skills needs in the openings posted by employers. A measure of this sort would be more easily attainable if employers posted their openings with the workforce programs, more precisely the employment service. Short of this, the growing use of the internet to post openings offers another solution. Organizations such as the Conference Board and private companies are offering services that search the internet for job postings. These services can be customized for specific locations and can glean from the job postings requirements related to educational attainment, certifications, experience, and other qualifications. Furthermore, this information is offered in real time, and even on a daily basis. Using past data, trends can be detected that given training providers a better sense of the future demand for various skills sets. This information than then be used to have a more meaningful conversation with employers as to their future needs and training providers as to their future capacity to train individuals to meet these needs.
Summary: An Intelligent Integrated Information System
As outlined in this paper, customers of the workforce system demand more relevant and current information. Jobseekers ask for information that will help them identify the occupations and skills demanded by businesses, find jobs and move into more meaningful careers; businesses are looking for information about the pool of qualified workers. Workforce program administrators seek information to help them make better management decisions. Such information is readily available for the most part, but is underutilized and not presented in a meaningful way. The data source is the administrative data generated directly by the programs, particularly the three Workforce Investment Act programs. These programs, combined with UI
16
Kevin Hollenbeck and Wei-Jang Huang, “Workforce Program Performance Indicators for the Commonwealth of Virginia,” Upjohn Institute Technical Report No. 08-024, 2008.
wage records, generate information on the services offered, employment outcomes before and after entering the workforce program, and demographics of the participants. Currently, this information is used primarily to report outcomes to higher levels of government. Little of this information is available to customers to help job seekers find jobs and businesses find more qualified workers. Furthermore, even the performance measures are of little value in
incentivizing continuous improvement in workforce programs or in holding them accountable for various outcomes. Bits and pieces of an integrated information system have been developed over the past decade or so, but the workforce system is yet to see these components put together in an integrated manner.
The U.S. Department of Labor has recently encouraged states to target services, integrate data-driven counseling and assessments into service strategies, more fully integrate programs, and provide easy and seamless access to all programs. As a performance management system , the Frontline Decision Support System comes the closest. It integrates administrative workforce data with education and wage data, it develops statistical algorithms that can help understand trends and circumstances for individuals with specific characteristics, and it brings this information back down to the customers and front-line staff who are making decisions. As Barnow and Smith (2004) suggest, this framework can be combined with counterfactuals that provide a better sense of the value added of the program and more specifically the services provided within these programs. Such a system is not perfect, of course. It does not substitute for rigorous evaluations of the effectiveness of programs nor does it guarantee that incentives are properly aligned with desired outcomes. It does make significant advances in getting relevant information in an easily accessible format to the customers and decision makers of the workforce system.
One need not look very far from the workforce system to see how an integrated information system can be developed and used. The healthcare sector has embraced such a system through the electronic medical records initiative. Hospital systems and the federal government are spending billions of dollars to integrate the medical records of patients in order to promote more comprehensive and integrated healthcare and to encourage evidence-driven healthcare delivery. On a much smaller scale, the federal government has provided funding, through the Workforce Data Quality Initiative, to develop the fundamental platform for a similar information system in the workforce and educational areas. As with the development and
implementation of electronic medical records, pursuing such an initiative within the workforce system requires a clear vision and strong leadership. Based on the development and evaluation of components of such a system, the result promises to lead to a more effective and cost-efficient workforce system.
