A key problem for agency personnel is the perception that the legislature is holding them accountable for outcomes that are driven by outside forces over which they have little or no control.
R E C O M M E N D A T I O N : States and their agencies should consider adding to their performance
reports a rating for each outcome indicator that shows the extent to which the agency has influence over the results. For example, three such rating categories might be “substantial influence,” “some influence,” and “little influence.”
Including such ratings—although they are subjective—will alert higher-level agency officials, the legislature, and the governor’s office of the extent to which agency and program staff believe they are able to affect particular outcomes.
Agencies should be able to influence, at least somewhat, the outcomes reported for each of their outcome indicators. Agencies will almost always have considerably more influence over intermediate outcomes than end outcomes.
So far, we do not know of any state that has formally included such information as part of its outcome measurement process. These ratings of extent of influence necessarily will be subjective, initially made by program managers but preferably subject to review by higher-level agency and central executive branch review. Written explanations for the ratings would likely be needed in some cases.
An important way for an agency to reduce the likelihood of being blamed unfairly for outcomes caused by outside factors is for it to provide explanatory information with its performance reports for indicators whose values fall substantially short of expected outcomes. An example of such a procedure is the Texas analysis-of-variation reports required by the state legislature, as described above.
We found few situations in which agencies and the programs were systematically examining the outcome data, particularly deficiencies, to identify reasons contributing to the outcomes.
Another approach is that of Florida’s Department of Environmental Protection (FDEP). As described in the section on using performance information, FDEP requires that programs with deficient performance prepare corrective action plans in which the program officials discuss the problem, analyze potential causes, and identify possible solutions.
R E C O M M E N D A T I O N : State agencies should, shortly after each performance report is
prepared, focus on those outcome indicators whose level indicates substantial deficiencies compared to expectations and therefore requires a performance improvement, such as a corrective action plan within a specified time period. The agencies should monitor such plans until the performance levels have become satisfactory.
Programs with such deficiencies should consider steps such as establishing task forces to look into reasons for the deficiencies and to identify potential corrective actions.
For some program issues, more in-depth, systematically collected information will be needed. Systematic, in-depth evaluation of a program can require substantial time, effort,
and cost. The federal government sometimes provides states with funds for in-depth program evaluations. In addition, some state legislators occasionally include program eval- uation requirements for particular programs in which they are interested, such as welfare reform programs. Some states, such as Minnesota, have legislative offices that undertake such evaluations for the legislature.
We found little formal, systematic program evaluation activity in state executive branches. In-depth evaluation appears to be most common in health, transportation, and human service agencies, probably because of the availability of federal funding or federal requirements. As far as we have been able to determine, few, if any, states have central executive branch offices explicitly responsible for program evaluation (such as in a budget and planning office).
Because such in-depth evaluations can be quite costly and require considerable time to complete, some agencies undertake less elaborate evaluation efforts to find reasons for prob- lems, such as analyzing accident data on state highways in order to identify causes and how such causes might be corrected.
R E C O M M E N D A T I O N : Agencies and their programs should establish procedures for under-
taking “quick-response evaluations.” These might include such procedures as small-scale surveys of various stakeholders (such as customers and other persons affected by the program) and service provider personnel (whether state employees, local governments, or contractors) to identify service problems and reasons for them.18Focus groups are also a useful tool for this exploration.
R E C O M M E N D A T I O N : To complement the performance measurement process, state agencies,
particularly large ones, should develop an annual evaluation agenda that establishes priorities for in- depth evaluation. Such evaluations might be done internally, if resources are available, or through contracts. Evaluation candidates should be selected based on the importance of the program, when important decisions are needed, and the need for additional knowledge about program effectiveness, as well as the cost.
An example of quick-response evaluations is provided by Washington’s Department of Ecology evaluation of its motor vehicle–emission control program as it related to low- income motorists. The concern was that such owners might not be able to afford the repairs needed. A team comprising department staff, an area pollution-control authority, a county government representative, and members of the Salvation Army, with assistance from the automotive repair industry, examined the problem. The Salvation Army screened low- income motorists who applied for aid and arranged for up to $450 in repairs for eligible vehicle owners. In addition to preventing a financial burden on families, the repairs achieved significant reductions in hydrocarbon and carbon monoxide emissions.19
Many state audit offices undertake performance audits, a form of evaluation aimed at developing information on the effectiveness of programs and identifying reasons for the
18 Information on procedures and uses for surveys of citizens/customers are contained in a number of published reports, such as Miller and Kobayashi, 2000; and Hatry, Marcotte, van Houten, and Weiss, 1997.
results. These are done in addition to the traditional financial and compliance audits.20The Minnesota Legislative Auditor produces some eight to ten program performance audits annually. Florida’s Office of Program Policy Analysis and Government Accountability (OPPAGA) also does a substantial number. A number of other states, including Arizona, Kansas, Mississippi, and Oregon, have also focused major effort on performance audits.
In the past, performance audits have been hampered by the lack of outcome data avail- able from executive branch agencies, and state agencies conducting the audits have had to develop what they could with their limited resources. In the future, as state agencies increase their regular collection of outcome information, performance audits should be better- equipped to undertake outcome-based auditing.
R E C O M M E N D A T I O N : As agency outcome data become more available, legislative audit agen-
cies should make use of these data to strengthen their efforts to evaluate state programs.
Another gap in evaluation is systematic assessments of past legislative initiatives. The only examples we found were those of the Texas Criminal Justice Policy Council, which provides periodic reports to the legislature on its criminal justice initiatives. Examples include its reports on the use of progressive sanction guidelines for juveniles referred to juvenile probation agencies and on the effectiveness of rehabilitation programs for offend- ers.21These assessments should be “big picture” reviews that summarize the findings on the effectiveness of such legislative initiatives.
R E C O M M E N D A T I O N : Operating agencies should provide the state legislature with clear, con-
cise, visually attractive reports analyzing the outcomes of major legislative initiatives implemented in recent years.
20 A compendium of state audit offices and legislative offices that undertake performance-type evaluations appears in “Ensuring the Public Trust: How Program Policy Evaluation Is Serving State Legislatures,” National Conference of State Legislatures, Spring 2000.
1
G
overning-for-results contains a number of elements that are somewhat foreign to many elected and appointed state officials and other employees, both in the executive branch and in the legisla- ture. The notion of outcome-based performance measurement and the use of such information are somewhat new. While these elements have received considerable publicity in recent years, most employees need to learn and understand more about them. On-the-job training is essential, but more formal training and occasional technical assistance usually are also needed to enable personnel to implement successful governing-for-results practices.Most states implementing a performance measurement process have provided some form of centrally sponsored training efforts on performance management. Some central offices have provided limited technical assistance to their agencies. Such efforts are often sponsored by the central budget office or through a state university.
For example, North Carolina has used the North Carolina Institute of Government and the University of North Carolina at Chapel Hill to train managers in performance measurement. The Institute of Government offered 14 one-day training sessions to state manage- ment personnel. The sessions started with a performance measurement primer on concepts and then moved to material tailored to participants’ particular functions.
Texas has used the Governor’s Center for Management Development at the Lyndon B. Johnson (LBJ) School of Public Affairs for similar training. Its two-day course teaches managers statutory performance measurement requirements as well as the uses of performance data for internal management. Managers also get the opportunity to walk through their agency’s strategic plan and suggest ways to tie their own program’s plan to their agency’s strategic plan. Minnesota used its Department of Administration’s Management Analysis Division to train state personnel in performance measure- ment, offering a one-day overview of performance measurement. While participants were primarily mid-level managers and budget personnel from the executive branch, some legislative personnel have also been trained. Management Analysis Division staff were also available to give executive agencies technical assistance on perfor- mance measurement issues.
An unusual approach has been used in Louisiana, where a key legislator, the primary author of the state’s governing-for-results legislation, held training sessions with agency budget analysts. He felt this would contribute considerably to a culture change in the executive branch by showing the importance of the process to the legislature.