1. Focusing primarily on monitoring statewide aggregated results and not enough on important segments, such as on particular customer groups or on specific state oper- ating units;
2. Too little systematic seeking of explanations for unexpected results; and
3. Overly concentrating on annual reporting, whereas more frequent reporting (at least quarterly) is likely to make the outcome information considerably more timely for state operating personnel.
The following discussion explains the significance of these gaps and makes suggestions for correcting them.
1. The need to provide disaggregated data. Aggregate data showing only statewide
outcomes, while important for broad planning, are not likely to be very useful for program managers and legislators whose responsibility is often much more targeted, such as to particular governmental functions, districts, or citizen demographic groups. Providing information on the outcomes for each such segment will enable managers and elected officials to assess where progress is being achieved and where problems exist.
R E C O M M E N D A T I O N : Agencies and their programs should not only provide statewide aggre-
gated data for each performance indicator, they should also provide breakout data, such as outcomes broken out by key demographic characteristics, by location within the state (such as by county or region), and for each specific operating unit (such as each facility, park, local office, prison, state hospital, etc.).
Breakouts of the information should be considerably more useful to managers, allow- ing them to identify which client groups, locations, and operating units are doing well or poorly, so that attention can be devoted to lower-performing areas. Such information can also be used to identify underserved customer groups that may need special attention.
For example, the Florida Department of Environmental Protection reports performance information by geographical region. This has enabled the agency to pinpoint areas that require more and closer inspections. The agency generates county-level data relating to inspections for underground petroleum tanks, waste water treatment plant permits and dis- charges, and drinking water quality. Agency personnel have pointed out that until local data
are provided, citizens and the media are much less likely to be interested, since aggregate data may not directly pertain to their scope of interest.
Minnesota’s Department of Transportation breaks out the data on a number of its outcome indicators for each of its nine administration districts. The data are also broken out according to whether roads are principal, minor arterial, or collector/local roads. The district-level outcome data include safety indicators—such as crash and fatality rates—and road and bridge condition indicators. The indicators also include the efficiency indicator “maintenance cost per lane mile.” MNDOT also breaks out its statewide data on these indi- cators by whether the roads are in rural or urban areas and for four types of highway design (such as freeways, two-lane/two-way roads, etc.). MNDOT has recently begun to track the
indicator “the time between the end of a winter snow event and the time that 95 percent of the pavement is regained (i.e., is free of snow and ice).”15
2. The need to provide explanatory information along with the performance informa- tion. Performance information seldom, if ever, tells why a particular result occurred. Similar
to a sporting event score, the running score tells managers whether they are winning or los- ing and by how much. The scores do not tell why the team is winning or losing. The same is true with state performance information. The performance data identify areas that need attention and subsequently identify whether the desired results have occurred after actions were taken. However, some steps can be taken to help managers put outcome data into a bet- ter perspective.
R E C O M M E N D A T I O N : State governments should encourage their agencies to provide
explanatory information along with their performance reports. Such information is particularly needed where data indicate worse-than-anticipated outcomes. If managers are encouraged to provide explana- tory information, this is likely to reduce any fears they might have that performance information will be misused or be used incorrectly by state officials or the legislature, or be misinterpreted by the public.
For example, the Minnesota Department of Corrections tracks the number of dis- turbances within each facility and computes the “number that occur per offender” housed in each facility. A related indicator is the “cost per offender housed for a day.” Both these indi- cators can be affected by inmate density at the facility. As density increases, the cost-per- inmate-day tends to go down, but the number of incidents tends to increase. Thus, in presenting data on these two indicators—when making comparisons across facilities and when aggregating the data across all facilities, and when comparing performance over time— the corrections agency also reports on density. This helps users better understand why costs and incidents vary among facilities and why they have increased or decreased over time.
Florida’s Department of Environmental Protection found low compliance rates for underground storage tank leak detection actions, and that 75 percent of these failures to detect leaks were due to a lack of adequate leak detection equipment. Department person- nel talked to tank owners, both those in compliance and those not in compliance, and found extensive confusion about leak detector requirements. The department reviewed its publi- cations on underground storage tanks and developed improved, easily readable, compliance- assistance materials for use by both county inspectors and the regulated community.
Florida’s Department of Revenue found that its service quality indicator “response time to answering phone calls” was slower than desired. Instead of just adding operators to answer the phones, the department sought to identify why so many people were calling. This information guided efforts to improve public education, rather than adding staff to answer phones. The result was reduced numbers of calls and improved response times in answer- ing the remaining calls.
Delaware’s Department of Education monitors school district student SAT scores and tracks the “percent of students taking the SAT.” The department was concerned that the key
outcome indicator (SAT test scores) might be affected by the number of poorly performing students not taking the test. Thus, the department now reports the percentage of students taking the test in each school district. This provides potential explanatory information for comparing school districts and examining changes in SAT performance from year to year.
Texas requires agencies to provide quarterly explanations to the legislature regarding any performance indicators whose values exceed plus or minus 5 percent from the agencies’ projected values. Exhibit 8 is an excerpt from a Texas Natural Resource Conservation Commission quarterly performance indicator report. (Because the most recent value for the indicator “percent reduction in pollution per capita from permitted wastewater facilities” was considerably below the projected reduction, the agency provided an explanation. On the indicator “percent of surface waters meeting or exceeding water quality standards,” the actual value was very close to the projected value and no explanation was required.)
Texas Auditor’s office representatives report that although the public safety department had a standard of three business days for processing motor vehicle licenses, the measure- ment process revealed that they were taking an average of six days. The agency found that customers had to mail requests to Austin or drive there to get their licenses. Because of legislative caps on hiring, the agency could not hire more people to process the claims. To remedy the problem, they switched to acceptance of electronic requests from field offices.
State agencies have a number of ways to examine program results to seek answers to why levels of performance are lower than expected. Such procedures range from in-depth program evaluations that use sophisticated statistical procedures to relate results to poten- tial explanatory factors, to a number of less-demanding problem-solving strategies, such as forming teams of program personnel to interview customers and other knowledgeable persons about what happened and why. This issue is addressed in more detail in section 11 (“Analysis of the Information: The Need for Explanations”).
3. The need for frequent performance feedback. Yearly performance reports from agen-
cies and their programs to the legislature for the annual budgeting cycle are not likely to provide agency managers with timely information for making program decisions during the year. Operating managers need more frequent performance information. Pressure on program personnel to respond to annual budget preparation and reporting requirements may lead to the neglect of more frequent data collection and review.
R E C O M M E N D A T I O N : Operating managers should obtain and review outcome-based
performance reports on a regular basis throughout the year. Typically, traditional internal government progress reports have been prepared at least quarterly. More frequent reporting is needed for some indicators. Even with surveys of customers, which typically have been undertaken at most once a year, state agencies should consider spreading their surveys throughout the year. For example, 25 percent of the customers might be surveyed each quarter. Data precision for each quarter will be reduced some- what, but the timeliness of the feedback based on the survey data will be considerably improved.
For example, Texas and Iowa conduct quarterly reviews of their indicators and provide feedback on the highs and lows to management, staff, and the legislature.
It is not likely that many legislators will be interested in reviewing reports more frequently than once a year (or that they will have the time to do so). In Texas, however, quarterly and