poses, such as making
resource allocation, budget,
and policy decisions, users
become very concerned about
the accuracy and reliability
of that data.
R E C O M M E N D A T I O N : The executive branch, and each executive branch agency, should estab- lish its own quality control process for performance information. If a state has internal auditors, these offices might be used for this purpose. However, the executive branch should place the major responsi- bility for quality control on the agency and its originating programs.
In Florida, each agency has an inspector general (IG) appointed by the department head. The IGs have been asked to examine each performance indicator to assess how the indicator was developed, how data were collected, and the data’s validity, reliability, and appro- priateness. The IG also assesses data processing and calculations to see if they were done correctly. In addition, the legislature’s OPPAGA reviews the performance indicators and is influential in their selection, and can conduct its own validation of agency performance measures.
R E C O M M E N D A T I O N : Each year, the state audit/legislature audit office should examine a sam-
ple of data collection procedures. This will help motivate agencies to focus on data quality. The legisla- ture should establish a process in which an organization external to the executive branch, such as a state audit office or equivalent, annually examines a sample of key performance indicators from the state agencies to certify that the data and data gathering processes appear to provide reasonably accurate outcome information. The focus of data quality reviews should be on outcome indicators, rather than on output indicators. The reviewing organization should follow up to make sure the agency makes the needed corrections.
At least a small sample of outcome indicators from each major state agency probably should be examined annually. Smaller state agencies might be surveyed on a random, rotating basis. An annual review of a sample of indicators is also applicable to states with a two-year budget cycle.
Texas has instituted such a process. The Legislative Budget Board annually recom- mends agencies and indicators for audit to the State Auditor’s Office (SAO). The SAO selects the agencies to examine based on the amount of money appropriated, indications from previous audits of agencies with potential data quality problems, and the past frequency with which it has reviewed the agency’s performance indicators. The SAO examines a combination of key outcome, output, and efficiency indicators.
The SAO determines whether each indicator should be classified as “certified” (if reported performance is accurate within plus or minus 5 percent and it appears that controls are in place); “certified with qualification” (when performance data appear accurate but the controls over data collection and reporting were not adequate); “inaccurate” (when the actual performance was not within 5 percent of reported performance); or “factors prevented certification” (if documentation was unavailable and controls were not adequate to ensure accuracy).22
22 See the Texas State Auditor’s Office, Legislative Budget Board, and Governor’s Office of Budget and Planning, 1999.
Having such a process for reviewing a random sample of key indicators each year has the advantage of putting agencies on notice that their indicators will periodically be exam- ined for accuracy. For example, the Texas Department of Transportation finance director made the following statement in a 1997 memo to division directors: “Please keep in mind that the fiscal year 1997 performance measure audit is just around the corner and no one knows which measures will be selected by the Legislative Budget Board. Please be orga- nized and prepared in the event you are audited. The fiscal year 1996 review went well because the above-mentioned divisions reported accurately and were organized and pre- pared at the time of the audit.”
Regardless of who performs them, quality control reviews should include elements such as the following: Data are reported accurately and properly, data and files are properly main- tained, the program and/or agency has a review process prior to submitting its final numbers, and definitions based on the detailed definition of each performance indicator are used consistently. Indicators that the review body does not “certify” should be reexamined the next year.
The governor’s office and legislature should expect that any deficiencies will be corrected. In Texas, legislators and legislative staff have, on occasion, asked personnel what the agency was doing to gain State Auditor’s Office certification of indicators, and they can require corrective action plans.
No audit process can adequately examine all of the many performance indicators each year, even if the audit focuses only on outcome rather than efficiency or output indicators. Outcome data are most likely to be subject to large uncertainties. Requiring outcome data to be accurate within plus or minus 10 percent of projections should be adequate for many, if not most, state decisions. (Clearly, however, plus or minus 10 percent is not adequate for financial/expenditure data.)
S E C T I O N