The most common pitfalls to performance measurement are attribution and measurement corruption and the only way to detect and avoid them is to know how they may arise. Ask enough questions to be reasonably confident that the pitfalls don’t undermine the usefulness of the measure.
Attribution
Attribution is the assertion that a connection can be made between an outcome and the actions of a government policy, program or initiative. Determining attribution for outputs is relatively straightforward as outputs are the tangible products produced through activities. Demonstrating attribution for outcomes is more complicated because a number of intervening factors, in addition to the activities, may contribute to the outcome. Creating a logic model is the strongest method for identifying the contribution of an activity to the achievement of intended results.
To test for attribution, ask the question, “Is the result we are measuring produced by our actions?” Sometimes, the answer to the above question is clear, as in the example on the right. The fact that no dead horses were dragged down main streets of Ontario in 2000 is actually a better indicator that the law is no longer relevant than it is that the law is working. Other times, the answer is less clear.
Government activities are often not solely responsible for the results achieved in relation to their objectives. There may be factors or events over which the government has no control that affect the outcome. For example, in 2003, the tourism industry in Ontario was negatively affected by the outbreak of SARS, despite ongoing government-funded programs to increase tourism in the province.
Thus, when designing performance measures, it is important to measure the contribution that government-funded activities make, or the influence they exert, rather than measuring only those things over which government has direct control. At the same time, it is important to maintain an awareness of changes that could affect the results. These changes are external risk factors and include broader economic, environmental and sector-specific trends. Clearly identifying the risk factors also qualifies the performance measure so the results reported can be interpreted correctly.
When we think of problems of attribution, we usually think about how performance measurement information is used to claim undue success. But problems of attribution can also make success appear as failure, as illustrated in the example below.
If you can’t see success, you can’t reward it
In the US, the Social Services department was responsible for assuring the welfare of children who were “wards of the state.” Though the state did not directly deliver services to children, they were
accountable, and they established a whole range of output measures to ensure children were well cared- for. For example, third-party providers of care to children had to report to the Department of Social Services on how many children received their annual medical and dental examinations, were in full attendance at school, etc. The results were rolled up into a Social Services Department performance measure of “compliance.”
Compliance rarely reached 60 per cent and the Social Services department was concerned. On further examination they learned that children might have missed a dental examination because they had the flu (and therefore could not be treated safely) or had missed school because they were in transition to adoption. In other words, low compliance by the third party delivery agent was not necessarily an indicator of poor care. In fact, sometimes the lack of compliance was an indicator of high quality service to children that went unrecognized. The Social Services department re-examined their compliance indicators in consultation with third-party service providers. Not only did they obtain higher quality information, rates of compliance increased, as did public confidence in Social Services.
The example above also raises an important area in which attribution issues arise – in situations where a broader public sector organization or agency delivers service on behalf of the government, but for which the government remains accountable for the result. Where such organizations also have a high degree of independence, ministries sometimes have reservations about their ability to demonstrate attribution and therefore accountability for the results achieved by broader public sector organizations. This is why it is essential to involve broader public sector organizations and agencies in the creation of logic models for ministry strategies to which their activities contribute. Obtaining agreement about the objectives and strategies and involving third party organizations in the identification of appropriate performance measures is critical to resolving problems of attribution.
Figure 5: Good performance measures measure the contribution of activities to government priorities Attribution Barrier
Output
3
Desired
Results
Government Priorities
Outcome
2
Ministry
Strategies
and
Activities
Output
2
Outcome
1
Factors Outside ControlOutput
1
Attribution BarrierWhat gets measured gets done
The Russian furniture industry, in an attempt to increase efficiency, adopted a pay-for-performance policy in which workers were paid a commission on every pound of furniture produced. Production rates remained stable, but Russian furniture became the heaviest furniture in the world.
If you aren’t rewarding success,
you’re probably rewarding failure
In the UK, national government provided funding to support training programs targeted to people who faced significant employment barriers and had been unemployed for more than two years. The training was provided on a four-year contractual basis by private sector companies whose contracts were renewed on the basis of performance - the number of successful graduates.
After four years, the training companies had met their targets and their funding was renewed. However, research showed that there had been no significant change in the target group. This was because the training
companies had not admitted members of the target group to the training because they were more difficult to train, less likely to complete within the contract time, and would therefore jeopardize the company’s ability to meet its targets.
Measurement illustrates the contribution that activities make, without falling into the trap of incorrectly attributing all success or failure to the activity.
Even when all appropriate steps have been taken, attribution can remain an issue. This is why providing a qualitative description and analysis of performance information is valuable. It explains the larger context and data collection methods, the limitations on the interpretation of the data, and the key risk factors so that attribution issues are clearly identified. In this way, users of information are less likely to use the performance measurement information inappropriately.
For a more detailed discussion of attribution and how to avoid attribution problems, see John Mayne’s article, “Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly,” (Office of the Auditor General of Canada, 1999), which can be found at the following internet address: http://www.oag-
bvg.gc.ca/domino/other.nsf/html/99dp1_e.html/$file/99dp1_e.pdf.
Measurement Corruption
The second pitfall of performance measurement is measurement corruption. People adapt their behaviour in relation to the thing being measured, as in the example on the right. It is important to try to anticipate any undesirable behaviours the measure could inspire and revise the measure accordingly. Another example of measurement corruption in the context of target setting is given on page 36.
The risk of measurement corruption in performance measurement is relatively high, due to our natural desire to be successful and our boundless creativity and adaptability. It is rarely intentional and is usually produced by the measurement itself.
Performance measures are less likely to become corrupt if all those involved understand the measure, what outcome it is intended to measure and agree to the choice as relevant and appropriate. This is another reason why everyone affected should be involved in the development of the logic model and identification of performance measures.
Another way to deal with the effects of measurement corruption is to ensure performance measures are part of a system of
measurement. If one measure becomes corrupt, other measures may signal that corruption by their lack of change. The example of training programs on the right illustrates this point. The lack of change in employment rates of the target population provided a clue that the measurement of training completion may have been corrupt by raising the question, “Why isn’t there change for the target group as a result of the government-funded program to improve its access to employment?”
If you can’t recognize failure
you can’t correct it
A transitional employment (TE) program was designed to place clients with mental illness – particularly those with little work history - in short-term, entry-level jobs on a rotational basis. At any given time there were 20 jobs and 300 clients. Program staff were certain that all clients were being given the opportunity to
participate. When a new
performance measurement system was established, staff were surprised to learn that most of the time the same 20 clients were being offered the TE jobs as they became available. The fast pace of the program, the heavy
demands on staff time and the lack of objective information had resulted in available jobs being offered to clients with the strongest employment record, a total contradiction of the program’s fundamental purpose. In response to the performance information, the program changed their job allocation practices to ensure that everyone on the wait list,
particularly the most disabled clients, were given access to TE jobs.
Unintended Consequences
Performance measures can also help to identify unintended consequences of implementing a policy or program. In the example to the right, staff assumed they were treating all clients equally, but performance measurement demonstrated the rapid response times of the program led to jobs being given to those clients who were already the most prepared for employment. Measuring performance will not “save the world.” Performance measures are a tool. They help us to:
• discover quickly if progress is being made toward
objectives as expected and to tell the difference between success and failure;
• make ongoing decisions about how to best use resources, when to make changes to services and identify existing gaps;
• identify when corrective action may be needed and guide what that action should be;
• most activities are on a large scale and delivered over vast regions. Performance measurement helps us to know when we are successful and to identify potential problems quickly.