Performance Measurement: A Reference Guide

Performance Measurement:

A Reference Guide

>

What gets measured gets done

>

If you don’t measure results you can’t

tell success from failure

>

If you can’t see success you can’t

reward it

>

If you aren’t rewarding success,

you’re probably rewarding failure

>

If you can’t see success you can’t

learn from it

>

If you can’t recognize failure you

can’t correct it

>

If you can demonstrate results you

can win public support

© Queen’s Printer for Ontario, 2004.

Please do not reproduce this document without the permission of the Queen’s Printer for Ontario

Acknowledgements

Kevin Perry, Sheila DeCuyper, Barbara Adams and Godfrey Edwards at Management Board Secretariat would like to thank all of those involved in the development of the Performance Measurement Guide. Nilam Bedi, Nancy Elliot, Sam Erry, Rob Glaister, Rian Pienaar, Sara Sandhu, Simon So, Michel Theoret, Scott Tyrer, Henric Wiegenbroeker, and Bev Wolfus formed the inter-ministerial committee who kindly read the early drafts and helped to define the structure and contents of the final product. A number of programs offered examples of their work in performance measurement for the appendices. Thanks to Geoff Steers of the OPP for the article on the PRISM system, Bohdan Wynnycky for the description of the Municipal Performance Measurement Program and Bev Wolfus for sample program logic models. Our thanks to Russ Obonsawin and Simon Trevarthan of the Ministry of Finance for ensuring consistency between the guide and Modern Controllership’s performance measurement training. A number of people took the time to provide detailed feedback on the guide as part of our consultation with CAOs across government. Thank you to Richard Clarke, Angela Faienza, David Field, Chris Giannekos, John Kerr, Robert Montgomery, Sheila McGrory, Mercedes Nolan, Anne Premi, Elizabeth Rogacki, Len Roozen, Deborah Ross, Carol Townsend, Lisa Watson and Kim White.

Thanks to central agency staff who generously contributed their expertise and creativity to make the guide interesting and align the framework offered in the Guide with the direction of government. Deputy Ministers Kathy Bouey and Carol Layton and Assistant Deputy Minister, Peggy Mooney provided direction and encouragement at each stage of development.

Special thanks to the members of the Performance Measurement and Evaluation Network without whose experience developing performance measures and systems across government the guide could not have been written. We hope that you will find the guide useful in your ongoing work to develop results-based management practices across the Ontario Public Service.

Table of Contents

Executive Summary... 4

Performance measurement is the method for demonstrating results...4

Performance Measures: Six Steps to Creating Performance Measures ...5

Logic Model ...6 Meaningful Measures...7 Performance Targets...8 Reporting Performance...8 Internal reporting...8 Public Reporting...9 Measures Checklist...9 Conclusion... 10

SECTION 1: The Big Picture ... 11 Key Terms:... 11

1.0 Context: Results-Based Management ... 12

1.1 Performance Measurement as part of Results-Based Management... 13

1.2 Performance Measurement Systems... 15

1.3 Reporting Performance Measures ... 16

1.4 Performance Measurement Terminology... 16

1.5 Summary of Key Points... 16

Checklist ... 17

SECTION 2: The Who, What and Why of Performance Measurement ... 18

Key Terms... 18

2.0 What is a Performance Measure?... 19

2.1 Why Measure Performance? ... 19

2.2 Who Uses Performance Measurement Information and Why?... 20

2.3 Levels and Types of Measurement ... 22

2.4 Summary of Key Points... 23

Checklist ... 24

Table of Contents

SECTION 3: ... 2How to Develop Performance Measures 5

Key Terms... 25

3.0 Overview ... 26

3.1 Developing Performance Measures ... 26

STEP 1: Define the strategy in relation to government priorities ... 26

STEP 2: Identify and consult on cross-ministry and horizontal initiatives... 28

STEP 3: Identify the performance measures ... 28

STEP 4: Check the measures for pitfalls ... 30

STEP 5: Establish Baselines... 34

STEP 6: Set Performance Targets ... 34

3.2 Summary of Key Points... 35

Checklist ... 36

SECTION 4: Reporting Performance Measures ... 37

Key Terms:...37

4.0 Introduction... 38

4.1 Canadian Comprehensive Auditing Foundation / La Fondation canadienne pour la vérification integrée (CCAF/FCVI) Principles for Reporting... 38

Appendix 1 ... 40 Performance Measurement Resources in the Ontario Public Service... 40

Other Resources ... 40

Appendix 2 ... 41 Glossary of Performance Measurement Terms ... 41

Introduction... 41

Definitions ... 41

References ...45

Appendix 3a ... 46

Examples of How Ontario is Using Performance Measurement ... 46

Appendix 3b ... 48

Municipal Performance Measurement Program... 48

Appendix 4 ... 53

Outputs... 53

Outcomes... 54

High-Level Indicators ... 55

Appendix 5 ... 56

Sample Logic Models... 56

Appendix 6 ... 60

Step-by-Step Instructions for Creating a Logic Model ... 60

Appendix 7 ... 64

Measures Checklist... 64

Appendix 8 ... 65

Example of How Benchmarking Can Be Used... 65

Executive Summary

This guide describes the Ontario Public Service’s approach to performance measurement and explains how performance information is used in decision-making and business planning.

Performance measurement is the method for demonstrating results.

The diagram below illustrates the relationship of performance measurement to other parts of the results-based management process. The focus is on ministry strategies for meeting government priorities or serving another important public interest. Performance measures for these strategies demonstrate the contribution that their results make to government priorities.

The results achieved at the activity level, in the form of outputs or short-term outcomes, will be used to support project plans, quarterly reporting and Management Board of Cabinet and Cabinet policy

submissions. However, the latter submissions should also be supported by evidence that demonstrates intermediate-term outcomes.

The performance measures that ministries report centrally will be a combination of: • measures related to public reporting and;

• intermediate level outcome measures that demonstrate the contributions of ministry strategies to meeting government priorities or serving other important public interests.

Figure 1: Performance Measurement is a key to results-based management

Performance Measurement: A Reference Guide – Executive Summary 4

Government Priorities

All other

Government

activities

Results

Ministry Strategies

Ministry Activities

• Public results measures and high level indicators • Intermediate-term outcome measures

• Outputs and short-term outcomes • Project Plans and

milestones

Broader Public

Sector Activities

Agency Activities

Performance Measurement: A Reference Guide – Executive Summary 5

Performance Measures: Six Steps to Creating Performance Measures

Developing performance measures is not easy. Poorly integrated performance measurement systems can be worse than no system at all and may actually support poor decision-making. There are six steps to establishing good performance measures.

Use a logic model to define the ministry's strategies in relation to

government priorities

Ministries should use logic models to illustrate relationships among activities, strategies and the results to be achieved.

Step 1

Identify and consult on cross-ministry or horizontal initiatives

Ministries should consult with third party service providers, broader public s organizations and other ministries to align performance measurement systems to promote greater coordination and avoid duplication.

ector

Step 2

Identify individual performance measures

Developing performance measures is an iterative process and it is rare to get a satisfactory product the first time. It may be helpful to research measures that have been developed for similar activities elsewhere.

Step 3

Check the measures for pitfalls

A number of pitfalls can compromise the value or usefulness of a performance measure. The most common pitfalls are attribution and measurement

corruption.

Step 4

Establish baselines

A baseline is the level of results at a given time that provides a starting point for assessing changes in performance and establishing objectives or targets for future performance.

Step 5

Set performance targets

A target is a clear and concrete statement of planned results (including outputs and outcomes) to be achieved within the time frame of parliamentary and departmental planning and reporting against results, which can be compared. Every ministry is asked to use comparative data to set targets based on its own performance, established industry standard, articulated customer preference and/or performance of a comparable organization. Reasonable targets are challenging but achievable.

Once established, measures are used to measure progress, take corrective actions and adjust targets where applicable.

Let’s look at Step 1 – “Using a logic model” in more detail, because the logic model provides the building blocks for good performance measures.

Logic Model

The logic model provides a foundation for developing performance measures that will support decision-making. A logic model is a tool that can help define strategies and activities in relation to government priorities. It clearly shows the relationships among government priorities, ministries' strategic objectives, and how ministry activities contribute to achieving those objectives and priorities through their expected outcomes. The process of creating a logic model and making the linkages among inputs, outputs and outcomes can help build common understanding of what is expected, prioritize activities and identify appropriate performance measures.

Many people will be familiar with program logic models, but they can also be used at a strategic level or created for the work of an entire organization. Everyone who uses logic models adjusts them to meet their own purposes, but a standard logic model is shown below.

The important points to remember when creating a logic model are:

• a logic model is an iterative process that involves many people working together rather than a product which one person can produce;

• the information entered into the logic model at the early stages may need to be revised as new information is entered;

• it's not just the information in the boxes that counts, but the relationships between the boxes.

Figure 2: Standard Logic Model Diagram

List the benefits or changes in

conditions, attitudes and behaviour that are expected to result from the activities in the

intermediate-term List the changes

in participation, awareness, behaviour, compliance or capacity that are expected to result from the activities in the short term List the tangible

products of the activities List all activities

List all inputs

Desired

Intermediate

Outcomes

Activities Outputs

Short-Term

Desired

Outcomes

Inputs

Objectives Of The Strategy:

What does the strategy hope to accomplish

Customers: Who Benefits

Larger Public Interest:

(Government Priorities are usually here)

Figure 3: Logic Model Process – How Each Component Leads To The Next Stage

Your Planned Work

Achieving Objectives

Activities

Outputs

Outcomes

High Level

_Changes

Inputs

A logic model will help identify all the potential measures for a strategy or activity, but we don’t want to measure everything, just those things that are meaningful for decision-making.

See Appendix 6 for step-by-step instructions for creating a logic model.

Meaningful Measures

Meaningful performance measures should subscribe to the following criteria. They should: • show how ministry activities contribute to achieving results;

• use reliable, verifiable and consistent data collection methods; • provide key information for decision-making;

• capture all areas of significant spending;

• identify and track impact as well as progress towards meeting desired outcomes; • incorporate consideration of risks and act as “thermometers” for risk management.

The Ontario government uses three levels of performance measurement:

should be developed to demonstrate the short-term progress that ministry activities make towards achieving the objectives of ministry strategies.

Output measures

(short-term and intermediate-term) should be developed to demonstrate the achievement of ministry strategies and/or the contribution of ministry strategies to meeting government priorities.

Outcome measures

measure social, environmental or economic conditions for which government alone is not accountable, but which reflect the extent to which the government's priorities are being achieved.

High-level indicators

Performance Measurement: A Reference Guide – Executive Summary 8 Identifying output measures and high-level indicators is relatively easy but identifying good outcome measures can be difficult. The Ontario government uses three types of outcome performance measures.

The extent to which a strategy is producing its planned outputs in relation to use of inputs.

Efficiency:

The extent to which a strategy is producing its planned outcomes and meeting intended objectives. At least one outcome effectiveness measure is required for each ministry strategy.

Effectiveness:

The degree to which the intended recipients or beneficiaries of a product or service indicate that the product or service meets their needs and expectations for quality and efficiency.

Customer satisfaction:

Ministries should use outcome measures of effectiveness, efficiency, and customer satisfaction wherever possible.

In addition to using performance measurement information to support internal decision-making,

ministries will be asked to report their progress to support central decision-making and public reporting.

Performance Targets

Every ministry is asked to use comparative data to set targets based on its own performance, established industry standards, articulated customer preference and/or performance of a comparable organization. Reasonable targets are challenging but achievable.

Reporting Performance

Government and ministries need to be able to report on performance so that the public and stakeholders can make informed judgements about achievements with public resources. Reporting performance internally to Ontario Public Service management and staff is also important in order to guide decision-making and support continuous improvement efforts. There are two types of reporting: internal and public.

Internal reporting

• Annual reporting on the results achieved to date;

• Quarterly reporting includes performance measurement information to show how short-term objectives are being achieved and to identify risks to achievement.

Public Reporting

• Evidence to support reports to the public

Ministries will also use performance measurement in a variety of other reporting relationships. For example, Deputy Ministers and Ministers will require ongoing performance reporting for internal ministry management and many ministries also contribute data to sector-specific federal, provincial and territorial bodies.

The 2004 Ontario Budget referred to the nine principles for public reporting developed by the Canadian Comprehensive Auditing Foundation (CCAF/FCVI) as a model for improving public reporting.

Measures Checklist

Is the measure actually a good measure of the objectives the strategy intends to achieve?

>

Relevance:

>

Validity:

Does the measure actually measure what it is supposed to?

>

Reliability:

Do different users of the same measure report the same result?

>

Verifiable:

Will the measure produce the same data if measured repeatedly?

>

Attribution:

Does the measure relate to factors that the ministry can affect?

>

Clarity:

Is the measure clearly defined and easily understood?

Does the measure provide correct information in accordance with an accepted standard?

>

Accuracy:

>

Cost Effectiveness:

Is the value of the measure greater than the data collection costs?

>

Sensitivity:

Is the measure able to measure change?

>

Timeliness:

Can data be collected and processed within a useful timeframe? Can the data be compared with either past periods or with similar activities?

>

Comparability:

Does the data feeding the measures relate to the same factors in all cases at all times?

>

Consistency:

>

Integrity:

Will the measure be interpreted to encourage appropriate behaviours?

Conclusion

In order to demonstrate results we need to measure our performance and use that performance information for planning and on-going management of government activities. Reporting performance is important to guide decision-making and support continuous improvement efforts.

For resources to support performance measurement see Appendix 1.

SECTION 1:

The Big Picture

Key Terms:

Accountability: The obligation to answer for results and the manner in which responsibilities are discharged. Accountability cannot be delegated. Responsibility is the obligation to act whereas accountability is the obligation to answer for an action.

Activity: An activity is the work performed by ministries to implement public policy and provide services to the public. All activities consume resources and produce products and/or services. One or more activities will be critical to the achievement of overall public policy objectives. Ministries must be able to demonstrate a direct causal link between the activity and the outcome(s).

Monitoring: The process of collecting and analyzing information to track program outputs and progress towards desired outcomes.

Outcome: The actual effects/impacts or results of the outputs. Outcomes provide relative information, for example: “Percentage of total cases resolved to client satisfaction within “x” weeks”.

Output: The products or services that result from activities. Outputs are often expressed as, “Number of “x’”, for example: “Total number of cases”.

Priority: Higher order goals of government that reflect its commitment to citizens and contribute to enduring human, economic, civic and environmental benefits.

Result: A condition (outcome) or product (output) that exists as a consequence of an activity. Strategy: Plan outlining how specified ministry activities and programs contribute to a government priority and results.

This guide:

• describes the Ontario Public Service’s approach to performance measurement;

• explains how performance information is used in decision-making and business planning; • provides guidance on how to use a logic model to develop performance measures; • establishes criteria for good performance measures;

• identifies pitfalls to performance measurement;

• details central performance measurement and reporting requirements.

The appendices will be of particular interest to staff with ministry leadership roles for performance measurement and the guide may be used as a support to internal ministry training.

1.0 Context: Results-Based Management

Transparency is an important part of government accountability and governments all over the world are increasingly expected to demonstrate what results they achieve with taxpayers’ money. It is not enough to know what government is spending; decision makers want to know the outcome or impact of

government decisions. The process of identifying objectives and desired outcomes, measuring progress and making decisions on the basis of results is called results-based management.

Results-based management is a comprehensive, government-wide approach that informs results-based decision-making, ensuring that all government-funded activities are aligned with strategies that

contribute to meeting government priorities or serve an important public interest. Results-based management incorporates the following principles:

• government priorities drive planning processes across all government function;

• effective accountability – within ministries and between ministries and third party service

providers and broader public sector organizations – requires that performance measures be built into programs and activities so that expectations are clearly articulated, to monitor and evaluate performance on a regular basis and take corrective action where necessary;

• horizontal integration of strategies and activities across ministries and the broader public sector helps to demonstrate how wide ranging activities complement each other in achieving

government priorities;

• demonstrable results drive the development of strategies and activities, policy and legislative agendas, investment decisions and public accountability.

Results-based management requires reliable, objective information at all levels of government. This information is gathered through performance measurement. Strong performance measures, that demonstrates objectives of government-funded services are being met, and are critical to implementing results-based management successfully.

The vehicles for implementing results-based management are:

• Ministries’ results-based plans, which are the key government documents guiding ministry decisions, and

• Performance measurement, which is how results are demonstrated.

1.1 Performance Measurement as part of Results-Based Management

Performance measurement has been used in the Ontario Public Service for over 20 years but has not always actively informed ongoing decision-making. The critical changes to performance measurement that are introduced by a results-based management approach are:

• In addition to being used by ministries internally, performance measurement is integrated into all key government decision-making processes including:

o budget setting and printed estimates;

o Management Board of Cabinet and Cabinet Policy Committee decisions; o broader public sector contract management;

o ongoing internal government management (results-based planning, quarterly reporting,

expenditure and risk management, report-backs).

• Performance measurement becomes part of a larger management process that aligns ministry strategies and activities with broad government priorities and specific results.

• Broader public sector organizations are accountable to the government for the results that they achieve. Ministries’ performance measures for contract and service management must be aligned with ministry strategies to meet government priorities and specific results.

Figure 1: Performance Measurement is a key to results-based management

Performance Measurement: A Reference Guide – Section 1: The Big Picture 13

Government Priorities

• Measures

associated with results reported to the public may be drawn from either strategies or activities

All other

Government

activities

Results

Ministry Strategies

Ministry Activities

• Intermediate-term outcome measures, reported annually in RbP

• Some may be linked to results

• Outputs and short-term outcomes to support quarterly reporting, release from hold-back, and activity-based costing

• Project Plans and milestones • Some may be

linked to results

Agency Activities

BPS Activities

Some performance measurement information will be used for both internal management and for publicly reporting on progress towards achieving government priorities.

The diagram on the previous page illustrates the relationship of performance measurement to other parts of the results-based management process. The focus is on ministry strategies for meeting government priorities or serving another important public interest. The results achieved by various government- funded activities - whether delivered directly, through contracts, agencies or the broader public service – should measure results that demonstrate the contribution they make to ministry strategies. The results achieved at the activity level, in the form of outputs or short-term outcomes, will be used to support project plans, quarterly reporting and Management Board of Cabinet and Cabinet policy submissions. However, the latter submissions should also be supported by evidence that demonstrates intermediate-term outcomes. The performance measures that ministries report centrally will be a combination of: a) measures related to public reporting and, b) intermediate level outcome measures that demonstrate the contributions of ministry activities and strategies to meeting government priorities or serving other important public interests.

Figure 4: Identifying the right performance measures means knowing how activities align with strategies

Strategy 1

Strategy 2

Strategy 3

A number of activities may be needed to meet the objectives of a strategy.

Intermediate-term outcome performance measures demonstrate that the objectives of the strategy are being met.

It is important to know what portion of an activity is contributing to a strategy

Short-term outcome and output performance measures demonstrate the progress that activities are making to meeting the objectives of the strategy. 100% Strategy 1 Strategy 1 40% Strategy 2 S S 70% Strategy 3 Strategy 3100% 30% trategy 1 100% trategy 2 60%

Activity 5

Activity 4

Activity 3

Activity 2

Activity 1

As the diagram on the previous page illustrates, it will be advantageous for ministries to develop integrated sets of performance measures that can be combined or disaggregated to serve a variety of purposes.

A ministry strategy usually will include numerous activities. A single activity may contribute to meeting the objectives of more than one ministry strategy. Knowing how and to what degree ministry activities contribute to meeting the objectives of ministry strategies is a precondition to identifying the key performance measures that demonstrate success and provide information to support decision-making. For more information on the relationship between performance measurement and results-based management, see PIL - Results-Based Planning: Further Reading

http://opspolicy.gov.on.ca/scripts/index_.asp?action=31&N_ID=5&PT_ID=14198&U_ID=32832

1.2 Performance Measurement Systems

In consultations about making the shift to a results-based management approach within government, an Ontario Public Service staff member observed that; as public servants, we have always been conscious of meeting a need and providing a service, but it’s also important to think about the impact those activities have and the results they achieve. The results based management approach helps us measure our achievements.

Results-based management focuses on outcomes, not activities and thus, a systematic approach to performance measurement is needed to demonstrate the results that government-funded activities produce. A performance measurement system is a comprehensive, integrated set of measures (of various types and levels) that provides a multi-faceted picture of the ministry’s progress toward its targeted outputs (products and/or services) and outcomes. A good system contains strong performance measures and is easy to access, update and adapt so that individual pieces of information can be grouped for different purposes. Performance measurement systems enable ministries to manage their strategies and demonstrate they are achieving their own, and government, objectives.

Rather than simply stating facts, performance measurement will be used to provide a full picture of results including a:

• description of the overall context for which performance is being assessed; • statement of what was expected to be accomplished and at what cost; • description of what was accomplished in light of the expectations; • discussion of what was learned and what will be done next; • description of what was done to assure the quality of the data1.

Ministries will need to assess existing performance measurement systems to:

• ensure existing performance measurement systems are working as intended;

• coordinate with transfer payment recipients and other ministries to ensure horizontal integration and agree on roles and responsibilities for data collection and reporting;

• identify gaps in performance measurement and create appropriate measures and systems for consistent data collection and reporting.

1 Adapted from Mayne, J. 2003. “Reporting on Outcomes: Setting Performance Expectations and Telling Performance Stories,” Ottawa: Office of the Auditor General of Canada.

1.3 Reporting Performance Measures

There are two types of reporting: internal and public. Internal reporting

• Annual reporting on the results achieved to date (primarily outcome measures, linked to ministry strategies)

• Quarterly reporting includes performance measurement information to show how short-term objectives are being achieved (primarily output measures, linked to ministry activities) Public Reporting

• Evidence to support reports to the public

1.4 Performance Measurement Terminology

There is an extensive vocabulary associated with performance measurement that is often a subject for debate. In order to ensure consistency in the way we talk about performance measures in the Ontario Public Service, Management Board Secretariat has developed a glossary of terms, some of which are found at the beginning of each section in this guide. Appendix 2 contains the full Glossary of Performance Measurement Terms.

1.5 Summary of Key Points

• Results-based planning is an integrated, government-wide approach that aims to ensure all government activities are aligned with strategies that contribute to achieving government priorities.

• Results are demonstrated through performance measurement and may be reported publicly. • A systematic approach to performance measurement needs to be developed in order to

demonstrate the various ways that government-funded activities contribute to achieving results.

Checklist

Ministries are expected to use performance measurement to: • Manage activities internally

• Monitor the performance of broader public sector organizations via contract and service management

• Report progress on a quarterly basis to Management Board of Cabinet • Support requests for resources or policy change

• Report to the public on results

Ministries’ performance measures for contract and service management must be aligned with ministry strategies to achieve government priorities and specific results.

Ministries will need to assess existing performance measurement systems to: • Ensure they are working as intended

• Coordinate with transfer payment recipients and other ministries to ensure

horizontal integration and agree on roles and responsibilities for data collection and reporting

• Identify gaps in performance measurement and create appropriate measures and systems for consistent data collection and reporting

SECTION 2:

The Who, What and Why of

Performance Measurement

Key Terms

Customer Satisfaction: The degree to which the intended recipients or beneficiaries of a product or service indicate that the product or service meets their needs and expectations for quality and efficiency.

Effectiveness: The extent to which an organization, policy, program or initiative is producing its planned outcomes and meeting intended objectives.

Efficiency: The extent to which an organization, policy, program or initiative is producing its planned outputs in relation to expenditure of resources.

High Level Indicator: A measure of changes in social, environmental or economic conditions.

Inputs: The resources (human, material, financial, etc.) allocated to carry out activities, produce outputs and/or accomplish results.

Performance Measurement: A Reference Guide – Section 2: The Who, What and Why 19

2.0 What is a Performance Measure?

A performance measure is quantifiable information that provides a reliable basis for assessing achievement, change or performance over time.

2.1 Why Measure Performance?

We measure performance because:

Managers need evidence to help improve activities to

meet ministry strategies. The public wants to know

– and has a right to know – what government does with

their money.

Government needs to report accurately to the public on

what it has done.

Ministers (and Deputy Ministers) need to tell Cabinet what their on an annual basis. ministry has accomplished

Government-funded activities need evidence

to support their recommendations to

their Minister, Management Board of Cabinet or Cabinet for changes to ministry

activities.

Management Board of Cabinet and Cabinet need

evidence to support their decisions.

HOW WELL

ARE WE

DOING? CAN

WE DO

BETTER?

Systematic measurement and assessment of performance supports results-based planning by generating evidence of:

• how well existing government activities perform;

• the extent to which these activities are in the public interest and meet client needs; • whether these activities are consistent with government priorities and expectations.

20

2.2 Who Uses Performance Measurement Information and Why?

Performance measurement information has four uses: • demonstrating accountability for expenditure; • supporting informed decision-making;

• providing accurate information about government-funded activities to the public;

• promoting continuous improvement of government-funded activities and the administration of government itself.

Different users have different information needs:

• external users like clients of government programs and citizens might use performance information to better understand government’s accomplishments and as a way to be more involved in the democratic process;

• internal users such as Ministers, senior management and central agency staff use performance information to support continuous improvement of ministry activities and strategies and in making strategic and resource allocation decisions.

The charts below summarize the different uses of performance measurement information by external and internal government users. For examples of effective use of performance measurement information in Ontario see Appendix 3.

Table 1: How government performance information is used externally

Performance Measurement: A Reference Guide – Section 2: The Who, What and Why

Accountability

Public Reporting

• To understand what the government thinks is important

• To see what results government produces with the money spent

• Enables analysis, interpretation and evaluation of government performance

• Public reports demonstrate that government is making good use of performance information (e.g., improvements to programs/services and

operational efficiencies)

• Citizen education about performance information can help citizens and stakeholders understand performance data and the various ways to use it, from how to ask questions and evaluate

performance to how to influence public decisions

Decision-Making

• To understand what the government thinks is important

• To see what results government produces with the money spent

Citizen/Stakeholder

Performance Measurement: A Reference Guide – Section 2: The Who, What and Why 21

Table 2: Levels of organization at which performance measures are used internally

Across

Government

Individual Level

Activity Level

Ministry Level

Performance measures are used across government to support the

development of the Fiscal Plan and the Printed Estimates Performance

measures support accountability for expenditure at the ministry level via Results-Based Plans and quarterly reporting to Management Board of Cabinet Annual Deputy Minister performance contracts Individual performance plans Performance measures support accountability f expenditure via Integrated Financial Information System, Management Board of Cabinet and periodic audits by Internal Audit Division and the Ontario Provincial Auditor or

A

cc

ou

nt

ab

ili

ty

for Expenditure

Performance measures are used at the level of

Management Board of Cabinet and other Cabinet Committees to support

expenditure and policy decisions in the context of broader government priorities Performance

measures are used to inform decision-making at the activity level by:

Providing evidence to managers within the ministry about how well objectives are being met Helping to identify gaps in service or manage risks to activity delivery by ministries or other broader public sector organizations

Performance measures are used at the ministry level to help identify or adapt ministry strategies to serve vital public interests and deliver on government priorities Performance measurement information is used to guide individual decisions, at all levels of the organization, from program managers to Cabinet Committee members

Decision-Making

Uses of Performance Measurement Information

Performance measures are used to inform decision-making at the activity level by:

Providing evidence to managers within the ministry about how well objectives are being met Helping to identify gaps in service or manage risks to deliver activities by ministries or other broader public sector organizations Performance

measurement information is used to guide individual decisions, at all levels of the organization, from program managers to Cabinet Committee members

Performance measures are used at the level of

Management Board of Cabinet and other Cabinet Committees to support

expenditure and policy decisions in the context of broader government priorities Performance

measures are used at the ministry level to help identify or adapt ministry strategies to serve vital public interests and deliver on government priorities

Public Reporting

Performance measures are used at a ministry level to support continuous improvement by ensuring:

Activities align with ministry strategies to meet objectives Opportunities for integration among activities are maximized Performance

measures are used to support continuous improvement at the activity level by helping staff and managers to adapt activities to meet the changing needs of clients

All Ontario Public Service employees can use performance measures to identify opportunities for continuous improvement

Performance measures are used to support continuous improvement across government by promoting horizontal coordination, communication and integration among government-funded services to improve public access, and maintain a high quality service

2.3 Levels and Types of Measurement

There are different types and levels of performance measures for different uses. Later sections of this guide will explain how to develop the right type of performance measure for each purpose.

There are three levels of performance measurement used in the Ontario Public Service:

Output measures measure the tangible products or services that result from activities and are often raw data expressed in terms of frequency. “Number of (x)” is an output measure. For example, “Number of acres of green space protected through legislation” is an output.

Outcome measures (short-term and intermediate-term) measure the effects/impacts or results of the outputs. Outcomes provide information in relation to other information, or the broader context. Thus, outcomes contain a level of analysis. For example, “Percentage of new growth on existing developed land, e.g., infill, brown- or grey-fields" is an outcome measure because it measures the outputs (new growth) in relation to the existing developed land base.

High-level indicators measure social, environmental or economic conditions. High-level indicators provide important information about the broader context to which government must respond and are therefore useful to support decision-making. However, they are rarely influenced solely by government initiatives. For example, “Percentage of households living below the national poverty line” provides important information about the economic security of families, which is important to know when government is developing social policy. However, government programs in and of themselves are unlikely to affect a change in this measure.

Short-term and intermediate-term outcome performance measurement information is required to support decisions concerning legislation, policy, resource requests and budget and allocations. In some cases, output information may also be valuable. Output or short-term performance measurement information is required for quarterly reports of progress towards meeting the objectives of ministry strategies and may be included in public reports.

Appendix 4 contains detailed descriptions of output measures, outcome measures and high-level indicators for ministries’ internal training purposes.

Identifying output measures and high-level indicators is relatively easy. Identifying good outcome measures can be difficult. The Ontario Public Service uses three types of outcome measures:

Efficiency:

The extent to which an organization, policy, program or initiative is _{producing its planned outputs in relation to expenditure of resources.}

Effectiveness:

The extent to which an organization, policy, program or initiative is producing its planned outcomes and meeting intended objectives.

The degree to which the intended recipients or beneficiaries of a product or service indicate that the product or service meets their needs and

expectations for quality and efficiency.

Customer

Satisfaction:

Ministries are already required to conduct customer surveys and, if adapted, these tools may also be used to support measurement of customer satisfaction. (See http://intra.pmed.mbs.gov.on.ca/ for more information on Ontario Public Service quality standards).

2.4 Summary of Key Points

• The Ontario Public Service uses performance measurement information at all levels of the organization to demonstrate accountability for expenditure, to support decision-making and continuous improvement and to report to the public.

Performance Measurement: A Reference Guide – Section 2: The Who, What and Why 24 Each outcome measure should measure one of the following: Efficiency, Effectiveness and Customer Satisfaction. Ministries should have a range of outcome measures.

Ministries are expected to use performance measurement information to support internal decision-making, policy submissions and requests for resources.

External and internal users of performance measurement information have different needs and these uses need to be considered in the development of performance measurement systems.

The three levels of performance measurement used in the Ontario Public Service are output measures, outcome measures and high-level indicators. Output measures illustrate short-term progress; outcome measures demonstrate the impact of that progress.

SECTION 3:

How to Develop

Performance Measures

Key Terms

Attribution: The demonstrable assertion that a reasonable connection can be made between a specific outcome and the actions and outputs of a government policy, program or initiative.

Baseline: The level of results at a given time that provides a starting point for assessing changes in performance and for establishing objectives or targets for future performance.

Benchmarking: The process of measuring and comparing one’s own processes, products or service against a higher performing process, product or service and adapting business practices to improve.

Target: A clearly stated objective or planned result [which may include output(s) and/or outcome(s)] to be achieved within a stated time, against which actual results can be compared.

3.0 Overview

Developing good performance measures is not easy. Poorly integrated performance measurement systems can be worse than no system at all and may actually support poor decision-making. There are six steps to establishing good performance measures:

"There is nothing so useless as doing efficiently that which should not be done at all."

Peter F. Drucker 1. Define the strategy in relation to government priorities.

2. Identify and consult on cross-ministry initiatives. 3. Identify the performance measures.

4. Check the measures for pitfalls. 5. Establish baselines.

6. Set performance targets.

The following sections take the reader through each step in some detail.

3.1 Developing Performance Measures

STEP 1: Define the strategy in relation to government priorities

The first step involves articulating how ministry strategies contribute to meeting government priorities or serving other important public interests. This takes time, involves many people and requires tools to help organize information logically.

A logic model is a tool that describes ministry activities as part of ministry strategies to meet government priorities. The framework for organizing the logic model presented in this guide is the same as that used by ministries to develop their results-based plans.

A logic model clearly shows the relationships among government priorities, ministries’ strategic

objectives, and how ministry activities contribute to achieving those objectives and priorities through their expected outcomes. The process of creating a logic model and making the linkages among inputs, outputs and outcomes can help build common understanding of what is expected, prioritize activities and identify appropriate performance measures.

A standard logic model diagram is shown below. For examples of real logic models that ministries have created or found particularly useful see Appendix 5. Step-by-step instructions for how to develop a logic model are provided in Appendix 6.

Figure 2: Standard Logic Model Diagram

The logic model provides a foundation for developing performance measures that will support decision-making and is also the basis for meeting Management Board Secretariat reporting requirements. The important points to remember when creating a logic model are:

• developing a logic model is a process that involves many people working together rather than a product which one person can produce;

• the information entered into the logic model at the early stages may need to be revised as new information is entered;

• It’s not just the information in the boxes that counts, but the relationships between the boxes.

Figure 3: Logic Model Process – how each component leads to the next stage

Larger Public Interest:

(Government priorities are usually here)

Inputs Activities Outputs

Short-Term

Desired

Outcomes

Desired

Intermediate

Outcomes

List all inputs List all activities List the tangible

products of the activities

List the changes in participation, awareness, behaviour, compliance or capacity that are expected to result from the activities in the short term

List the benefits or changes in

conditions, attitudes and behaviour that are expected to result from the activities in the intermediate-term

Your Planned Work

Achieving Objectives

Objectives Of The Strategy:

What does the strategy hope to accomplish

High

Level

Changes

Outcomes

Activities

Outputs

Inputs

Customers: Who Benefits

Is there agreement about what

will be measured and how?

"If every general contractor defined a foot as the length of his own foot, our buildings and the construction industry in general would be in a state of disarray. Yet that is precisely what we have done in health care. Every hospital, practice and specialty society has traditionally measured key processes of healthcare in unique and differing ways."

Gene Beed, M.D., 1996 Conference of the (US) National Committee for Quality Assurance on Health Data.

STEP 2: Identify and consult on cross-ministry and horizontal

initiatives

The next step is to identify and consult with other ministries and broader public sector organizations that are contributing to the same result, or that are working to serve a common public interest to:

1. Share performance measures for like activities. 2. Reduce the number of performance measures used. 3. Increase the value of each measure.

For example, both the ministries of Natural Resources and

Municipal Affairs and Housing are involved in activities that protect natural areas or prime agricultural land from certain kinds of development. However, Natural Resources developed its activities with respect to “green space”, while Municipal Affairs and Housing developed its activities in relation to clearly defined “green belts”. Rather than have two measures measuring slightly different things related to the same public interest, the ministries worked together to develop a mutually acceptable definition of “green space” and performance measures to which both will contribute data.

STEP 3: Identify the performance measures

Performance measures should:

• show how ministry activities contribute to achieving government objectives; • provide key information for decision-making;

• capture all areas of significant spending;

• identify and track impact as well as progress towards meeting desired outcomes; • incorporate consideration of risks and act as “thermometers” for risk management.

Data collection methods should ensure that measures are:

• Valid (actually measure what they are intended to measure);

• Reliable (different users of the same measure will report the same results) and; • Sensitive (able to measure change).

The SMART rule provides a way to test the strength of a performance measure:

Specific:

Performance measures state clearly and concisely what will be measured. They focus on significant activities and capture all major areas of spending. They are outcome focused and not a list of activities

S

Measurable:

Performance measures should be quantified, even if based on qualitative data.

M

Achievable and Attributable:

Performance measures relate to things that the ministry can influence and achieve.

A

Realistic:

Performance measures are based on reliable, verifiable data that reflect the ministry/activity contribution to achieving government priorities and results.

R

Timely:

Performance measurement data can be collected, processed and distributed within a useful timeframe and at reasonable cost.

T

Appendix 7 contains a complete checklist for testing performance measures.

What does the measure really

measure?

In Ontario, in 2000, it was illegal to drag a dead horse down a main street. Was the fact that no dead horses were dragged down main streets in Ontario in 2000 proof that the law was working?

STEP 4: Check the measures for pitfalls

The most common pitfalls to performance measurement are attribution and measurement corruption and the only way to detect and avoid them is to know how they may arise. Ask enough questions to be reasonably confident that the pitfalls don’t undermine the usefulness of the measure.

Attribution

Attribution is the assertion that a connection can be made between an outcome and the actions of a government policy, program or initiative. Determining attribution for outputs is relatively straightforward as outputs are the tangible products produced through activities. Demonstrating attribution for outcomes is more complicated because a number of intervening factors, in addition to the activities, may contribute to the outcome. Creating a logic model is the strongest method for identifying the contribution of an activity to the achievement of intended results.

To test for attribution, ask the question, “Is the result we are measuring produced by our actions?” Sometimes, the answer to the above question is clear, as in the example on the right. The fact that no dead horses were dragged down main streets of Ontario in 2000 is actually a better indicator that the law is no longer relevant than it is that the law is working. Other times, the answer is less clear.

Government activities are often not solely responsible for the results achieved in relation to their objectives. There may be factors or events over which the government has no control that affect the outcome. For example, in 2003, the tourism industry in Ontario was negatively affected by the outbreak of SARS, despite ongoing government-funded programs to increase tourism in the province.

Thus, when designing performance measures, it is important to measure the contribution that government-funded activities make, or the influence they exert, rather than measuring only those things over which government has direct control. At the same time, it is important to maintain an awareness of changes that could affect the results. These changes are external risk factors and include broader economic, environmental and sector-specific trends. Clearly identifying the risk factors also qualifies the performance measure so the results reported can be interpreted correctly.

When we think of problems of attribution, we usually think about how performance measurement information is used to claim undue success. But problems of attribution can also make success appear as failure, as illustrated in the example below.

If you can’t see success, you can’t reward it

In the US, the Social Services department was responsible for assuring the welfare of children who were “wards of the state.” Though the state did not directly deliver services to children, they were

accountable, and they established a whole range of output measures to ensure children were well cared-for. For example, third-party providers of care to children had to report to the Department of Social Services on how many children received their annual medical and dental examinations, were in full attendance at school, etc. The results were rolled up into a Social Services Department performance measure of “compliance.”

Compliance rarely reached 60 per cent and the Social Services department was concerned. On further examination they learned that children might have missed a dental examination because they had the flu (and therefore could not be treated safely) or had missed school because they were in transition to adoption. In other words, low compliance by the third party delivery agent was not necessarily an indicator of poor care. In fact, sometimes the lack of compliance was an indicator of high quality service to children that went unrecognized. The Social Services department re-examined their compliance indicators in consultation with third-party service providers. Not only did they obtain higher quality information, rates of compliance increased, as did public confidence in Social Services.

The example above also raises an important area in which attribution issues arise – in situations where a broader public sector organization or agency delivers service on behalf of the government, but for which the government remains accountable for the result. Where such organizations also have a high degree of independence, ministries sometimes have reservations about their ability to demonstrate attribution and therefore accountability for the results achieved by broader public sector organizations. This is why it is essential to involve broader public sector organizations and agencies in the creation of logic models for ministry strategies to which their activities contribute. Obtaining agreement about the objectives and strategies and involving third party organizations in the identification of appropriate performance measures is critical to resolving problems of attribution.

Figure 5: Good performance measures measure the contribution of activities to government priorities Attribution Barrier

Output

3

Desired

Results

Government

Priorities

Outcome

2

Ministry

Strategies

and

Activities

Output

2

Outcome

1

Factors Outside Control

Output

1

Attribution Barrier

What gets measured gets done

The Russian furniture industry, in an attempt to increase efficiency, adopted a pay-for-performance policy in which workers were paid a commission on every pound of furniture produced. Production rates remained stable, but Russian furniture became the heaviest furniture in the world.

If you aren’t rewarding success,

you’re probably rewarding failure

In the UK, national government provided funding to support training programs targeted to people who faced significant employment barriers and had been unemployed for more than two years. The training was provided on a four-year contractual basis by private sector companies whose contracts were renewed on the basis of performance - the number of successful graduates.

After four years, the training companies had met their targets and their funding was renewed. However, research showed that there had been no significant change in the target group. This was because the training

companies had not admitted members of the target group to the training because they were more difficult to train, less likely to complete within the contract time, and would therefore jeopardize the company’s ability to meet its targets.

Measurement illustrates the contribution that activities make, without falling into the trap of incorrectly attributing all success or failure to the activity.

Even when all appropriate steps have been taken, attribution can remain an issue. This is why providing a qualitative description and analysis of performance information is valuable. It explains the larger context and data collection methods, the limitations on the interpretation of the data, and the key risk factors so that attribution issues are clearly identified. In this way, users of information are less likely to use the performance measurement information inappropriately.

For a more detailed discussion of attribution and how to avoid attribution problems, see John Mayne’s article, “Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly,” (Office of the Auditor General of Canada, 1999), which can be found at the following internet address:

http://www.oag-bvg.gc.ca/domino/other.nsf/html/99dp1_e.html/$file/99dp1_e.pdf.

Measurement Corruption

The second pitfall of performance measurement is measurement corruption. People adapt their behaviour in relation to the thing being measured, as in the example on the right. It is important to try to anticipate any undesirable behaviours the measure could inspire and revise the measure accordingly. Another example of measurement corruption in the context of target setting is given on page 36.

The risk of measurement corruption in performance measurement is relatively high, due to our natural desire to be successful and our boundless creativity and adaptability. It is rarely intentional and is usually produced by the measurement itself.

Performance measures are less likely to become corrupt if all those involved understand the measure, what outcome it is intended to measure and agree to the choice as relevant and appropriate. This is another reason why everyone affected should be involved in the development of the logic model and identification of performance measures.

Another way to deal with the effects of measurement corruption is to ensure performance measures are part of a system of

measurement. If one measure becomes corrupt, other measures may signal that corruption by their lack of change. The example of training programs on the right illustrates this point. The lack of change in employment rates of the target population provided a clue that the measurement of training completion may have been corrupt by raising the question, “Why isn’t there change for the target group as a result of the government-funded program to improve its access to employment?”

If you can’t recognize failure

you can’t correct it

A transitional employment (TE) program was designed to place clients with mental illness – particularly those with little work history - in short-term, entry-level jobs on a rotational basis. At any given time there were 20 jobs and 300 clients. Program staff were certain that all clients were being given the opportunity to

participate. When a new

performance measurement system was established, staff were surprised to learn that most of the time the same 20 clients were being offered the TE jobs as they became available. The fast pace of the program, the heavy

demands on staff time and the lack of objective information had resulted in available jobs being offered to clients with the strongest employment record, a total contradiction of the program’s fundamental purpose. In response to the performance information, the program changed their job allocation practices to ensure that everyone on the wait list,

particularly the most disabled clients, were given access to TE jobs.

Unintended Consequences

Performance measures can also help to identify unintended consequences of implementing a policy or program. In the example to the right, staff assumed they were treating all clients equally, but performance measurement demonstrated the rapid response times of the program led to jobs being given to those clients who were already the most prepared for employment. Measuring performance will not “save the world.” Performance measures are a tool. They help us to:

• discover quickly if progress is being made toward

objectives as expected and to tell the difference between success and failure;

• make ongoing decisions about how to best use resources, when to make changes to services and identify existing gaps;

• identify when corrective action may be needed and guide what that action should be;

• most activities are on a large scale and delivered over vast regions. Performance measurement helps us to know when we are successful and to identify potential problems quickly.

STEP 5: Establish Baselines

A baseline is the level of results at a given time that provides a starting point for assessing changes in performance and establishing objectives or targets for future performance. Ministries are expected to have identified baselines for existing performance measures. When a ministry introduces a new performance measure, it should establish the baseline during the first year the performance measure is in use. Every time

performance measures or data collection methods are changed, new baselines need to be set.

STEP 6: Set Performance Targets

A target is a clear and concrete statement of planned results (including outputs and outcomes) to be achieved within the time frame of parliamentary and departmental planning and reporting against which results can be compared. Every ministry is asked to use comparative data to set targets based on its own

performance, established industry standard, customer preference and/or performance of a comparable organization. Reasonable targets are challenging but achievable.

Setting annual and long-term targets

Performance targets set expectations for results and are the basis from which measurement takes place and improvement begins. Without targets, it isn’t possible to know whether the organization is improving or falling behind in achieving results in priority areas.

Targets show whether the ministry proposes to meet or exceed the standards for performance. They should be clear and quantifiable and expressed in absolute number, percentage or ratio terms. They should also define the timeframe within which they will be achieved. Targets are used as a key tool to drive, measure, improve and control performance.

Benchmarking

Benchmarking is the process of measuring and comparing one’s own processes, products or services against a higher performing process, product or service and adapting business practices to improve performance. Where possible, ministries should benchmark their performance information against other Ontario Public Service ministries and public or private sector standards. Benchmarking performance information against another internal or external high performing organization may help ministries to identify more effective and efficient processes for achieving intended results. For an example of how one jurisdiction used benchmarking to demonstrate its performance see Appendix 8.

There are three commonly accepted forms of benchmarking:

• Standards Benchmarking: Setting a standard of performance in relation to the performance of other organizations, which an effective organization could be expected to achieve. The publication of a challenging standard can motivate staff and demonstrate a commitment to improve services;

• Results Benchmarking: Comparing the performance of a number of organizations providing a similar service. In the public sector, this technique can allow the public to judge whether their local provider makes effective use of resources, compared to similar providers;

• Process Benchmarking: Undertaking a detailed examination within a group or organization of the process that produces a particular output. This is done to understand the reasons for the

variances in performance and incorporate best practices.

While there is currently no formal requirement for ministries to benchmark their performance, it is expected that such comparisons will be incorporated into all public reporting by 2007.

3.2 Summary of Key Points

There are six steps to establishing good performance measures: 1. Define the strategy in relation to government priorities. 2. Identify and consult on cross-ministry initiatives. 3. Identify the performance measures.

4. Check the measures for pitfalls. 5. Establish baselines.

6. Set performance targets.