Key Players in Performance Management & Performance Measurement

IAS Conference:

Performance Audit by Public Sector Internal Auditors

October 11-12, 2012 2009 William Dorotinsky

Key Players in Performance Management &

Performance Measurement

1

Table of Contents

•

Objectives of performance systems

•

Useful concepts and constructs

•

Users & uses of performance information

•

Performance models & techniques

•

Pressure for results

•

Role of internal auditors

Objectives of Performance Systems

1. To learn from

• Tools: strategic planning, internal management information systems, benchmarking, risk analysis, business process reengineering

• Primarily for internal ministry management

• Examples: US Statestat, Dutch benchmarking in sectors

2. To steer and control, align efforts to goals

• Tools: monitoring and management scorecards, performance pay, strategic performance budgeting

• Primarily for central government (MoF/Budget, PM etc..)

• Examples: Japanese program and policy evaluation, South Korea performance budgeting

3. To provide accountability

• Tools: citizen charters, annual reporting, performance contracts • Primarily for central government (MoF/Budget, PM etc..)

• Primarily for reporting to citizens, media, external stakeholders • Examples: UK Charters/Compacts, Peru’s RECURSO program, US

Expectmore.gov

2

Types are not mutually exclusive, but reflect the relative emphasis present in any given system. Performance systems emphasize various objectives:

Value is in:

Strategic planning

• Focus on longer term objectives

• Align components towards same objectives • Enable coordination towards objectives

Performance monitoring

• Monitor progress towards objectives

• Identify areas off-track • Manage off-track areas to move them forward

A Complex Transition in the Public Sector

Control & compliance

o Rules governed o Limited flexibility o Non-preventive o Negative focus

o Control of individuals

Performance & Accountability

o Mission driven o Customer driven o Employee responsive o Flexible

o Preventive

o Positive team focus

A significant cultural & institutional change

Performance is for the most part determined by where you sit

– As a purchaser of a service, your interests are typically different from those of the supplier of the service o Buyer of car; versus manufacturer of car

– As a regulator, your interests are typically different from those of the regulated o Compliant behavior versus lightness of burden

– In some cases, parties may have interests as both purchasers and suppliers (e.g. co-operatives)

Government performance is complex in part because of the variety of interests

– Standard problems of government production

With different degrees of clarity, performance systems attempt to address these various interests

– Institutional fabric is important

o Westminster; Presidential; other Parliamentary etc o Common vs Civil Law

Gauging Interests

Evolution of Control Approaches

Ex ante

Ex Poste

External

Centralized commitment control (transaction approval)

Allocations (commitment limits) Warrants (cash limits)

Procurement procedures Personnel/pay rules “continuous auditing” Disbursement rules

Central internal audit External audit

Regular reporting, management intervention

Quarterly close-outs Cash rationing

Transparency

Internal

Procedures to minimize risk (internal controls)

Transparency

Ministry internal audit Performance management

Needs

Objectives Inputs Activities Outputs Final Outcomes

Environment

Intermediate Outcomes

Utility and Sustainability

Economy Cost Effectiveness Efficiency Effectiveness Socio-economic Situation Organization or Programme Relevance

Concepts & Constructs

A production model of performance:

The Relationship between Performance Measurement &

Management

Across the Public Sector: Users & Uses of Performance

Information

User

Potential uses of performance information

Presidency/Cabinet•_• Key steerering agent

Commonly set key objectives and policy priorities for the government

Ministry of Finance/Budget

• Reviewer of rowers’ performance

• Prepares consolidated budget submission with increasing use of performance information to justify allocations

• Monitors within year performance

• Typically guide performance reforms with firm regulation and oversight

Line Ministries & Providers

• Key rower

• Generate majority of performance information

• Tend to use performance information for internal management, but are beginning to use it in negotiating budgets with the executive

Legislative/Parliam ent

• Key steerer or reviewer, depending on characteristics of system

• Access to increasing amounts of performance information, mixed evidence of use in the approval or review of the budget

External Audit

• Key attester of financial and increasingly, non-financial information • Increasing use of performance audits and program/sector reviews

• May also conduct audits of performance information systems and performance measures/indicators/targets

Public

• Key consumer, taxpayer and voter

• Experiential information, formal use of performance information has been fairly limited,

although interest groups, Think-Tanks and the media focus on areas, such as league tables/citizen charters in service areas, e.g. school exam scores, hospital treatment waiting lists etc

Within Ministries: Users & Uses of Performance

Information

User

Potential uses of performance information

Program Managers

• Management of programs, real-time readjustments

• Quick assessments of impact for learning what to do better • Reporting

Senior Officials in central agencies & ministries

• Assessing performance of programs across the Ministry • Managing entity performance

• Reallocating resources (time, $, staff) across programs • Tracking key priorities

Internal Audit

(Potential Roles)

• Performance Audit

• Ensuring use/incorporation of performance information into decision making • Verification of quality of performance information

• Assessing relevance of performance tools in place • Conducting performance assessments

Ministers • Holding staff accountable for performance

• Reporting upward

• Communicating with the public/clients

Performance Models & Techniques

Different performance tools have been developed to meet different objectives and

problems, and address different linkages and accountability arrangements among various actors within the public sector.

9 Citizens Ministers Front Line Service Providers MoF/MoE/ Treasury Legislature Executive Ministries Ministries Front Line Service Providers

▪

▪

▪

▪

▪

▪

Agreements (UK PSAs)

▪

Pressure for Results – hypothetical

Ministry

Department

Front Line

The primary use and benefit of performance information resides within Government ministries & departments for management purposes. Actors external to this environment exert pressure

The Role of Internal Audit

Internal Audit’s scope within an organization

can be broad and may involve topics such

as:

¾ the efficiency of operations

¾ the reliability of financial reporting

¾ deterring and investigating corruption

¾ compliance with laws and regulations

¾ IT audits

¾ risk assessments

¾ performance auditing

¾ etc…

11

Definition of Internal Audit

“An independent, objective

assurance and consulting activity designed to add value and improve an organization's operations.

It helps an organization accomplish its objectives by bringing a

systematic, disciplined approach to evaluate and improve the

effectiveness of risk management, control, and governance processes.”

Source: The Institute of Internal Auditors

In most countries, performance audits of governmental activities are carried out by the external audit bodies, but performance audits may also be conducted by

Performance Auditing

Performance audit

- the “independent examination of the efficiency and

effectiveness of government undertakings, programs or organizations, with due

regard to economy, and the aim of leading to improvements.”

In most countries, performance audits of governmental activities are carried out

by the external audit bodies, but performance audits may also be conducted by

internal auditors.

Performance auditing vs. Performance monitoring

12

Source: INTOSAI, “Implementation Guidelines for Performance Auditing”

Ex-post Continuous Conducted by internal or external auditor or evaluator Management responsible Immediate feedback loop into operations Time lag in feedback

Key Questions for Internal Auditors in the Performance

Space

Internal auditors may also consider assessing the quality of performance

management & measurement that is occurring at different levels of an

organization.

Key questions include:

- Are managers being held accountable for performance? If not, why?

- What factors may be limiting individual accountability?

- How is performance information being used? What incentives exist to produce, incorporate, and use performance measurement?

- What is the quality of the performance information being produced at various levels?

- What impact are the performance management tools (e.g. Balanced Scorecards) having on the entity?

Systemic Obstacles to Unleashing Performance

14

It is difficult to hold managers to account if:

9 They were not involved in budget setting, planning, target setting 9 Performance metrics, targets change annually

9 Do not actually receive approved budget levels; no predictability in funds availability 9 Have no influence over procurement processes for their work

9 Have no influence or human resource management for their office (positions or employment)

9 Do not receive continuous or regular spending reports for their units 9 No regular management or performance and output reports

9 Have no flexibility or discretion in resource allocation

9 Have treasury/accounting systems on a cash basis, necessitating their own parallel commitment accounting

9 Managers selected for technical skills, not management ability 9 No managerial training, support system

9 Auditing focuses on compliance, while legislature and chief executive focus on performance results

Misalignment of Various Systems

Entities need to know what they are supposed to accomplish

Tougher than it sounds

Valid measures of performance need to exist

Tougher on the non-observable outcomes and outputs

Accurate measures of cost need to be developed

Cash and accrual

Systems requirements

Cost and performance information should be brought together in

budget process

Per stages identified

Participants need to have incentives to use the information

Probably as important as any aspect

Urban Institute 2001 –

Foundation for Managing Performance

Concluding Remarks

Understanding performance models & tools:

•

What are the uses and who are the users of each

model?

•

How is the impact of the model measured?

•

What is the broader environment in which these

models will operate?

•

How is the quality of the performance information

generated from these models assured?

•

What is the unit of analysis? (Ministry, Program,

Project, Individuals, Departments)

Bigger questions to keep in mind during the day:

•

What role should Internal Audit have in the

performance agenda? Under what circumstances

should IA assume such a role?

17

Key debates in incorporating a performance orientation for Internal Audit:

¾ getting the balance right between compliance & performance

¾ new skill requirements ¾ legal mandate,

objectives/roles, reporting ¾ relationship to External

Audit

¾ avoiding duplication in efforts, aligning with other performance assessments

Efficiency and Economy of Performance Monitoring?

18 The Worker

The man who monitors the worker

The man who monitors the man who monitors the worker

The man who monitors the man who monitors the man who monitors the worker

The man who wrote the draft policy guide on how to monitor the man who monitors the worker

Thank You.

20

Thailand’s Performance Tools – a bird’s eye view

21

OPDC

OPDC BOBBOB OCSCOCSC NESDBNESDB

BSC/PMQA

BSC/PMQA PARTPART HR BSCHR BSC Value for

Money Value for Money 20 Ministries 20 Ministries 141 Departmen ts 141 Departmen ts 18 Provincial Clusters 18 Provincial Clusters 75 Provinces 75 Provinces 73 Academic Institutions 73 Academic Institutions DOLA DOLA Service Standards/ LQM Service Standards/ LQM Local Administr ative Org. Local Administr ative Org.

Quantity _{the quality or standard of output expected} Quality _{the quantity of output to be delivered} Cost _{the cost of output delivery}

Timeliness _{the timeframe for delivery of the output}

Location _{the physical location where the output will be delivered}

Externally focused The output describes a distinct product or service.

Controllable Output specifications appropriately reflect the extent of control over the

delivery of each output.

Comprehensive The outputs should incorporate all of the core outputs that were delivered by the entity. This includes outputs where delivery has been subcontracted to another party. It also may include all outputs funded by parties other than ratepayers or taxpayers — for example, outputs funded through user charges.

Measurable Output specifications provide clear descriptions and standards against

which actual performance can be judged.

Informative Output descriptions are sufficient to meet the information needs of a relatively new or unfamiliar user of the information.

Outputs and Indicator Characteristics