H > C E E
Supporting Effective Evaluations: A Guide to Developing Performance Measurement Strategies
6.0 Performance Measurement Strateg Framework 6.1 Overview of the Performance Measurement Strateg Framework
T PM S F
. I :
;
, ;
;
.
P
. S
S 2.3
.
6.2 Content of the Program Performance Measurement Strateg Framework
T 3 PM S F . T
' PAA PAA
, . . , / - . I
' , (
), . F ,
:
( )
,
Table 3: Sample Table for the Performance Measurement Strateg Framework: PAA Elements Linked to the Program Program outputs and
outcomes Indicator Data
source Frequenc Baseline Target Date to achieve target
Organi ation and position responsible for data collection
managementData s stem
Output 1 I 1
Output 2 R
.
I 2
I 3
Outcome 1 I 4
I 5
Outcome 2
R a be added be f addi i a
c e .
I dica 6 I dica 7 I dica 8
6.3 Performance Measurement Frameworks and the Performance Measurement Strateg Framework
The Polic on Management, Resources and Results Structures (MRRS) e i e he de e e f a de a e a Pe f a ce Mea e e F a e (PMF), hich e he e ec ed e a d he e f a ce ea e be e ed f g a ide ified i he PAA. The PMF i i e ded c ica e he e a chi g f a e h gh hich a de a e i c ec a d
ac e f a ce i f a i ab he i e ded e f he de a e a d i g a . The i dica i he de a e a PMF a e i i ed i be a d f c i g de a e a
i i g a d e i g.
The PM S a eg F a e i ed ide if a d a h e f a ce i f a i i be c ec ed g i g i i g f he g a a d i e a a i . I i i e ded e effec i e b h da - -da g a i i g a d de i e a d he e e a e a a i
f ha g a . Acc di g , he PM S a eg F a e a i c de e ec ed e ,
a d i g e f a ce i dica be d he i i e ab i hed f he e ec ed e a d e f a ce i dica be i c ded i he MRRS PMF ( ee Fig e 2 be ). U i e he MRRS PMF, he PM S a eg F a e ha i ed i i he be f i dica ha ca be i c ded;
h e e , cce f i e e a i f he PM S a eg i e i e if i dica a e e a ea ab e be .
Figure 2: Comparison of Requirements: Performance Measurement Frameworks and Performance Measurement Strategies
Fig e 2 - Te Ve i
A i a ed i Fig e 2, he i dica i he PM S a eg F a e f c i g g i g g a i i g a d e a a i ac i i ie a d he ef e a ig i h a d c e e he
i dica i c ded i he departmental PMF. I i a ce he e he g a i h a a di i c g a i he PAA a d i dica ha e bee ide ified i he de a e a PMF, he PM S a eg F a e h d i c de, a a i i , he i dica e ed i he de a e a PMF.
Whe a PM S a eg i de e ed f a e g a ha i e e e ed i he de a e a
PAA, he c e , a d e a ed i dica de e ed f he PM S a eg F a e h d be c ide ed f i c i i he de a e a MRRS.[10]
6.4 Accountabilities and Reporting
The PM S a eg F a e h d be acc a ied b a h e ha de c ibe :
he e i g c i e a d h he i be e , i c di g h i a a e he da a, h i e a e he e , h he i be b i ed, b he , ha i f a i i be i c ded, he e f he e a d h he i be ed i e e f a ce;
a d
if e e a , he e ia cha e ge a cia ed i h da a c ec i a d e i g, a e a i iga i g a egie f add e i g he e cha e ge (e.g. he e a be a e ha ca be ed f da a a age e ).
6.5 Considerations when Developing the Performance Measurement Strateg Framework
The cha be ide guidance on how to develop the PM Strateg Framework.[11]
Step Description Comments
1. Start with the MRRS PMF: Re ie he MRRS PMF ha he de a e de e ed i
acc da ce i h he P ic MRRS. I c de, a a ia e, he e f a ce i dica
f he MRRS PMF i he PM S a eg
F a e .
The PM S a eg F a e h d be de e ed i i a i . I acc da ce
i h he P ic MRRS, a g a e e e ed i a de a e ' PAA c ib e i a egic c e( ). A
ch, he de e i g he PM S a eg F a e , he e f a ce i dica
h d c e e h e a ead e ab i hed i he de a e a PMF.
2. Identif performance indicators: De e e ec a ea e e f a ce i dica f each a d each c e (i edia e, i e edia e a d i a e) ha ha bee ide ified i he g a logic model.
Kee i i d ha , i addi i da - -da g a i i g, he e f a ce i dica i a be ed f e a a i
e . A ch, i i ec e ded ha g a a age a c ide he core issues[12] for evaluation (i.e. e e a ce a d
e f a ce) a d c i h head f e a a i he de e i g he e f a ce i dica .
The e a e e f i dica : a i a i e a d a i a i e.
Quantitative e f a ce i dica a e c ed f a
be a d a i . The be i dica e he ag i de (h
ch) a d he i gi e he
be i ea i g ( ha ), e.g. he be f i e c ai
ecei ed.
Qualitative i dica a e
e e ed i e i f , e.g.
a e e f e ea ch a i . A ch a ib e, a i a i e i dica h d be c de ed i a a i g ca e, e.g. e ea ch a i i a ed a "e ce e ," "a e age"
"be a e age." Thi i a f c a abi i e i e.
Ch i g a ia e e f a ce i dica i e e ia f effec i e e a a i g a d i i g a
g a ' g e . A ia e i dica a e cha ac e i ed a
f :
Valid The indicators measure what they intend to measure.
Reliable The data collected should be the same if collected repeatedly under the same conditions at the same point in time.
Affordable Cost-effective data collection (and analysis) methods can be developed.
Available Data for the indicators are readily and consistently
available to track changes in the indicator.
Relevant The indicator clearly links back to the program outcomes.
Keep the number of performance indicators to a manageable si e. A small set of good performance indicators are more likel to be implemented than a long list of indicators.
3. Identif data sources: Identify the data sources and the system that will be used for data management.
There are a number of possible data sources:
Administrative data Information that is already being collected through program files or databases or could be collected with
adjustments to regular programming processes or funding agreements;
Primar performance data Specialized data collection exercises, such as focus groups, expert panels or surveys, in which information needs to be collected;
and
Secondar data Information that has been collected for other
purposes, such as national statistics on health or economic status (e.g. Statistics Canada data).
Use readil available information.
Take advantage of an available sources of information at our disposal.
4. Define frequenc and responsibilit for data collection: Identify frequency for data collection and the person(s) or office
responsible for the activity.
Describe how often performance
information will be gathered. Depending on the performance indicator, it may make sense to collect data on an ongoing, monthly, quarterly, annual or other basis.
Whe a i g he f e e c a d ched i g f da a c ec i , a i a fac c ide i
a age e ' eed f i e i f a i f deci i a i g.
I a ig i g e ibi i , i i i a
a e i acc hich
a ie ha e he ea ie acce he da a ce f da a b a he ca aci ie a d e ha i eed be i ace faci i a e da a c ec i .
5. E abli h ba eline , a ge and imeline . Pe f a ce da a ca be ed f i i g a d e a a i if he e i
e hi g hich i ca be c a ed. A ba e i e e e a he a i g i f c a i . Pe f a ce a ge a e he
e i ed a e he e a i e
c ib i f he g a a d e e ha a ia e i f a i i bei g c ec ed.
Ba e i e da a f each i dica h d be ided b he g a he he PM S a eg F a e i de e ed. Of e , a d a ic a f i dica e a ed highe - e e c e , hi i f a i
i ha e a ead bee c ec ed b g a a age a a f hei i i ia eed a e e a d he de e e f hei b i e ca e.
Ta ge a e ei he a i a i e
a i a i e, de e di g he i dica . S e ce a d efe e ce i f ide if i g a ge i c de:
ba e i e da a ba ed a e f a ce e i i e a i f he g a ;
he achie e e f he , i i a g a ha a e c ide ed eade i he fie d (be ch a i g);
ge e a acce ed i d b i e a da d ; a d
b ic a ed a ge (e.g. e b he g e e , he fede a b dge ).
If a a ge ca be e ab i hed if he g a i ab e e ab i h ba e i e da a a he e f he g a , e ici i e i e f he he e i be e ab i hed a e a h i e ib e f e ab i hi g he h d be a ed.
Date Modified: 2010-09-29
6. Consult and verif : As a final step, program managers should consult[13] with heads of evaluation and other specialists in performance measurement to validate the performance indicators and confirm that the resources required to collect data are budgeted for.
Consultation with other relevant program stakeholders (e.g. information management personnel) is also encouraged.
This step complies with the Direc i e on he E al a ion F nc ion, which holds heads of evaluation responsible for reviewing and providing advice on the PM Strategies for all new and ongoing direct program spending, including all ongoing programs of grants and contributions, to ensure that the effectivel support an evaluation of relevance and performance (Section 6.1.4, subsection a).
It is highl recommended that program managers work with the head of
evaluation to ensure that the performance indicators selected will provide adequate data to support an evaluation of the program. The choice of consultation mechanism remains at the discretion of the program manager and head of evaluation.
Ke questions to be considered during the consultation and verification stage
include:
Does the selection of indicators align with and complement the indicators in the departmental PMF?
Are the indicators valid measures of the outputs and outcomes (i.e.
do the measure what the intend to measure)?
Are the indicators reliable (i.e.
would the data recorded be the same if collected repeatedl under the same conditions at the same point in time)?
Is it likel that the required data will be provided in a timel manner?
Is the cost reasonable and clearl budgeted for?
If the answer to an of the above questions is "no," adjustments need to be made to the indicators or to the resources allocated for
implementation of the PM Strateg .
Implementation
The full benefits of the PM Strateg Framework can onl be reali ed when it is implemented.[14]
As such, program managers should ensure that the necessar resources (financial and human) and infrastructure (e.g. data management s stems) are in place for implementation. Program managers should begin working at the design stage of the program to create databases, reporting templates and other supporting tools required for effective implementation. Following the first
ear of program implementation, the should undertake a review of the PM Strateg to ensure that the appropriate information is being collected and captured to meet both program
management and evaluation needs.
Keep in mind that, according to the Direc i e on he E al a ion F nc ion, the head of evaluation is responsible for "submitting to the Departmental Evaluation Committee an annual report on the state of performance measurement of programs in support of evaluation" (Section 6.1.4,
subsection d). Given that the PM Strateg Framework represents a ke source of evidence for this annual report, its implementation is especiall important.