Indicators of effectiveness

Indicators of effectiveness are used to measure whether a criterion (such as output) has been included in the design and implementation of a programme or project in a manner consistent effectiveness in addressing EDPs in developing countries.

A mix of standard and EDPs specific indicators of effectiveness for each criterion is used. The choice of indicators for monitoring policy focused-programme and project effectiveness is complicated and reflects the practical difficulty of identifying objective indicators for ‘soft’ infrastructure programmes or projects such as those used to address EDPs. This is particularly evident when the UN is supporting governments and works in partnership when they are unable to address many of the factors outside the control of the programme or project that may impact directly on the outcome of the project.

The indicators for evaluating the effectiveness of the EDP case studies are presented in Table 4.2.

Table 4.2: Case Study Criteria Indicators of Consistency with Programme & Project Effectiveness

Criteria Indicators to assess if the criteria are consistent

with programme/project effectiveness in addressing EDP(s).

Means of Verification

Situational Analysis

1. Analysis of problems & underlying causes is carried out and consistent with actual EDP and sets out the rationale for the

programme/project.

2. Analysis of higher-level policies, institutional, political-economy etc. issues that may impact on achievement of the desired results

3. Analysis of key institutional, policy, planning and implementation processes relevant to addressing the EDP.

ProDoc, progress reports,

meeting minutes, evaluations, guidelines on

causes & approach to addressing EDPs plus interview records. Government, Inter- Governmental (e.g. Strategy (includes implementation arrangements)

4. A participatory approach to development of

Strategy (& overall design).

5. Strategy consistent with problem statement & key national processes linked to the drivers of the problem

6. Ministries of Planning/Finance lead or co-lead the process & key EDP relevant sector ministries (e.g. Agriculture) are an integral part of programme management & oversight structures.

7. Programme/Project is integrated into relevant Government or regional processes &

institutional mechanisms.

8. Strategy is consistent with EDP good practice. E.g. as contained in relevant UN EDP

guidelines.

9. Reflects higher-level issues that contribute to the causes of the problem and/or make addressing the problem more difficult.

SADC, UN) reports, national and regional development planning documents. (E.g. Poverty Reduction Strategy Papers PRSPs, regional, regional development strategies) UN development analysis & priority setting

documents – e.g. UN Development Assistance Frameworks (UNDAFs).

Outcome 10. Outcome statement states actual or intended

change the programme interventions are seeking to support.

Outputs & Activities

11. Outputs included are environment &

development results to be produced within the project timeframe & budget and are consistent with the outcome.

12. Activities describe actions needed to obtain the stated outputs & are consistent with the Outputs that need to be produced.

Work plan 13. Work plan provides enough time for delivery

of all outputs & activities.

Budget 14. Budget sufficient to pay for activities &

outputs.

15. Budget is broken down by activities and/or outputs

Management & Co-ordination Arrangements

16. Arrangements specify clear responsibility for programme/project management, including co-ordination between different components & implementing agencies.

17. Ministries of Planning/Finance lead or co-lead the process.

18. Government steering and/technical committee

chaired by lead ministry responsible for programme/project.

19. Reflect capacity of Government & other partners

Risk Assessment 20. Risks to achievement of desired outcome &

sustainability, including higher level risks, are identified with actions to minimise risks contained in risk plan

Monitoring & Evaluation

21. Results based (rather than activity or output focused) quarterly & annual monitoring included

22. Progress reports are responded to with remedial actions implemented, documented and reported against.

The means of verification are the sources where the evidence to score the indicators will be obtained. As the same means of verification apply to most or all indicators, they are listed as applying across the range of indicators, rather than against each indicator.

The basis of choosing these indicators follows good practice in programme or project design in general and for addressing EDPs in particular. These standards are including in general programming manuals and in those targeting EDPs (Bass and Renard, 2009; DFID, 2003; DFID, 2011; Bass and Renard, 2009; NORAD 1999; NORAD, 2005; UNDG 2011; UNDP, 2009; UNDP, 2011a; UNDP 2011b; UNDP-UNEP PEI 2009). For example, UN and donor manuals state that the situational analysis should identify the problem and describe the institutional and political-economy context (DFID, 2003; UNDP, 2009). Guidelines and evaluation reports focusing on EDPs highlight the importance of focusing on the existing national development planning processes and of using economic evidence of the benefits of addressing EDPs. This necessarily involves a focus on influencing ministries of planning and finance (Bass and Renard, 2009; UNDP-UNEP PEI, 2009). (See Appendix 4 for the rationale for the choice of indicators).

A draft set of indicators based on the above referred to manuals and guidelines was prepared by the researcher and submitted to UN officials experienced with EDP projects and programmes for their comments and then finalised as set out in Table 4.2.

4.4 Analysis of Examples of UN Activities or Initiatives relevant to Addressing EDPs.

The second component of the methodology is set out in Box 2 of Figure 4.1. While this study is primarily based on full case studies of programmes or projects designed to address EDPs, analysis of a number of examples of other UN activities or initiative is also part of the methodology. These are analysed in a separate chapter, and do not involve a full application of the case study evaluation framework. Rather, the analysis of the examples provides additional information for assessing UN effectiveness and helps to validate the findings of the case study evaluation.

4.5 Research Diary and Interviews