One of the key step of the evaluation process consisted in defining the design of the evalua- tion and its corresponding methodological framework which served as a basis for the entire evaluation exercise.
Given the purpose and conditions of the evaluation, the most appropriate design for the evaluation was a multiple case study with literal replication based on the use of a mixed- methods approach. The elaboration of the methodological framework was based upon sev-
eral tasks.
A first task was to define the intervention logic underlying the EC support to decentralisa-
tion processes in the EU’s external co-operation with partner countries. This was a prerequi- site for the evaluation, since it facilitates understanding of the hierarchy of the objectives aimed at being achieved with a view to contributing to the overall objectives of the EC’s de-
Structuring phase Desk phase Field phase Synthesis phase Dissemination seminar Deliverables Meetings
Main Tasks Overview of EC
support to decentralisation processes Inventory Determination of EQs, JCs and Indicators Detailing of the methodology Interviews at HQ Analysis of policy/ strategy documents Document review & ROM analysis Questionnaire to EU delegations Drafting of preliminary answers to the EQs Refinement of the methodology Focused country visits
Debriefing with the Delegations Drafting country notes
Drafting complete answers to the EQs Drafting conclusions & recommendations Drafting final report
Dissemination seminar in Brussels
Inception meeting Inception Report
Desk Report Field phase reporting (incl. country notes)
(Draft) Final report Dissemination presentation
RG RG RG RG DS
velopment policy. It therefore constituted the basis for formulating the Evaluation Questions (EQs) and served as the benchmark against which to evaluate the activities financed.
A second task consisted in defining and structuring a set of EQs. Indeed, the purpose of
the evaluation is to verify to what extent the EC's intended objectives have materialised as envisaged. It should also allow for covering various evaluation criteria (including the five DAC criteria) and a number of key issues identified in the terms of reference and through discus- sion with key stakeholders. Accordingly, a set of eight EQs has been defined, so as to shed light on some critical points of the intervention logic and provide more concrete content to the evaluation criteria and key issues.
With a view to facilitate the data collection as well as the construction of answers to these questions at a later stage, each question has been further structured. To answer each ques- tion, appropriate Judgement Criteria (JC) and related indicators were defined. Further-
more, potential information sources were identified for each indicator, as well as appropriate methods and techniques for collecting and analysing the information. The next section
explains in more detail how the evaluation questions were defined, how they are linked to the evaluation criteria and the key issues, and how they were structured.
As mentioned earlier, the evaluation is based upon a multiple case study. Given the purpose and conditions of the evaluation, the most appropriate cases to be analysed during the desk study and the field work were deemed to be "country cases". Thus, a third important task during the structuring phase was to select the relevant "country cases". The ToR indi- cated: "The evaluation will include a comprehensive desk phase followed by a field phase with missions to 10 different countries. The choice of the countries will be done upon selec- tion criteria that will be defined taking into account geographical representation and the ap- proach to sector analysis". In order to reach a reasonable balance between generating a rich evidence base and keeping the study feasible, it was decided to focus on 22 countries dur- ing the desk phase and 10 countries selected out of the desk phase sample during the field phase. Overall, the country cases were selected to reflect the diversity of EC partner coun- tries and EC programmes and approaches.
2.2.2 Selection of country cases for the desk study and the field phase
The following criteria have been used for the selection of the 22 countries for the desk study:
Geographical and sub-geographical representation (i.e. Africa/Caribbean/Pacific,
Asia, Latin America, ENP countries): this criterion helps to take into account the vari- ety of country contexts and the corresponding programming/implementation opportu- nities and challenges.
Entry points: this criterion provides a meaningful way of explaining and categorising interventions in support of decentralisation processes and give an idea on the variety of contexts that can be faced as well as of the variety of corresponding approaches and instruments that can be used.
Amount and type of financing: this criterion helps taking into account the variety of aid delivery methods used by the EC to support decentralisation processes and also ensures that the analysis covers a significant part of financed interventions.
Last, some consideration was also given to look at support to decentralisation proc- esses in:
o countries with various level of progress on decentralisation21;
o post-conflict and/or fragile states22;
o countries with different levels of income23;
21 Data used relied mainly on information available in the CSP and in the First and Second Global reports by
United Cities and Local Governments.
22 Mainly based on the list of countries monitored by the OECD- DAC’s Fragile States Group. 23
o countries of various sizes, i.e. the size of the population24.
The selection of the counties has been automated, to the extent possible, using Excel data related to the inventory25 and a data set on country key indicators (population,
income-level, etc.).
The selection of countries for field study analysis has applied the same guiding criteria. However, these criteria have been complemented by the following considerations:
The country case study might highlight interesting lessons learned.
Avoidance of countries where a country level evaluation has been undertaken re- cently or is being planned.
The table below presents the list of the 22 country cases selected for the desk phase and indicates which ones among them were selected for the field visits.
Table 5 List of country cases selected for the desk phase & the field visits Region / Country
Africa
Benin Congo (Democratic Republic of)
Kenya Madagascar
Mali Rwanda
Senegal Sierra Leone
South Africa Tanzania
Uganda
Asia
Cambodia The Philippines
Caribbean Haiti ENP Jordan Lebanon Latin America Colombia Guatemala Honduras Nicaragua Peru Pacific
Papua New Guinea
Legend: country selected for the field visit = Country
The sample includes 11 countries in Africa, five countries in Latin America, two countries in Asia, two countries in the ENP region, one country in the Caribbean region and one country in the Pacific region.
As illustrated in the table below the sample covers:
75% of the total direct support going to decentralisation worldwide;
73% of the total funds corresponding to a bottom-up approach and 76% of the total funds corresponding to a top-down approach;
100% of the funds channelled through sector budget support.
24 Mainly based upon data from the 2008 UN World Population Prospects.
25 As this selection had to be made during the Structuring Phase of the evaluation process, it is evident that selec-
Table 6 Coverage of selected country cases Criteria Total direct support
(mEUR) Direct support in sample (mEUR) % Region All regions 789.3 581.3 74% Africa 586.4 449.9 77% Latin America 78.3 68.6 88% ENP 60.7 42.8 71% Asia 33.2 13.5 41% Other 30.8 6.4 21% Entry point Bottom-up 580.2 422.9 73% Top-down 209.1 158.4 76% Financing modality
Sector Budget Support (SBS) 113.4 113.4 100%
Moreover, the selection includes:
eleven low income, seven lower middle income and four upper middle income coun- tries;
and four fragile states.