Figure 6 provides an overview of the researchers data collection and analytical framework. The diagram is essentially into the three phases of research. The first field level work was part of the original evaluation fieldwork. The second phase at field level was pulling together a daily workshop to come to evaluative judgements under the researchers leadership. The third phase is a desk review undertaken by the researcher herself. These three phases and methodologies are examined in more detail below.
Phase I demonstrates the methods used at each field site. Oral histories were collected from two members of each Elder’s Committee. Members of the Elders’ Committee were used as these are some of the oldest members of the community and provide both a personal history and then a history of the community, village or town, which offers great insight and depth into the community. Focus group sessions were held with each Water and Environmental Sanitation (WES) Committee; a community focus group and the women’s committee. Most villages and towns in Somalia have established women’s groups, which were used to form the women’s focus group. The community group comprised largely of men and included a range of representatives from each wider family group. These groups included representatives from some of the poorest households in each location. Semi-
structured interviews were held with one male and one female member of the WES committee to gain greater insight into the personal accounts of the project and also to help triangulate the findings of the focus group sessions. Three semi-structured interviews were held with beneficiaries at each project site. Transect walks, mapping and observations were conducted at each site and further information on each method can be found in Table 8 Participatory Research Tools
An interpreter was used for the oral histories, semi-structured interviews and focus group sessions. There are a number of limitations of using an interpreter (Jandt, 2003) including the potential loss of conceptual equivalence. This is where the interpretation is not conceptually or technically representative of what the participant has said. Edwards (1998) explains that an incorrect interpretation of one word can lead to different themes emerging and subsequently a different analysis. The researcher does not speak Somali and the participants do not speak English thus, there is no independent validation of the interpretation. These limitations were minimised through a number of measures. Firstly, the interpreter was highly qualified with extensive experience of conducting field evaluations. Secondly, the interpreter had excellent references, was from a neutral clan and had nothing to gain from the evaluation or research findings. Finally, the researcher would observe participant body language and tone of voice to ensure it married with the interpretation given.
Phase II demonstrates the use of evaluative criteria. The evaluations for this research were analysed using a framework of the OECD-DAC evaluation criteria, which have previously been discussed (see Table 3). Table 13 presents this framework adapted from O’Keefe et al. (2002) and ALNAP (2006, which has been used to assess these case studies. The criteria are grouped into five major categories – relevance/appropriateness; effectiveness; efficiency; impact; and sustainability. Other over-lapping criteria including, coherence, timeliness, coverage, coordination and connectedness were considered sub-issues of these five major criteria. Table 14 summarises the themes for each case study whose results are detailed in Chapter 5 and
Chapter 6. Each evening the evaluation team would hold a workshop to discuss all of the findings from that day and explore them through the evaluative criteria. There are a number of limitations to this process of teamwork. Firstly as the OECD-DAC (1999) discussed there is the potential to disagree over findings. Prior to commencing the fieldwork the team agreed to conduct evening workshops every night in the field. Therefore, if there were any disagreements or uncertainties regarding findings, the team would have the following day to clarify the findings. Secondly as Barry et al. (1999) highlighted there is potential for team members to simply agree with other members to please the team. Finally, any personality clashes amongst team members can significantly hinder the analysis process. All team members were selected for their reputations and experience as quality professionals with excellent research and evaluation skills. It is believed all team members would put integrity and professionalism ahead of simply wanting to please other team members or equally causing disruptions to the teams. Furthermore, all team members agreed ahead of the research that the most important aspect is ensuring the voice of the beneficiaries is heard.
Phase III was conducted solely by the researcher once all fieldwork was completed. The researcher analysed and examined all the findings and began grouping exploring them through a number of key themes. This provided an additional level of insight and depth into the findings to ensure the researcher gathered a holistic understanding of the situation on the ground. This phase of analysis is open to subjectivism and misinterpretation by the researcher. These limitations were minimised through the use of mixed methods and multiple data sources in phase I and then the teamwork analysis conducted in Phase II. Hence, the researcher has made every effort to triangulate data sources and findings and minimise all of the research limitations.
Table 13 Evaluation Criteria Framework Criteria
Assessed What to Measure
Whose
Perspective Methodological Challenge Key Questions Relevance or
Appropriateness (coherence)
Relevance and
appropriateness in relation to local needs, priorities and donor policies. Appropriateness of increasing ownership. The society including women, men, girls and boys.
Lack of consensus regarding needs and priorities.
Understanding the perspective of different stakeholders
Are project aims and methods in keeping with local needs and
priorities? Are projects and methods of implementation culturally
appropriate?
Efficiency (timeliness and coverage)
The delivery of aid and implementation of projects.
Donor and implementing agency.
What standards to use a reference point.
Were aims achieved within allocated timeframes and deadlines? What were the alternative options for delivery (could it have been quicker, cheaper)?
Effectiveness (coordination)
The achievement of project aims.
Target groups and beneficiaries
Multiple aims; clarifying the extent to which something was effective; examining whether effects were equal across beneficiaries and groups.
To what extent have agreed aims and objectives been achieved or not? What are the reasons for achieving or not achieving agreed aims?
Table 13 Evaluation Criteria Framework Continued
(Source: Adapted from O’Keefe, 2002 and ALNAP, 2006)
Impact Intended and unintended positive and negative impacts. Wider effects of the project – social, economic, technical and environmental.
The society including
individuals, gender and age groups, communities and institutions
Linking cause and effect.
What are the positive and negative effects on individuals, groups and the wider society? What caused these effects (both intended and unintended)?
Sustainability (connectedness)
The longevity of the project and its associated benefits.
The society. Hypothetical answers. What is the agency exit strategy? Do beneficiaries have the skills, resources and will to maintain the project?
Table 14 Key Themes
Research Theme Assessment Criteria Source
ACT NCA
Context of research Literature review Literature Literature
The concepts of adaptive capacity, resilience and vulnerability
Literature review Literature Literature
Community participation and ownership Relevance/ appropriateness, effectiveness, impact, sustainability
Project documentation; interviews; focus group sessions with stakeholders including target groups, beneficiaries and implementing partners.
Project documentation; interviews; focus group sessions with stakeholders
including target groups, beneficiaries and implementing partners. Humanitarian Assistance Relevance/ appropriateness, effectiveness, impact, sustainability
Project documentation; interviews; focus group sessions with stakeholders including target groups, beneficiaries and implementing partners.
Project documentation; interviews; focus group sessions with stakeholders
including target groups, beneficiaries and implementing partners.
Somalia: Judgement on the uniqueness of the case
Comparative literature review, against authors’ evaluation experience.
Table 14 Key Themes Continued Livelihood security Relevance/ appropriateness, efficiency, effectiveness, impact, sustainability
Project and secondary literature; interviews; focus group sessions with stakeholders including target groups and beneficiaries. Government policies and other
implementing agencies.
Project and secondary literature; interviews; focus group sessions with stakeholders including target groups and beneficiaries. Government policies and other implementing agencies. Entry and exit strategies Relevance/ appropriateness, sustainability
Project documentation; interviews; and focus groups with project staff, beneficiaries and implementing partners.
Project documentation; interviews; and focus groups with project staff, beneficiaries and implementing partners.