Qualitative and quantitative methods in impact evaluation

Impact evaluations are carried out as they play a very essential role so that certain outcomes can be identified, better targeting is ensured, and effectiveness and efficiency of policies could be better assessed. Impact evaluation has been the area of research interest by the Center of Global Development (2006). Impact evaluation gap needs to be filled given that such evaluations improve effectiveness of development policies and accountability (Department for International Development [DFID], 2008). Impact evaluations moreover contribute to public good, both at national and global level. International participation for impact evaluation is also necessary. Other guidance ensures that impact evaluations are carried out in a way which is relevant to policy by asking the right questions in a timely manner seeking to determine the reason “why something worked or did not work, not just whether it worked or not”. This can be achieved by adopting combined quantitative and qualitative methods (DFID, 2008). Ethics play an important role in impact evaluations. It’s a well-known fact that constraints of resources undoubtedly lead to choices to be made and some groups are denied coverage for the sake of evaluation.

The DFID argues that impact evaluation should be carried out in its widest form and context and should be informed of evidence base decision making, where results are monitored and evaluated. This implies that monitoring results should focus on: o outcomes – what is being achieved

100 o inputs – what are the resources employed.

The Paris Declaration for Aid Effectiveness, which is driven by international consensus for achieving the Millennium Development Goals evaluate policies and their impacts on determining “what works, what doesn’t, where and why” while examining also who are the end beneficiaries from these policies (DFID, 2007). Such evaluations of policies make agencies more accountable to citizens. DFID (2007) reports that both the Network of Networks on Impact Evaluation (NONIE) and the International Initiative for Impact Evaluation (3ie) are two emerging entities with the remit of evaluating impacts of policies conducting design, analysis and review of research proposals in the process.

Garbarino and Holland (2009) argue on the “more and better” impact evaluations when combining qualitative and quantitative methods. A mixed method approach ensures that different impacts on different groups are measured and dimensions that are not readily quantified can be better evaluated. One very important belief in impact evaluation assessments is that to what McGee (2003, 135) refers to as “equality of difference”. This emerges as a key principle to address the aspect of selection bias of the research in determining the attribution factors (Center for Global Development, 2006). This methodological stance leads to a certain degree of shift from the norm in the development research where empirical research is then followed by qualitative research. Garbarino and Holland (2009) thus contends that contextual studies should play a rigorous role in impact evaluation and observation rather than confirming findings of ‘non-contextual’ surveys and recommend a qualitative approach to sharpen the focus of the policy process in which impact evaluations are embedded.

Five guiding principles were identified by DFID (2008) in a policy paper addressing impact evaluation. These included better quality impact evaluation, coordination at international level, relevance, ethical considerations and the use of appropriate methodologies. It is argued that impact evaluations should be nationally owned, embedded to lead to determine actions and changes required.

101

Ways of combining approaches based on qualitative and quantitative methods have been studied by Garbarino and Holland (2009) where different approaches are presented in a way of adding value in assessing impact evaluation. Hentschel (1999) had presented researchers with method-data framework identifying approaches varying from the more contextual approaches and less contextual approaches in both qualitative and quantitative approaches.

The ‘non-contextual research methods’ are more applied to achieve a broad coverage adopting random surveys where data is produced quantitatively which when analysed can produce behaviour patterns and relations.

On the contrary ‘contextual research methods’ are based to a specific setting whereby issues are explored in more depth and detail. In such methods, data is focused on the area being studied rather than the broader data achieved in the non- contextual methods (Booth et al.,1998). This research method generally employs ethnographic techniques including observation of participants and interviews. In such approaches, open ended questions will lead to capture judgements which are non-quantifiable.

The method data framework presented by Hentschel (1999) refers to the participatory research, based on the more qualitative and contextual approach whereby the data generated and its analysis are underpinned by the local knowledge and ownership (Chambers, 1994, 1997). The main component of this participatory research method is the component of ownership which will provide the necessary room for the local people to determine an analytical framework of their own. The participatory research method, in contrast to qualitative techniques which adopt individualised observation, focuses on the collectiveness where the public reflection and action become key informants of the participatory research method. This method provides data to researchers to understand the local context, whilst respecting the principles of ethics throughout the entire research. The participatory research thus

102

is a process, which in itself promotes the involvement of the population from the passive role to the active role (Garbarino and Holland, 2009).

This method can provide researchers with both quantitative and qualitative data. While data is produced, and adopted in its context, it can be scaled through participatory surveys. This method has the advantage of providing data in a quick and efficient manner ready for evidence-based analysis. Garbarino and Holland (2009) contend that it is possible, from participatory research methods that generate statistics for serious consideration by policy makers. This is based however on a key requirement that the sample informing the data is sufficiently large that can be considered for analysis at national level inference. Another important requirement is that the data is gathered from a representative sample with the aim to represent the population of interest, thus it has to be generalised. The data should also be suitable for standard statistical analysis with capability of the statistics to inform policy at national level.

Impact evaluations, combining qualitative and quantitative methods have been extensively carried out in social sciences. The World Bank Poverty Assessments during the 1990s as well as the analytical frameworks in Poverty and Social Impact Analysis have provided hand on experiences in employing methods and evaluations for policy research. Innovations in methods for evaluation throughout the years have provided policy development research (Gabraino and Holland, 2009).

Adopting quantitative and / or qualitative methods should be informed on an intensive consideration of the comparative advantages of the research methods on one another, (Gabraino and Holland, 2009), and to recognise that “strong fences make good neighbours” (Appleton and Booth, 2005). Attributes of qualitative and quantitative methods have been widely developed by scholars, researchers and policy makers. There has been also a convincing recognition of the powerful methods emerging from the combination of these two methods. It is however crucial at what levels and in what sequence these methods are combined to optimise these tools within the research process. (Garbarino and Holland, 2009). Three ways have

103

been identified by Carvalho and White (1997, p.18) for combining methods in research approaches:

1. Integrating methodologies for better measurement, 2. Sequencing information for better analysis and, 3. Merging findings for better action.

Table 5.2 indicates the differences between a qualitative oriented approach versus a quantitative oriented approach (Kanbur, 2003).

Table 5.2: Main differences between a qualitative oriented approach versus a quantitative oriented approach (adapted Kanbur, 2003)

‘Qualitative’ oriented research ‘Quantitative’ oriented research Non-numerical information Numerical information

Specific (contextual) population inclusion

General (non-contextual) population inclusion

Active population inclusion Passive population inclusion Inductive inference methodology Deductive inference methodology Broad social sciences disciplinary

framework

Neo-classical economics (and natural sciences) disciplinary framework

5.3.6.1 Challenges of impact evaluations

Impact evaluation requires intensive resources. Whilst it is acknowledged as public good as it is largely beneficial to the society mainly for its knowledge to the international community, the costs incurred to carry out the studies are borne by an agency or a government. Other challenges of impact evaluation are the need of having better quality impact evaluations. It is worth noting that impact evaluations have shifted from project base to sectorial base or government base. Gunning (2006) remarks that gaps have been identified between project-based methods of impact evaluation and the instruments required to evaluate impacts.

104

5.3.6.2 Impact evaluation in this research

Combining qualitative research with quantitative instruments that have greater breadth of coverage and generalisability can result in impact evaluations that make the most of their comparative advantages. Therefore, the following sequence was adopted to assess impact evaluation in this research study:

1. A literature review was carried out which informed the research on existing and proposed policies.

2. Identified policy documents highlighted key and priority issues that had to be addressed by the mixed methods research.

3. The quantitative survey method identified knowledge gaps which were key informants to the interviews.

4. Results of questionnaires enabled to determine which issues and which opinions are important to the local council population, and what are the reasons and explanations that participants give and informed the interview guides.

5. The quantitative and qualitative studies provide help to define areas in sub-group and sampling frames.

6. The quantitative study was therefore carried out first to generate the ‘working hypothesis’ that can be further examined through qualitative research with specific pre-defined questions and prompts.

7. This was followed by qualitative research to explain relationships, trends and patterns that emerged from the survey. Therefore, the main purpose of qualitative investigation in this research is for triangulation purposes.