Influences of Choice of Utilization-Focused Evaluation as Conceptual Framework

The multi-utility and multi-disciplinary nature of knowledge utilization has presented evaluators with multiple understandings of use as well as choices on how to use it. There are a number of influences on how knowledge is used in the context of different users. It has already been explained that use is dependent on a contextual influence. As indicated, key variables or attributes that evaluators can utilize in assessing utilization, according to Blake and Ottoson (2009:27-28), are, 1) utilization coverage that is dependent on the context of use; 2) dissemination and diffusion being the distribution or movement of information or knowledge and its processes to the intended users found to be the part of the solution of underutilization; 3) knowledge transfer through negotiations to be made accessible to users in organisations; 4) systematic implementation that enhances use and results in changed behaviours in organisations; and 5) knowledge translation as policy (Blake and Ottoson, 2009). Participatory evaluations such as the utilization-focused evaluation framework have long recognised the importance of translation.

2.15 CONCLUSION

In conclusion, this chapter outlined the literature reviewed on the utilization of evaluation information over time. It provided the research setting in the Department of Rural Development and Land Reform. It then explained what has been done in the field of evaluation utilization, what the emerging issues are and what is being done about them. It also provided different perspectives and schools of thoughts on the evaluation utilization as well as the theoretical and conceptual frameworks and models that emerged.

Brief examples of applications or studies that applied those frameworks and the results obtained were discussed.

58

The literature that has been reviewed explored an existing body of knowledge on the subject of utilization of evaluation findings. It provided the basis and the rationale of the research. It also explained what has been researched, past and current studies by various scholars and identified the gaps in the knowledge base. Furthermore, the literature outlined schools of thought and theoretical perspectives underlying the utilization of evaluation. It identified the broad field in which the study fits as well as the explanatory frameworks and models that have emerged over time. This facilitated the identification of the conceptual framework for this research as well as important attributes and variables of the research.

59

CHAPTER THREE

RESEARCH METHODOLOGY

3.1 INTRODUCTION

The research strategy for this study is qualitative. A distinction is made between quantitative and qualitative research. Bryman (2012) differentiates quantitative and qualitative methods by numbers.

Quantitative methods involve numerical values normally used in statistics to determine descriptive statistics, from population and samples associated with empirical positivist research. The qualitative strategy is associated with interpretivism and anti-positivist epistemology and it enjoys rich text and deep explanations of phenomena that cannot be analysed statistically. These are two methods that are commonly used in any research, and can also be used jointly in a single research, a method then referred to as mixed method research.

Mixed method research is commonly defined as the study that uses both quantitative and qualitative research strategies. However, it can also describe a study that combines various methods within any one of the two research strategies (Bryman, 2012:713). In evaluation research this method is advocated by scholars such as Bamberger (2010), mainly when evaluations are undertaken in less than perfect situations. This is because numbers only do not tell the story of underlying factors that affect the variables being researched. Mixed methods research can also be used as a way of triangulating information with the advantage of quantitative methodology being ease of collection and analysis of data, especially in the case where time is a limiting factor. Quantification eases the use of electronic tools that minimize the human error so as to ensure greater accuracy and credibility of the result.

60

In this chapter a detailed discussion on both these methods will be provided. Reasons will also be forwarded as to why preference is given to qualitative methods of research in this study.

There are a number of research designs that were discovered during the literature review. In the main, there are five major common categories of research design in social research, namely the experimental design, cross-sectional design, longitudinal design, case study and comparative design (Bryman, 2012; 44-75). For purposes of this study a meta-evaluation design of the cross-sectional meta-evaluation studies will be used. A meta-evaluation is defined as an evaluation of an evaluation that determines the merit or worth of an evaluation itself (Patton, 2002:211;

Wrotham, Sanders and Fitzpatrick, 1997:519). According to the OECD (2002), a meta-evaluation is described as an evaluation designed to aggregate findings from a series of evaluations. It can be used to judge the quality or to assess the performance of the evaluators (OECD, 2002).

Patton (2002: 211) asserts that one can learn a great deal about evaluation design through meta-evaluations.

This meta-evaluation focuses on a few cross-sectional evaluation studies that were conducted and verified as to whether certain processes were followed. A cross-sectional design is defined by Bryman (2012: 58-59) as a collection of data on more than one case study at a single point in time in order to collect a body of both qualitative as well as quantitative data to determine patterns of association. It is often referred to as a survey design (Bryman, 2012:58-59). In the main, the aim is to check whether the results of these evaluations were effectively utilised by the department concerned.

Patton’s (2012) framework on utilization-focused evaluations was central to this study. An account of how this framework is critical for this study was provided in the literature review chapter as a conceptual framework, with full description of the framework and the 17 steps that need to be followed for an evaluation to be deemed utilization-focused. A description of

61

Patton’s theoretical premises has also been discussed in the conceptual theory section of this report.

Primary data was collected through interviews and secondary data was obtained from documents, specifically evaluation reports. A purposive sampling method was used since the targeted population was departmental officials available to respond to questions.

Purposive sampling is defined as a non-probability form of sampling where random sampling is not necessarily critical. Its goal is to sample participants in a strategic way to ensure that those sampled should be relevant to the study (Bryman, 2012:418).