The evaluation framework

This study is a program evaluation designed to understand the worth and value of an approach adopted in the ASLP Mango Project (Scriven 1991; Owen & Rogers 2006). The literature associated with program evaluation is vast and fragmented and generally aimed at evaluation theorists rather than practitioners (Dart et al. 1996). Worthen et al. (1997) state that each form of evaluation has its own built-in assumptions therefore the selection of an evaluation framework depends on the objective and nature of the program being evaluated.

Scriven (1967) was the first evaluator to write about two different form of evaluation such as formative and summative evaluation. Since then the terms have become almost universally accepted in the field of evaluation. Formative evaluation involves collection of information about program activities and about the delivery of a program to judge the value of the program. Summative evaluations are conducted after the completion of a program and sets out to assess whether or not the desired outcomes

46

have been achieved. However, summative evaluation may be formative when the findings are used to improve future or other existing programs (Dart et al. 1996; Owen and Rogers 2006).

The shift in focus from production to market based capacity building in rural industry development and extension projects has placed more emphasis on the systematic collection of information about the activities, characteristics, and outcomes of a program that enables evaluators to make judgments about the program, improve its effectiveness, and/or inform decisions about future programming (Van den Ban and Hawkins 1996; Coutts 1997). Consequently, an effective evaluation process needs to include both formative and summative elements. Further, Patton (2002) and Rossi et al (2004) argue that the evaluation framework should be capable of addressing the research questions that underpin the evaluation.

The choice of the evaluation framework was guided by the objectives of the research but more specifically by the research questions that were developed in Chapter 1 such as:

1. Were the ASLP mango project activities successful in changing the knowledge, attitudes, skills and aspirations (KASA) of Pakistani collaborators and the mango industry as a whole in three components of the project, quality management, marketing understanding and supply chain management over a three year of the project period?

2. To what extent did post-harvest practices change among the targeted stakeholders, and at an industry wide level, as a result of change in KASA?

3. How could the ASLP mango project activities have been better planned, implemented or modified to achieve better outcomes in the post-harvest performance of the Pakistan mango industry?

The first two research questions focus on establishing the extent to which the ASLP program activities impacted on the targeted stakeholders who were involved in the project‟s activities and to the wider Pakistan mango industry. The third research question is concerned with processes involved in the planning and implementation of the activities.

Of the range of evaluation models identified by Stufflebeam (2001) and Owen & Rogers (2006) most are output orientated. For example:

Objectives based – measures the extent to which each stated objective was reached

47

Accountability focused – requires the determination of present performance criteria and the subsequent achievement of performance against these criteria

Benefit/Cost analysis – the value of outputs generated versus the cost of achieving those outputs

Outcome evaluation – to determine if a program added value to the participants

Case studies – provide stakeholders with an authoritative in-depth, well documented explanation of the project

A management systems approach – based on auditing the processes undertaken in the execution of the project.

Of these evaluation models, this research involves the evaluation of a case study that requires an in-depth explanation of the ASLP project. Such an evaluation involves not only elements of both outcome evaluation and management systems approaches but also an examination of the impact of the program‟s activities on the intermediary variables –the knowledge, skills, attitudes and aspirations of the stakeholders, that influence project outcomes. Bennett‟s Hierarchy (Bennett 1975) shown in Figure 3.1, provides such a model.

The first four steps focus around process evaluation, while the last three steps focus on outcome/impact evaluation. Dart et al. (1998) explained the application of Bennett‟s Hierarchy in rural development and

Figure 3.1 Steps in program performance under Bennett’s Hierarchy

2. Activities

1. Resources

7. End Results

6. Practice change

5. Change in KASA

3. People Involvement

4. Reaction

Outcome evaluation

Process evaluation

48

extension programs and concluded that the approach measures the program‟s value as it proceeds as well as its overall impact which can be used to inform decisions concerning existing or future programs. They also described how monitoring and evaluation for extension program management can be linked to Bennett‟s model. For example, monitoring and evaluation at Levels 1 and 2 can be used to assess accountability while similar processes at the higher Levels can evaluate the impact of program activities on intended beneficiaries in terms of their performance, as well as on the system in which they are embedded. Using this approach, the links between the levels of Bennett‟s Hierarchy and the evaluation of the ASLP project are presented in Table 3.2.

Levels Bennett’s description

Evaluation Focus

7 End Results What have been the impacts on the Pakistan mango industry as a result of the ASLP activities?

6 Practice Change To what extent did the program activities change practices in the Pakistan mango industry?

5 KASA Change Were the program activities successful in changing the KASA of participants in areas identified?

4 Reactions How were the program activities received by the targeted stakeholders?

3 People involvement

Were the program activities well targeted? How many targeted participants became involved in the program activities? How extensive and intensive was their involvement?

2 Activities Were the program activities appropriate? Were the activities delivered appropriately?

1 Resources Did the project have appropriate resources? Was the planning process participatory?

Ban & Hawkins (1996) along with Dart et al (1998) have suggested that difficulties can arise in establishing a causal link between project activities and end results because of external influences, such as changes in government policy, which may have occurred over the course of the project. Dart et al (1998) conclude that under such circumstances it may be more appropriate to focus the evaluation of projects at Levels 5 and 6 of Bennett‟s Hierarchy where relevant data can be obtained directly from participants in the project‟s activities. This approach has been adopted in this research as indicated in the next section.

Table 3.2 Bennett’s Hierarchy and the evaluation of the ASLP project

49