CHAPTER III E VALUATION IN O RGANISATIONS
3.6 Types of evaluation
Evaluation literature and practice describe and offer a wide range of different alternatives to conduct evaluations. Different types of evaluations ask different questions, concentrate on different purposes. In turn the development of evaluation questions is closely linked to respective evaluation purposes and types (Patton, 2008). Patton (2008) provides an ex-tensive overview of 79 types of evaluations and states that various options can be com-bined and used jointly within the same evaluation, or alternatives can be implemented in sequence over a period of time, such as implementation evaluation preceding an out-comes evaluation or summative evaluation following a formative assessment. Mertens &
Wilson (2012) mention that most evaluations pursue multiple purposes. The decision to select an evaluation type is based on the purpose of the evaluation which influences the evaluation strategy adopted, what is being or should be evaluated as well as when the evaluation should be conducted. The table below presents a selection of evaluation types based on Patton’s comprehensive list. These types are chosen as the questions or the approach followed could to some extent also be asked and used within the context of the case being examined in this work.
Table III-13: Focus or type of evaluation
Focus or type Defining question or approach Accountability
focus
Have resources been appropriately used to accomplish the intended results? Key issue: Who is accountable to whom for what? (Rogers, 2005)
Appreciative inquiry
What is best about the programme? (Preskill, 2005) Beneficiary
assessment
The perspective of intended beneficiaries about what they have expe-rienced, both processes and outcomes (Salman & Kane, 2006).
Compliance focus Are rules and regulations being followed?
Context focus
What is the environment within which the programme operates politi-cally, socially, economipoliti-cally, culturally, and scientifically?
How does this context affect program effectiveness?
Cost-benefit analysis
What is the relationship between programme costs and programme outcomes (benefits) expressed in dollars? (Levin, 2005a)
Cost-effectiveness analysis
What is the relationship between programme costs and outcomes where outcomes are not measured in dollars? (Levin, 2005b) Criterion
focused
By what criteria (e.g., quality, cost, client satisfaction) should the pro-gramme be evaluated?
Critical issues focus
Critical issues and concerns of primary intended users focus the eval-uation.
Focus or type Defining question or approach
Decisions focus What information is needed to inform specific future decisions?
Descriptive focus
What happens in the programme? (No “why” question or cause-effect analysis)
Effectiveness focus
To what extent is the programme effective in attaining its goals?
How can the programme be more effective?
Efficiency focus
Can inputs be reduced and still obtain the same level of output or can greater output be obtained with no increase in inputs?
Effort focus
What are the inputs into the programme in terms of number of per-sonnel, staff/client ratios, and other descriptors of levels of activity and effort in the programme?
Formative evaluation
How can the programme be improved?
Goals-based focus
To what extent have programme goals and intended outcomes been attained?
Goal-free evaluation
To what extent are actual needs of programme participants being met (without regard to stated programme goals)?
Impact evaluation
What are the direct and indirect programme impacts, over time, not only on participants, but also on larger systems and the community?
Impact evaluation often includes a focus on determining the extent to which results can be attributed to the intervention.
Implementation focus
To what extent was the programme implemented as designed?
What issues surfaced during implementation that needs attention in the future?
Inputs focus
What resources (money, staff, facilities, technology, etc.) are available and/or necessary?
Judgement focus
Make an overall judgement about the programme’s merit, worth, and/or significance (see also summative evaluation)
Knowledge focus, learning-oriented evaluation
What can be learned from this programme’s experiences and results to inform future efforts?
Focusing the evaluation on practice improvement and organisational learning (Rogers & Williams, 2006).
Monitoring focus, monitoring and eval-uation
Routine data collected and analysed routinely on an ongoing basis, of-ten through the management information system
Integrating monitoring and evaluation (Kusek & Rist, 2004; Jackson, 2005)
Outcomes evaluation
To what extent are desired client/participants outcomes being at-tained?
What are the effects of the programme on clients or participants?
Process focus
Evaluating the activities and events that occur as part of implementa-tion: What do participants experience in the programme?
What are the strengths and weaknesses of day-to-day operations?
How can these processes be improved?
Real-world evaluation
How can evaluation be done under budget, time, date, and political constraints? (Bamberger, Rugh & Mabry, 2006)
Summative evaluation
Should the programme be continued?
If so, at what level?
What is the overall merit and worth of the programme?
Utilisation-focus evaluation
Intended use by intended users: What information is needed and wanted by primary intended users that will actually be used for pro-gramme improvement and decision-making? (Patton, 2008) Source: Selected from Patton (2008, pp.300-305)
Within Section 3.1: The nature of evaluation, formative and summative evaluation are in-troduced as main categories. Within these two, there are more types and approaches, which can be conducted to serve their respective purposes. In the following subchapters those types are outlined which are mainly relevant to this work.
3.6.1 Case study evaluation
This section is more about the evaluation part of case study research. The chapter in the methodology section (4.3) deals with case studies more comprehensively as a re-search strategy.
Case study evaluation is highly appropriate and useful in the context of programme eval-uation (Stufflebeam & Shinkfield, 2007). It is concerned with in-depth examination, illumi-nation, description, analysis, and synthesis of a particular programme in its totality or components of it (Stufflebeam & Shinkfield, 2007). The main focus of this kind of study is to portray and make the programme understandable (Stake, 1995), rather than guiding its development or assessing and judging its merit or worth. Therefore it looks at multiple programme levels and aims for a holistic overarching view of the programme (Stufflebeam & Shinkfield, 2007).
The intention of a case study evaluation approach is not concerned with controlling the programme in any way but looking at it as it is or occurred in the past. Consequently, it can be conducted in the course of a programme or retrospectively (Stufflebeam &
Shinkfield, 2007). Case study evaluation is trying to understand the complexity of a sin-gular case with the driving question “What can be learned about the case under exami-nation” (Stake, 2005). Therefore, the evaluation analyses and describes the case as comprehensive as possible. In order to gather relevant data and information, it looks at the characteristics of the programme, its context with respective influences on it, goals, and plans. Further, it considers unique features and noteworthy actions, use of inputs and resources, internal operations and any other processes producing outcomes, in-tended as well as unexpected effects, achievements and outcomes.
In addition, needs, strengths and weaknesses, successes, problems, disappointments or failures are looked at (Stake, 1995; Stufflebeam & Shinkfield, 2007).
In order to gather relevant data and to interpret findings the researcher or evaluator needs to identify, interact, and engage with the programme’s stakeholders having been involved in the programme in different ways (Stufflebeam & Shinkfield, 2007).
Case study evaluation often deals with qualitative and subjective information (Stake, 2005) and systematic procedures for analysis. A thorough case evaluation provides stakeholders with an authoritative, in-depth, and well-documented explication of the pro-gramme, including judgemental information, perceptions held by different stakeholders and experts, and summary conclusions. Programme sponsors, managers or other stakeholders might use this information to understand the programme, for decision-making and/or taking actions for programme improvements (Stake, 2005; Stufflebeam &
Shinkfield, 2007).
As a summary, the questions in the table below present the variety of questions a case study evaluation can address.
Table III-14: Case study evaluation questions
Case study evaluation questions
What is the programme in concept and practice?
How has it evolved over time?
How does it actually operate to produce outcomes?
Who are the players and what do they do?
What has it produced?
What are the shortfalls and negative side-effects?
What are the positive side effects?
In what ways and to what degrees do various stakeholders value of the programme?
To what extent did the programme effectively meet beneficiaries’ needs?
What were the most important reasons for the programme’s successes and failures?
What are the programme’s most important unresolved issues?
How much has it cost?
What are the costs per beneficiary, per component, per line item, and per year?
What part of the programme has been successfully transported to other sites?
How does this programme compare with what might be called critical competitors?
Source: Based on Stufflebeam & Shinkfield (2007, pp.182-183)
3.6.2 Programme, implementation or process evaluation and monitoring
Dealing with the topic of implementation evaluation the literature often also refers to programme process evaluation or monitoring (Rossi, Lipsey & Freeman, 2004;
Stufflebeam & Shinkfield, 2007; Patton, 2008; Russ-Eft & Preskill, 2009; Mertens &
Wilson, 2012). The latter is dealt with at the end of this section. The former two are used interchangeably.
Patton defines programme evaluation as
“[…] systematic collection of information about the activities, characteristics, and results of programs to make judgements about the program, improve or further develop program effectiveness, inform decisions about future pro-gramming, and/or increase understanding […] (Patton, 2008, p.39).”
As defined by Davidson (2005) the evaluation of the process of a programme takes a critical look at the quality or value of everything about the programme, what it is and does, except outcomes and costs. Process evaluation is about verifying to what extent a programme is conducted and operates as planned (Scheirer, 1994; Mertens &
Wilson, 2012).
Implementation evaluation or programme process evaluation are often at the core of formative evaluations providing information and feedback to programme managers and sponsors (Rossi, Lipsey & Freeman, 2004). It is useful when a new programme is being implemented, an existing programme does not produce satisfactory deliverables or re-sults to identify positive as well as negative aspects of its implementation (Stufflebeam &
Shinkfield, 2007; Mertens & Wilson, 2012). Furthermore, Mertens & Wilson (2012) state that such an evaluation can also be conducted to reassess a programme’s appropriate-ness under changing conditions, examining stakeholders’ perceptions and/or experienc-es. The resulting information provides insights into a programme’s effectiveness (Mertens & Wilson, 2012). Patton (2008) notes that process evaluation focuses on inter-nal dynamics and operations in order to understand strengths and weaknesses of a pro-gramme. A process evaluation is interested in what happens in a programme and why as well as how participants experience and perceive the programme (Patton, 2008).
Therefore, it looks for how a programme is operating in day-to-day reality producing out-comes (Patton, 2008). This kind of evaluation looks at both formal activities and antici-pated outcomes but also scrutinises informal patterns and unexpected consequences in the full context of the programme implementation. In order to gather relevant data, pro-cess evaluation includes different perspectives from people involved in the programme providing unique insights from their understanding and experiences. Finally, it is about searching for major patterns and seeking explanations for success, failures, and changes in a programme (Patton, 2008).
In cases where process evaluation is an ongoing activity involving recurring measure-ments over time the term programme monitoring is used (Rossi, Lipsey & Freeman, 2004). The PMI (2013a) defines monitoring as follows:
“Monitoring and tracking is the process of tracking, reviewing, and regulating the progress to meet the performance objectives defined in the project man-agement plan. Monitoring is an aspect of project manman-agement performed throughout the project. Monitoring includes collecting, measuring, and dis-tributing performance information, and assessing measurements and trends to effect process improvements. Continuous monitoring gives the project management team insight into the health of the project, and identifies any ar-eas that may require special attention (PMI, 2013a, p.89).”
Monitoring is the systematic and continual documentation of key aspects of programme performance where programme operations during the implementation phase are as-sessed compared to initial plans (Rossi, Lipsey & Freeman, 2004; Mertens & Wilson, 2012). Programme monitoring provides routine data on programme completion rates, participation levels, indicators, issues as they arise and other programme characteris-tics such as status of programme implementation (Patton, 2008; Mertens & Wilson, 2012). Referring to Owen (2006) monitoring typically takes place when a programme is well-established and ongoing. It is often associated with a need to communicate the success of a programme. The information serves primarily the needs of the manage-ment and main stakeholders’ (sponsor, programme leader or programme managemanage-ment team) as early indications of progress, lack thereof, achievements, and results (Mertens & Wilson, 2012).
3.6.3 Outcome and impact evaluation
Another category for conducting evaluations to assess a programme’s effectiveness comprises outcome as well as impact evaluations (Mertens & Wilson, 2012). These kinds of evaluations are useful to illustrate that a programme is/was or is/was not achieving its goals potentially leading to subsequent decisions and actions such as granting additional funding, making revisions or replications (Russ-Eft & Preskill, 2009; Mertens & Wilson, 2012). According to Mertens & Wilson (2012) the emphasis of both forms are most simi-lar to the concept of summative evaluation (see also Section: 3.1.2 Summative evalua-tion; 3.1.3: Relationship between formative and summative evaluation).
Outcome evaluation focuses on short-term results as it is about continuously measuring intended outcomes of a programme (Rossi, Lipsey & Freeman, 2004; Mertens & Wilson, 2012). According to the “Programme Manager’s Planning Monitoring & Evaluation Toolkit” (United Nations Population Fund, 2004) outcome evaluation measures the extent of achieving a programme’s outcome, assesses inherent reasons for success or failure, and identifies critical lessons learned and recommendations to improve performance.
The purpose of an outcome evaluation is to leverage the understanding of intentional changes in knowledge, skills, attitude, and practices resulting from a programme and its implementation (Russ-Eft & Preskill, 2009).
On the other side, impact evaluations, also often termed impact assessment, look at the effects of a programme and its long-term results (Rossi, Lipsey & Freeman, 2004). Pat-ton (2008) states that this often includes the determination of the magnitude to which re-sults can be assigned to the programme or components of it.