Realistic evaluation is believed by some to have been emanated from the belief in ‘truth’ and therefore perceived as a belief and not a practice (Sayer, 1992; Kazi, 2003). The theory stemmed from another theory of evaluation called theory-driven evaluation (Ho, 1999). Realistic evaluation was actually further developed by Ray Pawson and Nick Tilley in 1997, as an improvement of the existing theory-driven evaluation. To understand more of the theory-driven evaluation and the creation of realistic evaluation, the succeeding explanation describes.
4.3.1 Theory-Driven Evaluation
Theory-driven evaluation, also known as ‘white box’ evaluation (or the ‘clear box’ evaluation) was developed as a result of the issues brought by the conventional type of evaluation; black box evaluation also known as the method-driven approach (see Chen, 1990; Astbury and Leeuw, 2010). It was claimed
64
by Ho (1999) that theory-driven evaluation was initiated by Huey-Tsyh Chen and Peter H. Rossi in 1983. Intrinsically, black box evaluation was a prevalent approach to evaluation in the 1960s and 1970s, otherwise known as “black box impact assessment” (Chen, 1990:p8). Unfortunately, black box evaluation generally focuses on the merits or outcomes of a programme, but does not divulge the cause of achievement or the reason behind the unsuccessful results (Chen, 1990, 2005; Rogers, 2000; Astbury and Leeuw, 2010). Apparently, black box assessment only provides information on whether a programme succeeded or failed, and not the explanation and rationale for its performance or delivery (Lipsey and Pollard, 1989). Chen and Rossi (1983) observed the limitation of black box evaluation as described below:
“In the evaluation literature there has been no dearth of interest in implementation, but too much of the attention has been given to worrying about whether programs have been delivered as intended, and not enough attention has been given to understanding the process of implementation” (Chen and Rossi, 1983:p296-297).
In relation to Chen and Rossi’s (1983) idea, Bledsoe and Graham (2005:p307) defined theory-driven evaluation “as using a synthesis of both stakeholder program logic and social science theory to define what a program does, in what manner, and how much of an effect each goal and objective can have on the outcome”. To Bledsoe and Graham (2005), the key idea to the theory-driven evaluation had enabled the assessor to determine the causal mechanisms of a programme. According to Astbury and Leeuw (2010), theory-driven evaluation encompasses two important elements; it anticipates the improvement of the current issues from the proposed programme and presupposes that the purpose and delivery of a programme can be effective in achieving improvement. Because of this, the newly derived theory-driven evaluation provides the answers in various situations, for various stakeholders involved (Astbury and Leeuw, 2010).
4.3.2 The Emergence of Realistic Evaluation
Even though theory-driven evaluation overcame the drawbacks of the conventional black box evaluation, yet, it was also realised that theory-evaluation was still lacking in certain aspect. It was obvious that, the method does not examine the cause for programme outcomes and who gains the benefit from programme’s achievement. It was therefore realised too that, understanding just the end results and formulating recommendation for improvement were not enough. The idea of the theory-driven evaluation is to therefore seen as to just uncovering the questions of ‘how and what’ but not ‘how, what, and who?’ In relation to this issue, it was then believed that it would be more effective to understand; (1) what had driven the achievement of a programme or caused to failure?; (2) what means or procedure or approach suits better?; (3) for whom these measures are effective?; and (4) under what circumstances the approach is more efficient?. For this matter, the creation of realistic evaluation which focuses on the mechanisms of
65
a programme had therefore fulfilled the needs to examine those measures. However, realistic evaluation is perceived as a framework to accommodate the process of assessing a programme and not an evaluation method (Pawson and Tilley, 1997; Pawson, 2002). This was distinguished through a formulation of a specified framework named the Context-Mechanism-Outcome (CMO) configurations. Pawson (2002:p340) stresses, “The realist approach is not an evaluation technique as such, but a framework for the whole enterprise. It is being developed to encompass programmes of all types (local to transnational) and evaluation tasks of all modes (developmental to impact assessment)”. The major difference between theory-driven evaluation and realistic evaluation is that realistic evaluation embarks on investigating the elements or factors that trigger or activate the mechanism, as well as understanding the efficiency or effectiveness of a programme from its outcomes through its contextual angle; meaning, under which context(s) that outcomes were produced and with what mechanism(s)? As pointed out by Pawson and Tilley (1997:p57) “Programmes work (have successful ‘outcomes’) only in so far as they introduce the appropriate ideas and opportunities (‘mechanism’) to groups in the appropriate social and cultural conditions (‘contexts’)”.
In short, the difference between the three, black box evaluation, theory-driven approach and realistic evaluation, is that realistic evaluation seeks to understand the mechanisms of how a programme achieves its outcomes, while black box evaluation only examines the outcome of a programme without investigating how and why the intervention or policy worked or failed, and theory-driven evaluation is only revealing the causal effect to the outcomes.