There are various approaches or types of evaluation in practice. However, this section does not attempt to elaborate on each type of evaluation comprehensively as the focus of this research is benchmarking, which will be presented thoroughly in a subsequent chapter. For this reason, the types of evaluation method discussed below are given based on their common application in planning. These are; Black Box Evaluation, Theory-driven Evaluation, Balanced Scorecard, Impact Assessment and Process Evaluation.
A widely acknowledged yet conventional type of evaluation is called black box evaluation5. Black box evaluation refers to an assessment which concentrates on the actual result of the programme but does not examine its process and the cause of those outcomes (Rogers, 2000). Rogers added that black box evaluation is more suitable when assessing a product or a standardised initiative rather than a programme (Rogers, 2000). Moreover, Chen (2005:p231) reveals that, “black box outcome evaluation, whether efficacy or effectiveness, does not systematically evaluate transformation processes…Instead, black box evaluation largely searches out information about a programme’s merits”. However, despite the lack of attention to causal investigation, this method has not been discarded. Regrettably, this researcher cannot comment conclusively on the extent of its practice at present.
20
Unlike black box evaluation, which only examines the output or outcome of a programme, the theory- driven evaluation6 approach adds an extra merit as it provides information on how the programme works or why it has not succeeded. Chen (2005:p232) describes theory-driven evaluation as an “evaluation that takes into account both the underlying causal mechanism and the implementation process when assessing the effect of a programme”. He added that, theory-driven evaluation is capable of providing stakeholders with information on the extent of the programme’s goal achievement, and constructively records causal effects to a programme’s achievement (Chen, 2005). It is therefore apparent that this method has an extra value or function as compared to the traditional black box evaluation as it investigates the causal effect of programme delivery. This method was further developed by Ray Pawson and Nick Tilley in their 1997 book, and the improved method was named the Realistic Evaluation or the Realist Approach.
Another type of evaluation which is also widely recognised is the balanced scorecard. This technique was formulated and further developed from the ‘controlling’ method by Kaplan and Norton in the early 1990s. The principle of the balanced scorecard focuses on the management aspects and is used by companies to measure and assess their performance (Stockmann, 2011c). The method informs corporate strategy by looking at the financial aspects, customers’ views, internal business operation, and knowledge (Stockmann, 2011c). The difference between the balanced scorecard and other evaluation methods is that it concentrates on the factors that can stimulate the company’s profit as well as other profit-associated indicators. To set down the balanced scorecard, the approach includes identifying the strategic process, concern with the financial side and target, as well as drawing up the causal factors to the outcome of the operation. The concept and application of the balance scorecard is illustrated below (see Figure 2.2).
Based on Figure 2.2 below, it is obvious that the balance scorecard’s approach to evaluation is interrelated between the central vision and the subjects of evaluation. As described by Stockmann (2011c), the assessment focuses on the core aspects which can accelerates the company’s profit. For that reason, the balance scorecard is normally conducted by examining the financial aspects, strategic approach, customer appraisal and the development of the company.
21
Figure 2.2: Networked balanced scorecard model
Source: Adapted from Kaplan and Norton (1997:p9), cited by Stockmann (2011c:p69)
Meanwhile, Impact Assessment is carried out to examine the observed outcome of a programme and is “likely to involve quantitative rather than qualitative data” (Hall and Hall, 2004:p45). According to GAO (2011), impact assessment examines the net effect of a programme. This is possible by comparing the forecasted output without the existence of the programme as compared to the actual programme outcomes. This approach is therefore very similar to black box evaluation due to its nature of looking at the end-result only. Clearly, by applying this method, the degree of programme effectiveness is not measured too but rather the focus is on the impact of an intervention.
Process evaluation on the other hand is performed with the purpose of understanding the programme delivery process and “judging the merits of the implementation process” (Chen, 1996:p124). The idea behind this approach is to investigate the internal processes in order to develop the strategic approach for the delivery of programme. It is normally performed to examine a programme’s “activities conformance to statutory and regulatory requirements, program design, and professional standards or customer expectations” (GAO, 2011:p2). Furthermore, Chen (2005) disagrees that process evaluation is to be conducted at a programme’s mature stage, which is why Hall and Hall (2004) concluded the activity as a ‘formative evaluation’. In relation to this, they (Hall and Hall, 2004; Chen, 2005) suggested the use of qualitative data especially through interviews and observations as they can provide rich and in-depth data about the programme process. In addition to that, Chen (2005) classifies process evaluation into two types which are the identification of implementation issue (also known as the ‘development-oriented process
Vision and Strategy
Internal processes
How must the business processes be organised?
Customers
How should we behave toward customers?
Finances
To which financial results will the strategy lead?
Perspective for learning and development
Which learning and innovation processes are necessary?
22
evaluation’) and examination of programme efficiency (otherwise known as ‘assessment-oriented process evaluation’). From another viewpoint, Royse et al., (2010) mentioned that, the reason to conduct process evaluation is to examine the achievement of a programme and to evaluate why it had or had not met its intended outcome(s). Moreover, they believed that an advantage of process evaluation is that it enables the programme to be documented, thus allowing the programme to be replicated, which also makes it beneficial for future reference.