The word ‘programme evaluation’ does not possess a regulated meaning (Rutman, 1980) but does have a role in judging a programme (Posavac and Carey, 1989; Weiss, 1998; Hall and Hall, 2004). More than 20 years ago, programme evaluation was considered as something innovative and inspiring (Posavac and Carey, 1989) but today, programme evaluation is a common procedure in many institutions and organisations. When examining a programme, Hall and Hall (2004:p6-7) thought that “usually evaluation focuses on the aims of a programme and investigates to what extent the intentions of the programme providers are being realised”. For this reason, Posavac and Carey (1989) pointed out that programme evaluation should not be mistaken for any other evaluation research, or any particular assessment. This helps to explain why Rossi et al. (2004:p16) defined programme evaluation as “the use of social research methods to systematically investigate the effectiveness of social intervention programmes in ways that are adapted to their political and organisational environments and are designed to inform social action to improve social conditions”.
It is therefore widely acknowledged that the purpose of performing ‘programme evaluation’ is to examine the programme’s effectiveness (Rutman, 1980; McDavid and Hawthorn, 2006). Posavac and Carey (1989) believed, regardless of how many motivations or sub-purposes in conducting a programme evaluation, nonetheless, programme evaluation has two distinctive reasons and these were adapted from Scriven of 19677. They (Posavac and Carey, 1989) supposed that programme evaluation can be performed as an attempt at formative evaluation8 and the other reason as being the summative evaluation9. The former is carried out at an initial stage of the programme, in order to assess whether the programme is feasible, or should be continued, or should not be implemented, or which programme should be preceded. The latter is done at the end of a programme with the aim of investigating the input(s), process, output(s) and outcome(s) of the programme and to understand the extent of its goal achievement and effectiveness (Posavac and Carey, 1989).
7 Michael Scriven introduced the summative and formative evaluation in 1967, when explaining the main functions of evaluation. 8 Formative evaluation is further described in Chapter Five, under Section 5.7
23
McDavid and Hawthorn (2006:p15) distinguished one of the key questions that is expected from a programme evaluation is “to what extent, if any, did the programme achieve its intended objectives?” From another angle, Stormer and Schubert (2007) mentioned that a programme evaluation is carried out in order to assess the relationship between policy and projects, as well as its effect. Meanwhile, Posavac and Carey (1989:p8) identified some of the reasons for programme evaluation as:
“fulfilment of accreditation requirements;
accounting for funds;
answering requests for information;
choosing among possible programs;
2.4.1 The Difference between a Project and a Programme
Prior to the aim and objectives of the research, the study is intended to examine urban regeneration initiatives through regional park programmes. In order to better understand the concept of programme evaluation, it is useful to be familiar with the difference between a project and a programme. A project and a programme are naturally different in character. A clear distinction between a project and a programme is that a project is an activity executed to achieve certain goal(s) but a programme is made up of several activities with a common aim as a direction. To verify this statement, the following discussion informs.
A programme can be determined from the group of activities it embodied (Smith, 1989; McDavid and Hawthorn, 2006; Royse et al., 2010) but encompasses several common features: “objectives, the use of resources, identifiable clients, content, processes, and outcomes” (Lee and Sampson, 1990:p157-158). Similarly, Moore and Spires (2000:p227) refer to a programme as “a group of interventions often linked together over time and encompassing more than one project” and describe a project as “a single intervention or discrete, one off, form of activity”. This means that a programme may consist of more than one project or activities and is designed to attain a particular target whilst meeting the need of certain groups of people. Additionally, a programme has its distinctive feature of an evidently defined goal in targeting a specific population for the effectiveness of the programme (Chen, 2005; GAO, 2011). To Cronbach et al. (1980:p14), a programme is “a standing arrangement that provides for a social service”.
Royse et al. (2010:p5) added, “Programmes are interventions or services that are expected to have some kind of an impact on the programme participants”. Moreover, they also agreed that a programme can be determined from its exact characteristics which encompass staffing, budget, identity and service philosophy (or slogan, or vision). Stockmann (2011b:p20) too gave his idea and said, “programmes are derived from a political strategy, planned and implemented in individual implementation steps, and as a
24
rule endowed with financial support for a limited period of time, in order to achieve desired effects”. According to him, (Stockmann, 2011b:p20) “The course of a programme, moreover, is embedded in complex social multi-level processes. Programmes are not developed independently of other existing or planned programmes. Programmes are often used by different actors to pursue different aims”.
From a different description, as classified by Owen and Rogers (1999), a programme is categorised under three levels, namely, mega level, macro planning and micro level. A mega level programme tends to focus generally on economic and social effects. Whilst macro planning may possibly be the duty of departments or branches and regions, micro level is seen as the responsibility of smaller groups within an organisation. In addition to that, they too added that, despite understanding the level of programmes, programmes can also be verified based on its type, for example, “educational, advisory, regulatory, case management, or service/product provision” (Owen and Rogers, 1999:p28). And for that reason they suggested that a programme should comprise two major elements which are a standard plan and strategic actions within the strategy (Owen and Rogers, 1999). Unlike a project, “the life course of a programme can be divided roughly into three main phases: the (1) planning and (2) implementation phases during the course of the actual programme and (3) the period following termination of the funding of the programme (sustainability phase)” (Stockmann, 2011b:p21). Conversely, “projects consist of a set of individual measures and programmes, in their turn, of a series of interrelated projects” (Stockmann, 2011a:p9). Furthermore, Maylor (2005:p4) describes a project as “any activity that has a start and a finish” and elaborated the activities defining a project as a “non-repetitive activity, which carries particular characteristics:
It is goal oriented – it is being pursued with a particular end or goal in mind;
It has a particular set of constraints – usually centred around time and resource;
The output of the project is measurable;
Something has been changed through the project being carried out” (Maylor, 1996:p3).
In relation to those sorts of definition, the researcher sees a programme as an intervention with a few or many projects but a project has only one scheme. It is therefore clear that a programme is larger in its components, with variety of projects whereas a project has a focus individually and specified plan.
2.4.2 Programme Evaluation and Its Relationship with Performance Measurement
There is a clear distinction between programme evaluation and performance measurement. Evaluation is conducted to assess a programme and to help understand the extent of the programme’s achievement (Thayer and Fine, 2001; GAO, 2011). Meanwhile performance measurement is an ongoing exercise
25
throughout the programme implementation for the purpose of programme evaluation. This simply means that performance measurement is part of programme evaluation. McDavid and Hawthorn (2006) strongly believe that performance measurement is a requirement in programme evaluation and not a replacement for evaluation. They believed that, for the last two decades, performance measurement had received greater attention whether through interest or resources. This was proven from a statement made by Jowett and Rothwell (1988:p21) stating, “Local government performance reviews began during the first half of the nineteenth century, when parliament established district auditors to investigate the extent to which spending related to authorised objectives”.
Performance measurement is not only the concern of the public sector but also among non-profit organisations in measuring the effectiveness of a programme (Carmona and Sieh, 2004). Moreover, it provides information for programme managers and organisations in examining the extent of programme achievement (Wholey, 1996). Performance measurement has a distinct feature of ongoing monitoring and reporting of a programme’s attainment. It is regularly undertaken to observe the programme’s operation and process, its outputs and outcomes. On the other hand, programme evaluation is an individual systematic assignment which is carried out “periodically or on an ad hoc basis” (GAO, 2011:p2), in order to analyse the accomplishment of the programme in relation to its original objectives.
The link between programme evaluation and performance measurement is fairly seen as programme evaluation comprised of a wide range of functions; it can be achieved by assessing the operation of a programme, examining the programme’s achievement, investigating the connection between the process and outcome, be conducted for the purpose of comparing its effectiveness with other similar interventions, as well as enabling an estimation of the impact (GAO, 2011). These activities, in turn, allow an in-depth investigation of the programme’s performance, which consequently provides the management with an overall judgment of the programme, thus making further recommendations possible (GAO, 2011). On the contrary, unlike programme evaluation which is often carried out sporadically, performance measurement is generally regarded as a continuous procedure (Royse et al., 2010; GAO, 2011). The advantage of this ongoing exercise provides an initial warning to the administration and management (Royse et al., 2010; GAO, 2011). However, although performance measurement and programme evaluation are executed at a different mode, nevertheless, both are completed with the same aim; to assist decision-makers with solutions for improvements (Posavac and Carey, 1989; Royse et al., 2010; GAO, 2011). In addition, Jowett and Rothwell (1988:p21) pointed out, “although performance assessment has been traditionally regarded as a central task, in reality evaluation of the ‘economy, efficiency and effectiveness’ of local services has taken place within the departments, subject to review and criticism by both the Audit Commission (and, more recently, the National Consumer Council)”. Not surprisingly, when evaluation itself has an important role in aiding management with an overall judgment
26
of the programme, similarly, programme performance too has its benefits in an organisation. Jowett and Rothwell (1988:p31) have listed ways in which performance measurement can assist the management by:
“Providing a basis for policy planning and control;
Enabling activities to be monitored at various levels to confirm that council policies are being implemented and that their intended results are being achieved;
Providing information for the review of policies, management practices and methods;
Providing evidence as to performance and as to the way in which management (collectively or individually) has responded to information revealed during the monitoring processes”.
Without doubt, performance measurement as an ongoing procedure in an organisation can continuously provide important information about a programme through its evaluation.