CHAPTER III E VALUATION IN O RGANISATIONS
3.1 The nature of evaluation
The activity of conducting evaluation in organisations is often termed research and the labels are often used interchangeably (Thornhill et al., 2000; Russ-Eft & Preskill, 2009).
As the definitions and the previous section signify evaluation addresses critical questions about how well an organisation, programme, or process is working. Consequently, eval-uation is of particular interest for decision-making purposes with the intention by various stakeholders to use the findings (Russ-Eft & Preskill, 2009). On the other hand, research is mainly an activity seeking truth with the intention to contribute to knowledge. Research is mainly conducted for the purpose of describing or explaining the world and is conduct-ed and governconduct-ed by those with a high level of proficiency or expertise (Coryn, 2007).
There are some similarities between these two forms of enquiry.
However, it is important to notice the similarities, differences, and relationships between evaluation and research, which are presented in the table below.
Table III-2: Relationship between evaluation and research
Evaluation Research
Purpose Provides information for decision making and learning (intention is use) Seeks to describe particular phenom-ena
Is undertaken at the behest of a client:
service oriented
Develops new knowledge Seeks conclusions
Seeks new laws, new theories Topic is determined by researcher
Audience Clients (internal, external) Other researchers Focusing the Reviews the literature on the topic Develops theory-based hypotheses and/or research questions
Identifies terms and definitions Identifies variables to be studied Designing
the study
Naturalistic/qualitative Experimental/quantitative
Often bounded by organisation’s time frame requirements
Naturalistic/qualitative Experimental/quantitative
Is based on researcher’s time line and available funding
Collecting data
Tests, surveys, observation, interviews, records, documents, and unobtrusive measures Ensuring Is rooted in values and politics
Generalisability of findings not a major goal or concern
Pilot testing, member checks, control-ling variables through design, triangu-lation, test/retest reliability measures Attempts to be objective and value free Reports results to evaluation clients Makes recommendations relevant to evaluation questions
Rarely publishes the results
Makes empirical conclusions Reports results to other researchers and practitioners
Makes suggestions for future re-search
Often publishers study’s findings Source: Russ-Eft & Preskill (2009, pp.6-7)
Russ-Eft & Preskill (2009) summarise, although these two different forms of inquiry use the same data collection and data analysis methods they differ significantly in at least the following three dimensions:
often initiated for different purposes
involve and respond to different kinds of audiences or client questions and needs
communicate and report their findings in different ways to different groups
This study comprises aspects of both evaluation and research approaches, which are made clear and more explicit in the course of this work (see also 3.6.1 Case study evaluation, Chapter IV Research Philosophy, Methodology and Research Design). As a first step, the different forms and purposes are being dealt with more explicitly in the following.
Literature differentiates between two different forms, basic and applied research (Thorn-hill et al., 2000; Millmore et al., 2007; Blaikie, 2009). Research can embrace different purposes, such as exploring, describing, explaining, understanding, predicting, changing, evaluating and assessing impacts (Blaikie, 2009). Basic research focuses on the first five elements: exploring, describing, explaining, understanding, predicting. As theory-oriented research, it is aiming at advancing fundamental knowledge about the development and testing of theories in particular. Hence, basic research is concerned with producing knowledge for understanding (Blaikie, 2009). Although applied research may include some of the basic intentions, its particular focus is on change, evaluation, and impact as-sessment. Therefore it is concerned with practical outcomes, trying to solve practical problems, helping practitioners accomplish tasks, and the development and implementa-tion of policies or programmes (Blaikie, 2009). Overall, applied research is about produc-ing knowledge for action (Blaikie, 2009).
The figure below illustrates these different evaluation approaches.
Figure III-1: Purpose and focus of different evaluation approaches
Source: Millmore et al. (2007, p.131)
As this work follows an applied research approach both formative and summative eval-uation are dealt with in more detail below.
3.1.1 Formative evaluation
Formative evaluation is about providing information for developing, ensuring quality, or improving a policy, programme, product, or service (Scriven, 1991; Stufflebeam &
Shinkfield, 2007). Conducting formative evaluation is a prospective and proactive activity which provides feedback where its findings support an improvement-focused process which further develops, refines, or revises the object being evaluated (Russ-Eft & Pre-skill, 2009). This kind of evaluation is being undertaken during the development process of a programme or its ongoing operation. It provides guidance for those being responsi-ble for ensuring and improving the programme’s implementation and quality (Stufflebeam
& Shinkfield, 2007). During the development process, formative evaluation assesses and assists with the formulation of objectives and priorities and provides directions for plan-ning. In the course of the programme, it supports the programme management by as-sessing implementation plans and interim results.
Formative evaluation focuses on improving and fine tuning and therefore uses monitoring and continuous feedback during an implementation (Blaikie, 2009). Overall, formative evaluation is directed to improving development processes, implementations and opera-tions, quality assurance, guidance for decision-making (Stufflebeam & Shinkfield, 2007).
The table below presents some typical formative evaluation questions.
Table III-3: Formative evaluation questions
Formative and developmental evaluation implementation questions How well is the program being implemented?
What are the challenges and barriers to implementation?
How have staff responded to these challenges and barriers?
How effective are the programme’s strategies and activities?
What are the programme’s key characteristics as perceived by various stakeholders: partici-pants, staff, administrators, funders?
How similar or different are those perceptions?
What are the bases for and implications of different perceptions?
What is participant and staff feedback about programme processes: What is working well and not working so well from their perspectives?
What original assumptions have proved true?
What assumptions appear problematic?
How accurate has the original needs assessment proved to be?
To what extent, if at all, are actual needs different from what was planned for?
What are the primary activities (in detail) in the programme?
What do participants like and dislike?
Do they know what they are supposed to accomplish as participants?
Do they “buy into” the programme’s goals and intended outcomes?
How well are staff functioning together?
Do they know and agree about what outcomes they are aiming for?
What are their perceptions of their own roles and effectiveness?
What has changed from the original design and why?
On what basis are adaptations from the original design being made?
Who needs to “approve” such changes?
How are these changes being documented and reflected on?
What monitoring system has been established to assess implementation on an ongoing basis and how is it being used?
What are the key factors and variables in the programme’s environment that need to be tracked so that programme can adapt to emergent conditions?
How are these variables interpreted and fed back to the programme to support ongoing adapta-tion?
For each new development what progress markers provide feedback on how that development is working out?
When have incremental changes accumulated to the point of constituting a new intervention?
What values, vision, and principles underpin the emergent developments being tracked?
Is the programme manifesting those values, visions, and principles as it unfolds and develops?
Have those values, visions, and principles changed?
If so, how and why, and with what implications?
Source: Adapted and based on Patton (2008, p.321), Russ-Eft & Preskill (2009, p.19)
3.1.2 Summative evaluation
Summative evaluation is conducted to determine the merit, worth, or value of the evalu-and leading to a final evaluated judgement (Russ-Eft & Preskill, 2009). It is undertaken after a respective policy, programme, or any other change has been implemented to es-tablish its overall effectiveness in achieving the intended objectives (Blaikie, 2009).
Summative evaluation is aiming at examining the consequences of the adoption of par-ticular courses of action and/or change (Blaikie, 2009). Accordingly, this kind of evalua-tion is a retrospective assessment drawing together and supplementing previously collected information (Stufflebeam & Shinkfield, 2007). Summative evaluation is useful in determining accountability for success and failures and helps interested parties increase their understanding of the evaluand (Stufflebeam & Shinkfield, 2007). Typical target groups or stakeholders are development staff, consumers, and decision-makers and sponsors in particular. Very often summative evaluation information derives from in-depth case studies (Stufflebeam & Shinkfield, 2007). The table below presents some typical questions being asked for summative evaluation purposes.
Table III-4: Summative evaluation questions
Summative and lessons learned implementation questions
To what extent can the programme be modelled as a coherent, high fidelity intervention treat-ment with clear connections between inputs, activities, and outcomes?
To what extent has implementation been routinised and implementing steps identified and doc-umented?
To what extent and in what ways was the original implementation design feasible?
What was not feasible? Why?
In what ways do participants benefit from the programme?
To what extent has the process improved employee productivity?
How stable and standardised has the implementation become both over time and, if applicable, across different sites?
To what extent is the programme amenable to implementation elsewhere?
What aspects of implementation were likely situational?
What aspects are likely generalisable?
Has implementation proved sufficiently effective and consistent that the programme merits con-tinuation or expansion?
Were the results worth the implementation’s costs?
What has been learned about implementation of this specific programme that might inform simi-lar efforts elsewhere?
What has been learned about implementation in general that would contribute to scholarly and practitioner-oriented research on implementation?
Source: Adapted and based on Patton (2008, pp.322), Russ-Eft & Preskill (2009, p.19-20)
3.1.3 Relationship between formative and summative evaluation
The nature and circumstances of the evaluand indicate the relative accesses of formative and/or summative evaluation (Stufflebeam & Shinkfield, 2007). The former is more likely to be conducted in early phases of a programme, whereas summative evaluation will be conducted as the programme concludes and after its implementation in particular.
Stufflebeam & Shinkfield (2007) further note that those responsible for a programme should be well aware and have a clear understanding of when and in what circumstanc-es rcircumstanc-espective evaluations shall take place. Both typcircumstanc-es of evaluation are needed in the development of a programme. Experts in this field recognise that too often summative evaluation is conducted only for judging on programmes, which restricts the development processes and may point to inappropriate, misleading, or even incorrect conclusions and consequently wrong decisions. Similarly, a lack of (or delayed) formative evaluation and respective findings (e.g. costs, efficiency, mistakes) might also cause inadequate deci-sions (Stufflebeam & Shinkfield, 2007). There is no clear distinction over who is conduct-ing which type of evaluation. However, it appears to be that formative evaluations are conducted by internal people whereas summative evaluation are often conducted by ex-ternal evaluators (outside regarding programme implementation, not necessarily outside the organisation/company) (Scriven, 1991; Stufflebeam & Shinkfield, 2007). However, there are various influencing factors such as timelines, budget, and competency of per-sonnel to undertake evaluations. Ultimately, Stufflebeam & Shinkfield (2007) argue that the decisive factor for the selection of internal or external evaluators should be whether the process and findings will be and are credible.
Frequently, summative evaluations are based on formative evaluations and the merit of worth of the latter can be strengthened by the former (Stufflebeam & Shinkfield, 2007).
The distinction between formative and summative evaluation is not discrete or absolute, as summative evaluation could have a formative effect on future developments even if it is presented after a particular implementation (Robson, 2011). The two approaches and their characteristics are contrasted in the table below.
Table III-5: Formative and summative evaluation
Formative evaluation Summative evaluation Purpose Quality assurance;
im-provement
Provide an overall judgement of the eval-uand
Use Guidance for
decision-making
Determining accountability for success and failures; promoting understanding of assessed phenomena
Functions Provides feedback for im-provement
Informs consumers about an evaluand’s value, for example, it’s quality, cost, utili-ty, and safety
Orientation Prospective and proactive Retrospective and retroactive When conducted During development or
on-going operations
Assists consumers in making wise deci-sions
Foci Goals, alternative courses of action, plans, implementa-tion of plans, interim results
Completed projects, established pro-grammes, or finished products; ultimate outcomes
Variables All aspects of an evolving, and developing programme
Comprehensive range of dimensions concerned with merit, worth, probity, safety, equity, and significance Audience Managers, staff; connected
closely to insiders
Sponsors, consumers, and other inter-ested stakeholders; projected especially to outsiders
Evaluation plans Flexible, emergent, respon-sive, interactive
Relatively fixed, not emergent or evolving Typical methods Case studies, observation,
interviews, not controlled experiments
Wide range of methods including case studies, controlled experiments, and checklists
Reports Periodic, often relatively in-formal, responsive to client and staff requests
Cumulative record and assessment of what was done and accomplished; con-trast of evaluand with critical competitors;
cost-effectiveness analysis Relationship
between formative and summative evaluation
Often forms the basis for summative evaluations
Compiles and supplements previously collected formative evaluation information
Source: Stufflebeam & Shinkfield (2007, p.25)
Overall, evaluation research, be it formative or summative, is concerned with programme development and implementation in particular, and with problem solving and decision-making. It seeks to answer questions mainly posed by practitioners and decision-makers rather than academics (Blaikie, 2009).
This research takes a summative evaluation approach in order to develop a framework for monitoring and evaluating strategic change programme implementation, which can be used for formative purposes later on.