Evaluation practice ( EP) is a widely recognised domain of evaluation that has recently seen a substantial increase in scholarly work (Alkin, 2003; Schwandt, 2005; Saunders et al., 2011; Chelimsky, 2013; Chouinard, 2013; Leviton, 2015; Schwandt, 2015). To advance the discipline of EP, it is relevant to review its conceptual definition first, so I present the current understanding of both terms separately– evaluation and practice – to then examine them as a whole. Moreover, providing the individual definition of both terms is an appropriate task to explore the literature, as it allows the establishment of clearer research boundaries.
2.3.1 Evaluation
Evaluation has been defined in many ways and still is. According to the Oxford Dictionary, the origin of the word ‘evaluation’ links with the determination of the value of a project, program or policy (Simpson et al., 2009). Evaluation has a variety of meanings that imply differing theoretical lenses, as well as purposes. For example, the OECD4 defines evaluation as: “the process of determining the worth or significance of an activity, policy or program. It
is as systematic and objective as possible, of a planned, on-going or completed intervention”
(OECD, 2002; emphasis added).
4 Organisation for Economic Co-operation and Development,
See http://www.oecd.org/document/32/0,3746,en_2649_33721_42632800_1_1_1_1,00.html#Evaluation; Accessed on 25th January 2013
33
Some authors argue that accountability purposes are the core issue in any evaluation (Huffman et al., 2008; Carman, 2009), while others advocate learning as the main purpose (Riddell, 2001; Hoole and Patterson, 2008; McCluskey, 2011). For the former, a program evaluation theory corroborates accountability as key, whereas in the latter, a value-pluralism or social practice theory promotes the learning purpose of the evaluation. Some others argue that evaluations are needed either for improvement purposes (Vo and Christie, 2015) or to attribute results to a particular intervention, rather than to other potential causes (Kusek and Rist, 2004; Mayne, 2004; Morra-Imas and Rist, 2009). According to Scriven (1991), one of the founders of modern evaluation, there were nearly sixty different terms for evaluation, even 25 years ago. These include: “adjudge, appraise, analyse, assess, critique, examine,
grade, inspect, judge, rate, rank, review, score, study, test…” (ibid. p.9). I would add that for
each term, or group of terms, a specific theoretical approach applies, and for this reason it is relevant to map the previous research attempts to define evaluation according to a variety of theoretical approaches (see Table 2.1: chronological list of the main studies linking evaluation theory and practice).
A social practice (SP) dimension of evaluation has been widely accepted in the literature, in relation to evaluation of social interventions, because the purpose of evaluation in that dimension is to attribute the worth of how a specific intervention contributes to its overall goal; as such, the realisation that the process of attributing worth and value is social and relational has advanced the conceptualisation of EP. Abma and Widdershoven (2008) argue that “evaluation is not only a scientific and technical affair, but a social practice in itself” (p.121); whereas for Saunders et al., (2011),
evaluation is characteristic of all social policy areas; involves dimensions of evaluative practice consisting of symbolic structures, particular orders of meaning in particular places and has unintended effects. It consists of practices
34
which use implicit, tacit or unconscious knowledge as well as explicit knowledge; can have progressive enabling characteristics but are also perceived as part of the “surveillance culture”. (Saunders et al., 2011, p.4, emphasis added)
The standpoint taken in this thesis is that evaluation is a SP, because it is “undertaken by
people, within structures of power and resource allocation” (ibid. p.3). Thus practitioners’
experiences of evaluation are seen as a routinised daily practice – an evaluation practice (Reckwitz, 2002; Saunders et al., 2011; Shove et al., 2012).
2.3.2 Practice
A primary concern of most theories of evaluation is its practice (Christie, 2003; Saunders et
al., 2005; Rogers, 2008; Carman and Fredericks, 2010; Patton, 2010), but different theoretical
traditions locate the social in different units of analysis. A SP perspective locates the social in the ‘practice’, which means that what one does, thinks, knows and wants on a routinised basis, is evaluative in nature - an evaluation practice. In the present research, the term, ‘practice’ is defined in terms of Reckwitz’s (2002) SPT stance, as follows:
A ‘practice’ is a routinized type of behavior which consists of several elements, interconnected to one other: forms of bodily activities, forms of mental activities, ‘things’ and their use, a background knowledge in the form of understanding, know-how, states of emotion and motivational knowledge. A practice – a way of cooking, of consuming, of working, of investigating, of taking care of oneself or of others, etc. – forms so to speak a ‘block’ whose existence necessarily depends on the existence and specific interconnectedness of these elements, and which cannot be reduced to any one of these single elements. (Reckwitz, 2002, p.250, emphasis added)
Within this thesis it carries a social perspective stamp, as a “set or cluster of behaviours
forming ways of thinking and doing associated with undertaking evaluative activity”
(Saunders et al., 2011, p.2). The SP branch of the theories of practice (discussed in chapter three) focuses on the way a practice itself, in whatever domains, become an object of
35
examination; so what people do on a daily basis is what is termed by practice; and all societal life can be interpreted as sets or clusters of practices within different contexts.
2.3.3 Evaluation Practice
Scriven’s (1991) idea that evaluation is a “new discipline, but an ancient practice” (p.5) challenged the separate use of both terms, ‘evaluation’ and ‘practice’. It is possible that the combination of the new and ancient elements represent his understanding of EP, but there are no certainties. What is known is that a multitude of views prompted scholars to clarify exactly what they mean by ‘evaluation’ and by ‘practice’ (Saunders et al., 2011). Following on from Scriven’s (1991) theorisation, other scholars have provided their views on how fundamental EP is, as a property of the conceptualisation of evaluation as a discipline (Patton, 2002; Rossi et al., 2005; Morra-Imas and Rist, 2009), for it constitutes the essential aim of the evaluation theories. Also, Shadish (1998) has put forward the contingency theories of EP to emphasise that:
All approaches to evaluation involve trade-offs among the many goals we try to maximize in evaluation (e.g., the goals of use, constructing valid knowledge, valuing, assisting in social change, etc.), so we need to have conceptual tools to help us understand those trade-offs. The contingencies in which we are interested in evaluation theory are those with implications that make a difference to evaluation practice. (Shadish, 1998, p.8, emphasis added)
Another core contingent to EP relates to the attribution of value to what happens in a specific time and in a given context (Mark and Shotland, 1985; Morra-Imas and Rist, 2009; Scriven, 2012; Freeman and Hall, 2012), alongside the attribution of worth or merit, as important for the society to define what is worthwhile – “how its agencies attribute value to its policy and
program interventions, how institutions decide on the quality and merit of its internal
36
(Saunders et al., 2011, p.1). In this thesis, the term ‘evaluation practice’ means a “social
practice bounded by the purpose, intention or function of attributing value or worth to
individual, group, institutional or sectoral activity” (ibid. p.3), so the notion of ‘worth’ in
evaluation is relevant for this research, because it challenges how evaluation is perceived, mainly by practitioners.