This section reviews Saunders’ work of RUFDATA and further contributions to EU program evaluations (Bonamy et al., 2004; Saunders, 2011, 2012). The reason for reviewing these studies is because of their common view informed by a SPT view of evaluation, inspired by Giddens (1976) and, later, by Reckwitz (2002). These studies were also applied in different contexts, so a critical review is important to define key messages and limitations in the current research of EP in a DE domain.
110
RUFDATA, which is a hallmark of Saunders’ (2000) work, has been employed in other domains of social sciences11 and stands for Reasons and Purposes; Uses; Focus; Data and evidence; Audience; Timing; and Agency. The context in which RUFDATA emerged was one where novice evaluators were being initiated into the process of evaluation, specifically, into the practice of planning the evaluation (Saunders, 2000). It is described as an approach to planning evaluation as a sequence of “knowledge-based practices” (ibid. p.13); referring to knowledge as one element of practice, visible through “ways of doing things” (ibid.). Additionally, it draws on the tradition of learning, from Lave and Wenger (1991), which resulted in Saunders’ (2000) conceptualisation of evaluation as taking place within a community of practice. This context of depicting evaluation allows for a period of deep thinking, using enabling tools, such as reflexive questioning, particularly important to novice evaluators, with little or no previous experience, but likely to be charged with an evaluative task. This form of practising evaluation advocates a flexible approach, “low in fidelity
to give a sense of manoeuvre to practitioners” (ibid. p.15), as well as the
sufficient guidance for them to get evaluation “off the ground” (ibid.). RUFDATA is a situated approach that takes on board the influential role of context (Pawson and Tilley, 1997), the “particularity” of each intervention (Simons, 2015, p.173) and the uniqueness in practitioners’ “evaluative
moments” (McCluskey, 2011, p.103), which would serve the funders on a
provisionally stable basis (Saunders, 2005).
111
This approach is enacted through the formulation of some reflexive questions that form the basis of the EP. From a foundational perspective, the contribution of RUFDATA to research EP is unquestionable, however, Saunders (2000) seems to privilege the stage of planning and conducting an evaluation, rather than, looking at practice across the evaluation process as a whole. Although the planning stage is critical, the application of RUFDATA or some of its principles, across the whole process of evaluation could possibly enhance its EP, as well as benefit its wider application in other research domains. In terms of my research, this leads to questions about how the funders’ request for the evaluation, the leadership of the organisation, and the existence of other practices and working activities, influence EP across the trajectory of the whole project.
Another important work is a case study of a European funded initiative, EQUEL (e-Quality in e-Learning), that analyses how a practice-based evaluation has been used in a complex multi-partnership context of an e-learning project (Bonamy et al., 2004). The case study indicates the diversity of positions each group or individual occupies within the project and the subsequent complexity arising from the intersection of their positions, discourses and perceptions. EQUEL’s framework combined Saunders’ view (2000) on practitioners’ understanding of the program circumstances, with Fullan’s (2011) acceptance of diversity and complexity as an a priori proposition in every process of change. While Bonamy et al.’s (2004) contribution suggests that it is necessary to support practitioners in sharing their questions and observations; it recognises that this support may, or may not be during the lifetime of the project. That seems to be a very critical aspect for my current research in EP within NPOs, as
112
most of their evaluations tend to operate within unpredictable circumstances of an external funding context.
Despite its use of a SPT view of evaluation framework, Bonamy et al.’s (2004) analysis would have benefited from further analysis about the factors influencing change, the effects of the co-occurrence of other practices, issues of sequence, time and space in the trajectory of practice, alongside their implications for the wider context. This could also be because it is prior to Shove et al.’s (2007; 2009; 2012) conceptualisation of DSP.
Moreover, Bonamy et al. s’(2004) study emphasises the aspect of learning in evaluative processes and how the notions of “provisional stability” and
“evaluation as bridging tool” (Saunders et al., 2005, p.39) may enhance EP.
They recognise the nonlinearity of EP, through the critique that a metaphor of ‘implementation stair case’ appears to suggest a certain stabilisation of EP within a project’s implementation; see also Reynolds and Saunders (1987).
It would have been interesting though to see identified some connections to allow the exploration of these notions in other domains of social sciences. Also, EQUEL’s study (Bonamy et al., 2004) re-frames Hall and Loucks (1978) seven stages of continuum from awareness to refocusing, regarding teachers’ stages of concern in project involvement (when dealing with new ideas), by simplifying it in three – exploratory, consolidation, and development - and this move has potential to be further applied and conceptualised in other domains, as it captures a broader sense of participants’ experiences of project implementation in evaluation.
113
In another research study about evaluation of interventions, policies and programs, Saunders (2011) outlines a practice-based approach to evaluation to capture the effects of EU interventions, particularly regarding the diversity and the complexity of the EU context. A fundamental feature of a practice-based approach to evaluation is the need for a shift from an attribution to a contribution perspective of outcomes achievement (Mayne, 1999). An attribution perspective contemplates the causality of outcomes achievement, due to a specific implementation or activity, whereas a contribution perspective accepts a provisional and yet stable input or practice as a relevant contribution towards the achievement of the desirable outcomes. In this evaluation of an EU program’s effectiveness, Saunders (2011) calls for a greater attention on the role of practices as a preferable vehicle to capture the effects of interventions; especially within a funder-recipient based context. The practice approach is attached to “whatever it is that people do as a response to a policy, an intervention,
opportunity or initiative, and this constitutes an inductive method” (ibid. p.93).
By using a SPT lens, informed by Reckwitz (2002), Saunders (2011) sheds light on the risk of an excessively logical view of understanding change, dominated by a predetermined way of thinking. Nonetheless, in a situation of understanding change, neither practitioners on the ground, nor the policy makers, “act in a
rational way” with established goals in mind (ibid.). People’s practice of
evaluation is sometimes unintentional and so a contribution approach to evaluation celebrates this possibility, whilst noting that some intentional influence occurs. Whilst Saunders’ study was situated in a complex area of EU project funding, it would have been interesting to explore how a SPT contribution to evaluation applies in other domains of social sciences.
114
A final study considers the use and usability of evaluation as practices in two dimensions; first, how evaluation is used by real people in real time, and second, how the design of evaluation influences its use in a wider sense. Saunders’ (2012) argument is that the impact of an evaluation can be traced back in relation to the changes in its practice over time. Furthermore, he argues that it is carried by practitioners over time, and that a SPT view adds value to evaluation in three dimensions: first by depicting use and usability as practice, practitioners strengthen their knowledge resources to make sense of evaluation. Secondly, by adopting a SPT view he suggests that use of evaluation is translated through the change of its practice, to the extent to which, and in which domains it is used or not used. Besides, Saunders (2012) challenges the view about “what might be
the nature of the knowledge resources available from an evaluation that creates
the momentum for change (i.e. for it to be used)”(ibid.p.426). Thirdly, a SPT
approach enables the identification of practices of engagement to depict patterns of change over time. One thread of Saunders’ (2012) argument lies in the fact that “it is difficult to get practices to change” (ibid.). My research draws on this argument to explore how evaluation is currently practised in a DE domain.
However, the assumption that practices are difficult to change is challenged from the DSP view that change is inevitably reproduced and enacted by people, regardless of their awareness (Shove et al., 2007; 2009; 2012). A further thought leads to a dual-dynamic, one of a conscious and intentional change being difficult, with another unconscious and unintentional change happening in peoples’ trajectories of life. Still, Saunders’ (2012) analysis may have benefited from considering the added value of DSP to analyse use and usability of
115
evaluation, for example, whether evaluation has been used in the same ways or how change has been manifested in the way evaluation is used.
Shove et al. (2009; 2012) emphasise the use of the elements of practice to depict the dynamics of change over time and in simplifying Reckwitz’s (2002) theorisation of seven elements to three – material, competence and meaning – (see Figure 3.1; the frameworks of the elements of practice), they were able to explore one element in-depth at a time, as well as how the others interwove. This process enables the capture of change in practice over time.
In sum, empirical and theoretical evidence strongly suggests the further application of a SPT approach in evaluation studies. Previous studies have demonstrated the usefulness of SPT in a number of dimensions. For example, in emphasising the core benefits of reflexive questioning in the planning of evaluation (Saunders, 2000); the recognition of a nonlinearity of EP (Bonamy et
al, 2004); the benefits of a contribution approach to better understand change in
evaluation (Saunders, 2011), and the relevance of individuals’ engagement to describe patterns of change in evaluation (Saunders, 2012).
However, these important studies have drawn upon a SP view (Reckwitz, 2002) that has been recently extended and updated by Shove et al. (2007; 2009, 2012). As illustrated in chapter three, Shove et al. s’ (2012) particularly emphasises a number of dimensions that are likely to impact in EP such as the co-occurrence of practices and how the effects of this co-occurrence influence and change overall practices. With notable exceptions (Saunders et al., 2011; Trowler et al.,
116
2012), these important issues have only been addressed superficially in evaluation theory and therefore remain an important gap to develop.
Moreover, while studies drawing upon a SP view have offered in-depth insights into how evaluation occurs in a number of contexts, repeated calls have been made for further studies that explore EP in other domains. For example, Saunders et al. (2011) encourage scholars to “develop a meta-framework that
yields the possibility of comparative research into evaluative practices into other
social policy domains” (p.226). In line with this call, the current study begins to
address these gaps by extending a SPT view of evaluation into the domain of Development Education (DE).
The next section summarises the gaps presented in the three chapters of the literature review, as well as devises this study’s research questions.