Realistic Evaluation

Tilley (2000) describes Realistic Evaluation as taking an alterative view to the interpretive and positivist approaches, and emphasises the purpose of evaluation

research in developing policy and practice. This leads to a shift away from a positivist focus on deriving universal laws and, instead, movement towards the contextualised question of:

“What works for whom in what circumstances?” (Tilley, 2000, p.4).

Realistic Evaluation is concerned with understanding causal mechanisms and the conditions under which they are activated to produce specific outcomes. It seeks to develop programme theories in order to understand social programmes, and how and why they work. Programme theories explain how context and mechanisms work together to produce outcomes. Within this research the word ‘programme’ used by Tilley (2000) has been replaced with the word ‘intervention’, although the term ‘programme theory’ is maintained.

Realistic Evaluation has roots in realism and seeks to develop theories to explain the real world, and explores the way mechanisms produce events and the circumstances needed for this. Pawson & Tilley (1997) call for evaluation to be realistic in terms that it has a useful, clear goal, and define realistic as:

“…trying to perfect a particular method of evaluation which will work for a specific class of project in well-circumscribed circumstances.” (p.xiv)

Pawson & Tilley (1997) emphasise that, according to Realistic Evaluation: ƒ interventions are theories leading to the concept of a programme theory;

ƒ interventions are embedded in social systems, so realistic evaluators must consider the different layers of social reality that make up and surround interventions;

ƒ interventions are active, therefore participants’ views and interpretations are integral to considering an intervention/programme’s outcomes; and

ƒ interventions are part of open systems and can not be isolated or kept constant – therefore universal laws can not be derived.

These key points in understanding the nature of interventions and how they work have important implications for research and evaluation. Pawson & Tilley (1997) recognise the importance of participants in interventions and social programmes, and that they are a critical factor in any intervention in any social context. Therefore, a social programme cannot be explored in isolation but must also examine the role of the people, the outcomes attributed to the programme and the context within which the programme occurs. Pawson & Tilley (1997) use the following basic realist formula, and describe this as the basis for designing realist research:

“mechanism + context = outcome” (p.xv)

Therefore it is important to consider what the contexts (Cs), the mechanisms (Ms) and the outcomes (Os) are, in order to derive a programme theory.

The mechanisms are those parts of an intervention that bring about effects and change, the structures of the social programme. Mechanisms are ways in which the resources available are used to generate outcomes. Pawson & Tilley (1997) state realistic evaluators must understand the action of mechanisms in order to understand why a social programme works. In one of the methodological rules of Realistic Evaluation they state:

“Evaluators need to focus on how the causal mechanisms which generate social and behavioural problems are removed or countered through the alternative causal mechanisms introduced in a social programme.” (p.216)

Pawson & Tilley (1997) emphasise that mechanisms are often hidden to the researcher, and whilst the researcher can anticipate mechanisms that may be raised by the participants, there should be a willingness to acknowledge the existence of other, previously unconsidered, mechanisms. Timmins & Miller (2007) highlight that different types of participants may produce different outcomes and, therefore, features of the participants can be considered mechanisms. Thus the intervention, the group supervision approach, may have a different effect on different types of participants. Thus mechanisms incorporate the relevant characteristics of the participants and characteristics of the intervention that lead to the outcomes of the intervention.

The context describes the features of the conditions in which the intervention is introduced that are relevant to the operation of the programme mechanisms. Pawson & Tilley (1997) illustrate that contexts go beyond the spatial location and include the conditions that allow the mechanism to come into operation. Within social contexts, this includes the social rules, norms and values, interpersonal and social relationships. They emphasise that contextual knowledge is crucial in order to ensure that the intervention is well-targeted.

The outcomes-patterns comprise both the intended and the unintended consequences of an intervention, and realism does not depend on a single outcome measure. This allows for a sensitive evaluation of complex interventions and, as such, outcomes need to be defined by the participants of the intervention.

The context mechanism outcome pattern configuration can also be considered the programme theory. This, in effect, explores the configuration of features needed to sustain an intervention.

The preferred methodology chosen to explore and develop a theory for a model of EP work in Children’s Centres was Realistic Evaluation. This would develop a programme theory to explain a model of EP work that was used within a particular context, through an examination of the context, mechanisms and outcomes for participants and other stakeholders in the intervention. Therefore, in the context of the present study, it could be said that the following possible mechanisms, contexts and outcomes have been identified by the researcher, although others may not yet be identified and will be uncovered through the research process. In addition, mechanisms, outcomes and contextual factors can individually work as positive or negative factors. Examples of mechanisms, contextual factors and outcomes that could be uncovered by the present study are:

ƒ Mechanisms: Group supervision approach

Family Support Workers

Facilitator/EP

Children’s Centres managers ƒ Context: Children’s Centres

Children’s Centres manager’s relationships with Family Support Workers

Autonomy of Family Support Workers

ƒ Outcomes: Increased/decreased abilities to cope with workload Abilities to deal with children and families

Increase or decrease in the willingness to learn from one another

Improved relationship Family Support Workers and EP ƒ Possible context mechanism outcome pattern configuration or programme

theory: Family Support Workers will feel highly supported, able to cope with their case load and more able to share ideas with each other (O) within Children’s Centres where the manager supports the autonomy and group learning of the Family Support Workers (C) who have group supervision on a regular basis with a facilitator/EP who can support group supervision

approaches (M).

Pawson & Tilley (1997) explain that Realistic Evaluation is not within the positivist paradigm and does not seek universal statements – such as this social programme always led to these outcomes – as this overlooks the issue of context. Alternatively, they state that the goal of evaluation is the “continual betterment of practice” (p.119) through seeking out the “descriptive particulars” (p.119) of an individual programme in order to build and cumulate sets of ideas into an organising framework. Pawson & Tilley (1997) describe this as:

“…an organising framework which ‘abstracts’ from a programme a set of essential conditions which make sense of one case after another.” (p.120)

Therefore, whilst Pawson & Tilley (1997) do not use the term ‘theory’, they state that evaluators must build their evaluation on research from previous evaluations and these previous evaluations must also consider the issue of context and mechanisms, not outcomes alone. Within the present study the information gained from the research

methods can be used to construct a set of descriptive particulars that builds upon previous research in order to lead to continued improvement in and changes to practice by EPs and others using group supervision with a range of workers within the Children’s Workforce.