Realist evaluation

2.5.1 An overview of realist evaluation

Realist evaluation, previously named realistic evaluation, is not concerned with “does this or doesn’t this work?”, but instead with “what works for whom, how and under what circumstances” (Pawson et al., 2005, p.32). A realist evaluator does not assume that what works in one situation will work in another; instead, they explore why programmes worked/did not work in different contexts (see glossary) (Westhorp, 2014). As a theory-based method of evaluation, it adopts empirical methods, forming hypotheses (see glossary) on how programme activities are understood to cause outcomes (see glossary) – termed the ‘programme theory’ (Westhorp, 2014). The programme theory is tested, utilising either (or both) qualitative and quantitative methods (Westhorp, 2014; Pawson and Tilley, 2004).

43 2.5.2 Key principles

Pawson and Tilley (2004) identified four key facets of the realist’s perspective on how an intervention brings about change:

1. Programmes are theories 2. Programmes are embedded 3. Programmes are active

4. Programmes are open-systems 2.5.2.1 Programmes are theories

Programmes are inputted into social systems as a solution to the social system’s problem. It may be theorised that the FCP role will reduce patient waiting times for an appointment (the programme theory) – leading to the introduction of FCPs across multiple Practices for a solution to waiting times (the problem). There are multiple theories to how a programme works, and the programme’s effectiveness as a whole will depend on the combined effect of these theories (Pawson and Tilley, 2004).

2.5.2.2 Programmes are embedded

The theory underpinning realism is that programmes are active and embedded in a social reality that is integral to its success. Programme resources can promote change, but the impact this programme has is contingent on the social circumstances of that person (subjects’ characteristics, their economic conditions, amongst others) (Pawson and Tilley, 2004). For instance, FCPs may be acceptable to patients who need self-management advice/exercises to return to work, whilst those not in employment may expect a greater level of practitioner input. A realist evaluation must decipher the multiple layers of social reality that make up a programme (Pawson and Tilley, 2004).

2.5.2.3 Programmes are active

For a programme to have its intended outcomes, active engagement from individuals who will be affected by the programme is usually required (Pawson and Tilley, 2004). For the role’s success, patients must be aware of the FCP and actively engage with the role through self-referring. The implications of this are that participants’ interpretations of a programme are integral to evaluating its outcomes (Pawson and Tilley, 2004).

2.5.2.4 Programmes are open-systems

Pawson and Tilley (2004) state that a programme’s delivery is impacted by a range of factors, including political change, inter-programme and intra-programme interactions, practitioner learning, media coverage, amongst others. A realist evaluation underlines the

44

importance of the interaction between the intervention, and the environment that it is implemented into (Pawson and Tilley, 2004). The programme may actually change the conditions that were inherent to its original success; therefore, the programme must be reflexive, through translating the knowledge gained into minute adjustments to the programme (Pawson and Tilley, 2004). A common policymaker concern is that removing the GP gatekeeper role could result in an unsustainable influx of physiotherapy referrals. If realised, this would indicate acceptability of the FCP, however, the role’s success would ultimately lead to failure through over-demand.

2.5.3 Contexts, mechanisms and outcomes

A fundamental principle of realism is that observational evidence (effects) alone cannot establish causal uniformities between variables (Dalkin et al., 2015; Astbury and Leeuw, 2010). Exploring effects is what is known as the ‘black box problem’, and it is the evaluator’s role to unpack the black box of complex interventions (Wong, 2013). For example, an FCP may have started injecting corticosteroids and, subsequently, the number of patients accessing the FCP role increased. Simply observing this does not identify what about the FCP injecting led to patients accessing the role. It may have been that the FCPs offered appointments quicker than GPs, or it may have been that the patients felt reassured by the FCP’s style of communication or skill when injecting. ‘White box evaluation’, more commonly named theory-driven evaluation, attempts to unpack this black box, in order to identify the complex components that constitute an intervention (Astbury and Leeuw, 2010; Scriven, 1994). Realists argue that we need to make causal links between three realist evaluation concepts of ‘context’, ‘mechanism’ and ‘outcome’, known collectively as ‘context-mechanism-outcome configuration’, or ‘CMO’. Exploration of the interacting concepts aims to unearth hidden causes that lie beneath desired outcomes (Pawson and Tilley, 2004):

Context + mechanism → outcome 2.5.3.1 Context

A programme does not operate in a vacuum, but is placed within a context; this context

‘triggers’ the mechanism which leads to mechanisms ‘firing off’ to create an outcome (Wong et al., 2016). On a micro level, context may include: people’s beliefs, expectations, and resources; staffing in the Practice; the workings of the Practice as a team and so on. On a macro level, the context may include the geographical setting of the Practice, cultural norms, and organisational setting (for instance partnerships in PCNs) to name a few (Westhorp et al., 2011).

45 Programmes may work differently in different contexts and through different mechanisms, consequently a programme that works in one context may not achieve the same outcomes in another (Westhorp et al., 2011). The potential issue with context sensitivity is a lack of transferability. For example, in Practice X there was outcome Y, but in Practice Z – which had a greater elderly population – it is questionable as to whether the findings of Practice X would be transferable. This issue is overcome by the ontology of realist evaluation; if there is reason to believe that in different contexts the same mechanism is causing the same outcome, then the findings of one setting are relevant to the other (Wong, 2013).

Contexts are not definite, they are constantly evolving and therefore, a programme that may not have worked in the past may in the future be able to achieve its desired outcome (Pawson and Tilley, 2004). This rationalises why one study may not indicate a theory which a subsequent study highlights; new programme theory can always be developed as

contexts change. Equally, a change of context may prevent a mechanism from working, or fire off a competing mechanism that inhibits the original mechanism and stops the

programme from achieving the desired outcome (Pawson and Tilley, 2004). For example, a Practice may have joined a PCN which had increased funding (context), which resulted in extended access hours including evening appointments which patients in employment found more convenient (mechanism). An intended outcome may have included a reduced wait for an appointment at a convenient time. Removal of the PCN funding could result in these hours being reduced and increased waiting times. Contextual knowledge is of the utmost importance to policymakers; successful programmes will be targeted at contexts which are most conducive to desired outcomes (Pawson and Tilley, 2004). It is vital that a realist evaluation collects data that are able to identify contexts that are relevant to the programme’s outcomes (Wong et al., 2016).

2.5.3.2 Mechanisms

Mechanisms can be defined as the underlying processes, entities or social structures that, when operating in particular contexts, lead to outcomes (Astbury and Leeuw, 2010). In the example regarding a FCP who could inject (see p.44), a suggested mechanism was the FCP’s style of communication, which reassured patients and resulted in patients accessing the role (outcome). As mechanisms are underlying, they are often ‘hidden’ and unobservable, therefore realist inquiries cannot rely purely on ‘demi-regularities’ to explain outcomes (Astbury and Leeuw, 2010). Demi-regularities are the causal associations that are considered universal due to repeated observations (for example, gravity) (Dalkin et al., 2015; Astbury and Leeuw, 2010).

46

It has been highlighted that mechanisms have erroneously been conflated with the

programme activity (Astbury and Leeuw, 2010; Rogers, 2007; Weiss, 1997). As Weiss states:

“The mechanism of change is not the program service per se but the response that the activities generate” (Weiss, 1997, p.46). Pawson and Tilley (2004) conceptualised mechanisms to describe how programme resources seek to change the stakeholder’s reasoning. Several scholars have been more explicit than Pawson and Tilley (2004) in the breakdown of resource and reasoning. Westhorp (2011) states that programmes ‘work’ by enabling participants to make different choices, and in order to sustain these choices requires a change in the participants reasoning – such as, values, beliefs or their logic – and/or the resources available to them – for example, skills or information. The combination of resource and reasoning is known as the programme mechanism, which allows programmes to have desired outcomes (Westhorp et al., 2011). Dalkin et al. (2015) argued for the disaggregation of mechanisms into ‘resource’ and ‘response’ (see glossary), suggesting that this encourages researchers to equally consider both concepts, rather than focus their enquiry on one. Dalkin et al. (2015) re-ordered Pawson and Tilley’s (2004) CMO formula to create a revised framework:

Mechanism (resources) + Context → Mechanism (reasoning) = Outcome This framework proposes that resources are implemented into existing contexts, in a way that enhances change in reasoning. It is reasoning that results in changed behaviour (the response) of stakeholders, and leads to outcomes (Dalkin et al., 2015). Placing context within the mechanism enables the researcher to clearly identify the role that context plays in triggering mechanisms, strengthening their understanding of how interventions work (Dalkin et al., 2015). Owing to the arguments put forward on the conflation of concepts, this study will adopt the framework proposed by Dalkin and colleagues (see Figure 2.1).

47 This study adopts the term ‘response’ rather than ‘reasoning’, as it encompasses both the process of reasoning and the stakeholder’s changed behaviour. Mechanisms can be further categorised into ‘latent’ or ‘unintended’ (Jagosh, 2019; Westhorp, 2014). Latent

mechanisms (see glossary) are those that are not currently active, however, could be revealed if the context was altered, reflecting the deeper layers of ontological depth (Jagosh, 2019; Lacouture et al., 2015). Unintended outcomes (see glossary) are where mechanisms are triggered that lead to unexpected or unanticipated effects (Wong et al., 2016; Astbury and Leeuw, 2010; Pawson, 2006). Westhorp (2014) distinguished between the generative mechanisms of these outcomes as intended and unintended mechanisms (see glossary) – mechanisms that had positive or negative effects respectively.

2.5.3.3 Outcome

Outcomes are the intended or unintended, short, medium and long-term changes that result from a programme (Punton, Vogel and Lloyd, 2016). Outcomes are entirely contingent on their associated context and mechanism, any change in either will impact upon the outcome (Pawson and Tilley, 2004). Through exploring the complex interaction between contexts, mechanisms and outcomes, a realist evaluation looks beyond outcomes that simply state a pass/fail of an intervention, outcomes that are traditionally attributed to randomised controlled trials (Wong et al., 2013; Pawson and Tilley, 2004). The intermediate outcomes (the transitional outcomes that come prior to the end outcome) are also of interest as they may open an insight that would otherwise be missed (Pawson and Tilley, 2004). Identifying only expected mechanisms would equally limit the programme

understanding, as it would not be possible to say whether the anticipated outcomes were achieved (Westhorp, 2014).

Figure 2.1 - CMO framework adapted from Dalkin et al. (2015) 4. OUTCOME

2. CONTEXT

MECHANISM

3. Response 1. Resource

48

Data must be collected about the relevant (or hypothesised to be) contexts, and when carrying out the analysis, outcome data and context can then be linked in order to explore associations (Westhorp, 2014). If it is theorised that outcomes for patient acceptability will be different for a population with chronic diseases compared to a population with acute diseases, then the outcomes will need to be disaggregated by the duration of the condition.

It is suggested by some realists that quantitative data be collected for outcomes, as disaggregated analysis is easier to achieve with numerical data (Westhorp, 2014).

Nonetheless, disaggregation of qualitative data can be achieved through separating the data from different subgroups, analysing the data and making a comparison (Westhorp, 2014).

2.5.3.4 Programme theory – context mechanism outcome configuration (CMO) Realist evaluations attempt to pinpoint the configuration of features needed to sustain a programme. This results in the formation of context, mechanism, outcome (CMO) configurations, also named ‘programme theory’ (Pawson and Tilley, 2004).

CMO configurations (see glossary) collectively constitute the programme theory – the theory of how the programme is expected to work. The realist evaluator begins with an initial, basic programme theory: 'If we do X then Y will happen because...'. From here the programme theories are developed into hypotheses relating to the following:

1. For whom will this programme theory work and not work, and why?

2. In what contexts will this programme theory work and not work, and why?

3. What are the main mechanisms by which we expect this programme theory to work?

4. If this programme theory works, what outcomes will we see?

(Westhorp, 2014, p.10).

A list of disaggregated contexts, mechanisms and outcomes is produced. In the next stage, CMO configurations are produced through linkage: “in this context, these mechanisms leading to ‘x’ outcomes; and in that context, those mechanisms leading to ‘y’ outcomes”

(Westhorp, 2014, p.10). The programme theories (now full CMO configurations) are tested via appropriate data collection method(s) and data analysis. The final programme theory is presented through the findings, which are linked to CMO configurations. The findings show how they supported, refuted and refined the programme theory (see Figure 2.2) (Wong et al., 2016; Westhorp, 2014).

49 2.5.4 Rival theory

There is frequently debate in realist inquiries as to exactly how a theory works, and methods should allow a platform for this discussion (Pawson, 2006). Rival programme theories (see glossary) critique a theory and offer alternative explanations through adjudicating between different theories (Pawson, Wong and Owen, 2011).