The analysis process

In order to provide a clear explanation of the analysis process, it is easiest to conceive of it as four separate tasks. In practice, as will be apparent from the outline of the research process, the divides between data collection and data analysis and between different elements of analysis are not always clear within this project. As Swanborn (2010: 114) suggests, this process whereby, “data collection and data analysis are not sharply separated in time, but go hand in hand in a permanently changing order” is not unusual in case study-based research.

6.8.1 Task 1 – ToC analysis within each case study

Within each case study, analysis of the Phase 2 data was undertaken

collaboratively with the participant organisation, in order to assess the evidence of impact. This was done using a standard ToC approach (Connell and Kubisch, 1998), which involves examining the input, activity and output data (i.e. did the project do what it was supposed to?), assessing whether the outcome data met the predetermined thresholds (i.e. did the expected impacts happen?) and

considering whether any other factors might have accounted for the changes seen (i.e. did the project cause the outcomes, or did something else?). The majority of this analysis took place through the Phase 3 discussions, although there is also a sense in which formative analysis and feedback occurred in a cyclical fashion throughout all three phases, in line with most participative action research approaches (Carr and Kemmis, 1986: 184-7).

Whilst this analysis might appear relatively straightforward in theory, there are substantial issues with the level of specificity which it is possible to establish for a ToC, in terms of both indicators and causal pathways (Mackenzie and Blamey, 2005), as well as more basic challenges of measurement for short-term and long-term outcomes (Gambone, 1998). Moreover, the difficulties of establishing

counterfactuals to account for other contextual factors and the risk of

psychological bias towards assuming success make the attribution of causality through such analysis problematic (Granger, 1998).

Chapter 6

187 However, such difficulties can be re-examined to some extent in the context of this particular study, since the internal ToC analysis with each participant organisation needs to serve two purposes, both of which are somewhat immune to the

challenges experienced by other ToC practitioners. On the one hand, the entire ToC approach is intended as a process of ‘social learning and capacity building’

(Connell and Kubisch, 1998: 38) for the participant organisations, enabling them to develop monitoring and evaluation skills and systems. And on the other hand, learning about ‘what works’ within an organisation can usefully occur without the level of generalisability and certainty that may be necessary for findings to be used more broadly. As Somekh argues, what counts as ‘actionable knowledge’ is not straightforward in practice:

“There is much to be gained by adopting a dual approach:

generating contextualised knowledge on the basis of careful, systematic inquiry and evaluating this through action oriented towards improvement; while at the same time maintaining a critical scepticism and openness to different interpretations that iteratively challenge the action research ‘findings’ in terms of both the

appropriateness of the action and any claims to improvement.”

Somekh (2006: 27)

This notion of a dual approach is particularly important for this study, since the level of rigour necessary to meet the needs of community organisations may be less demanding than that required for an academic or policy audience. Such a distinction highlights an additional reason why the challenges of ToC analysis mentioned above are of less concern for this study, since the comparison of local and national ToCs is not dependent on either the success of the participant

organisations in achieving their goals or on the evaluable specificity of the models, which leads neatly on to the second analysis task.

6.8.2 Task 2 – Comparison of local and national ToCs

A complete ToC evaluation of national community participation policy would involve two elements which are not feasible within this study. Firstly, the standard

Chapter 6

188 ToC approach, involving a collaborative process of surfacing and articulating the model(s) with key stakeholders, is precluded by the impossibility of gaining such a commitment from national policy-makers, as noted earlier. Thus the models

developed in Chapter 4 may have significant analytical value, but are inevitably not specified to the level of detail which would be necessary for a full ToC

assessment. Secondly, even if the models were sufficiently detailed, the examination of inputs, activities, outputs and outcomes would not be feasible within the timescale of this project, given the nascent state of the policy framework on both sides of the border.

However, the empirical evidence from the case studies does facilitate the earlier stage of standard ToC which is to examine the ‘plausibility, doability and testability’

of the models (Connell and Kubisch, 1998; Anderson, 2005). Whilst the issues of specification make it impossible to assess testability, the case study evidence provides a valuable opportunity to explore whether the national policy ToCs make sense and can be delivered in practice, thus examining plausibility and doability. In particular, this stage of the analysis aimed to scrutinise the extent to which the underlying assumptions of national policy, set out in Chapter 4, were shown to be plausible or doable through the local ToCs and their implementation.

In order to use the local evidence in this way, a path needed to be negotiated between internal analysis of each case and comparison between cases. As Yin (2003; 2013) amongst others suggests, one potentially fruitful approach is to begin with analysis within each case, using the learning from this initial stage to inform both the further exploration of these cases and the cross-case comparative analysis. This was particularly useful here, since Phase 3 of the participative ToC process inherently involved within-case analysis, for the purpose of assessing the local ToC as well as beginning the analysis of national policy ToCs. In this respect, it was important to be aware that the local ToC diagrams represent the outcome of an analytical process with the participant organisations. Thus, deconstructing them in order to find individual snippets of data to examine national policy risked

undermining their narrative form and therefore the learning encapsulated within the models (Flyvbjerg, 2013).

Chapter 6

189 The process of exploring the case study evidence to assess the plausibility and doability of the national policy assumptions was therefore structured into four stages. Firstly, the local ToCs for each case and the assumptions underlying them were examined to identify what they said about power, the role of the state and community capacity-building. This was undertaken through the Phase 3

discussions as far as possible. Secondly, building on the points of interaction between local ToCs and national policy assumptions identified in the first stage, a more detailed examination of all of the data within each case study was

conducted, using a simple coding framework based on the relevant national policy assumptions, with a secondary level of coding to identify evidence which

suggested support for plausibility or doability and that which raised questions for these assumptions. At this stage, the local authority interview transcripts were also use to provide a means of triangulating the evidence generated with the

organisations (Simons, 2009). Thirdly, the within-case evidence from the first two stages was brought together, enabling an additional level of comparative analysis.

Thus the findings relating to the national policy assumptions were compared across the case studies, to identify similarities and differences and thereby facilitate a more sophisticated understanding of the plausibility and doability of these assumptions in different contexts. Finally, these cross-case points were utilised to inform a further reading of the evidence within each case, to check whether the original within-case analysis needed any revision in the light of findings from the other cases. At this stage, the analysis included a thought

experiment exploration of how the findings from the two compound case studies of Scotland and England related to each other, by examining how the findings from the case studies in each nation related to the policy assumptions from the other.

Where possible, these thought experiments were also discussed with participant organisations.

6.8.3 Task 3 – RE analysis of CMO configurations

The Realist Evaluation analysis was structured around the hypothesised

mechanisms of additionality identified in Chapter 5. Importantly, this study is using RE methodology in an exploratory fashion, for the process of what Pawson and Tilley (1997: 87) call ‘theory formation and development’, for two reasons. Firstly,

Chapter 6

190 the limited evidence of outcomes from community participation means that the postulated mechanisms are somewhat tentative at this stage. And secondly, the necessary breadth of data to draw convincing conclusions about

context-mechanism-outcome (CMO) configurations is not available from six case studies.

This is not to say, however, that the process is of limited value, since the aim is to start to build stronger theories which can be more rigorously tested elsewhere, addressing the limitations in the theoretical and evidential basis for community participation.

As a starting point, therefore, I examined the possible operations of the mechanisms of additionality set out in Table 5.7 in relation to each case. This analysis drew most heavily on Phase 2 of the research, during which a range of qualitative and quantitative evidence was collected to explore the range of outcomes generated in each situation. In relation to each mechanism, the

examination of each case attempted to identify whether the mechanism might be operating and whether there was any evidence of outcomes which may have been generated thereby. Again, given the focus on theory formation and development, the emphasis at this stage was on identifying any relevant evidence, rather than being concerned with the strength of that evidence.

Having examined the possible operation of each mechanism in each case, the evidence was tabulated to begin the process of identifying ‘regularities’, where the same mechanism is producing similar outcomes in similar contexts (Pawson and Tilley, 1997: 71-2). Since it would not be possible to examine the regularities relating to all 17 of the hypothesised mechanisms from Chapter 5 and, in any case, there was not sufficient evidence to consider for all of them, a selection of mechanisms for further analysis was made at this stage. This selection was based on the level of evidence for the regularity, requiring significant evidence from at least two cases, and an assessment of which mechanisms might be theoretically most interesting, following on from the previous ToC analysis.

For each of these selected mechanisms, the possible regularity was then examined by a close reading of the data within each case to identify relevant contextual factors, informed by the analysis of contexts in Chapter 5 and the detailed knowledge of each case generated through the research process. For

Chapter 6

191 each case a descriptive analysis of the contextual factors affecting the

mechanism’s operation was produced and the findings tabulated, which enabled a cross-case comparison of similarities and differences in context. From this

analysis of contexts, postulated CMO configurations were developed, attempting to identify "what works, for whom, in what circumstances" (Pawson and Tilley, 1997: 77).

Finally, the postulated CMO configurations were examined further, drawing on the findings from the earlier ToC analysis to identify implications for policy, and

highlighting potential issues with the evidence from the case studies in order to delineate areas for further research.

6.8.4 Task 4 – Methodological analysis

The final analytical task was to examine the evidence relating to the

methodological approach in order to address Research Questions 5 and 6. This essentially consisted of two main elements. Firstly, the discussions with participant organisations in Phase 3 of the research were used to reflect collaboratively on the experience of using the combined ToC and RE methodologies, to consider how useful this combination had been in practice. Perhaps unsurprisingly, these discussions were of more value in assessing the usefulness of ToC methodology for each organisation, as the RE approach had impinged in a relatively limited fashion for the majority of the research process. Nevertheless, it was possible at this stage to introduce some tentative findings from the RE analysis to explore their potential value for practice.

As noted earlier, a reflexive approach was taken throughout the research, partly in order to provide evidence for the methodological research questions. Hence the second element of this analysis involved a close reading of the fieldnotes and associated reflections recorded throughout the research process in order to identify evidence for the benefits and challenges of combining the two

methodologies. Given the relatively small amount of data, no formal coding system was used for this analysis, but key points from the original Blamey and Mackenzie

Chapter 6

192 (2007) article which had suggested the possibility of combining ToC and RE

approaches were used to inform the reading process.

6.9 Conclusion to this chapter

In this chapter I have established the specific research questions to be addressed by the study as a whole, building on the analysis of literature and policy in

Chapters 2-5. I have set out the criteria and process used to identify and recruit participant organisations for the fieldwork and provided an outline of each

organisation and area. Finally, I have described the process used for the empirical work and analysis of the findings.

The next four chapters provide the results of this analysis. Chapters 7 and 8 set out the ToC findings for the English and Scottish case studies respectively, with the final section of Chapter 8 drawing these together. Chapter 9 provides the RE analysis of mechanisms operating within the broader ToCs, whilst Chapter 10 presents the methodological findings. The results of all of these analytical elements are brought together in the final chapter, which provides a concluding discussion to the study.

193

Chapter 7 – Theories of Change analysis of Chester case