Validity, Reliability and Transferability

Concern with generalising findings stemming from research is rarely applicable to qualitative, participatory research as the main goal is to ‘provide thick description, or to address particularities, rather than to provide typical accounts or generalisable

findings’ (Green & Thorogood, 2014). The principal focus in changing practice within a specific, localised context for those working or living in that context often hinders the extent that participatory research can make substantial contributions outside the specific constraints of the local action (Green & Thorogood, 2014). Replication is core to experimental research in the natural sciences (Robson & McCartan, 2016), but when dealing with multiple people, working within dynamic environments, replication is a virtual impossibility – even if the same individuals were to engage in the same study again (Robson & McCartan, 2016).

130 Conversely, both PAR and design research are often concerned with the degree of transferability of its findings and conclusions, because the focus is often on direct application of methods and results (Ireland, 2003), or recommendations for

implementations within policymaking (Green & Thorogood, 2014). Transferability is the extent to which the research results can be extrapolated to different circumstances that are similar or comparable to those of the original study (Green & Thorogood, 2014). The expectation is, then, that certain aspects will have a greater degree of transferability than others, and that some adaptation to address the specificity of each context will need to occur. Transferability differs from generalisability regarding expectations concerning the predictability of results: while in the latter the higher the predictability the ‘better’ the research results are, in the former, the expectation that behaviours and phenomena can be predicted with precision is little or none.

Replication and its connection to generalisation are contentious issues in the debate around the value of quantitative versus qualitative research, which affects the confidence of people – academics and practitioners – in the findings generated by different research evidence. The apparent objectivity of numbers and statistics generated by quantitative research conflicts with the more interpretative, subjective results of qualitative research (Green & Thorogood, 2014). This conflict is critical within healthcare, where randomised controlled trials (RCTs) – the epitome of quantitative methods (Pope & Mays, 1995) – are regarded as the ultimate research design; the one that generates the ‘best’ evidence. Variations in the ‘confidence’

inspired by evidence and how the evidence was achieved (i.e. which research design was used) can be observed in some proposed attempts to categorise research by the type of evidence it generates and the level of causality and generalisability it provides.

Stern (2015) correlated design approaches, their variants and the ways in which

causality can be established (Table 3.11). His model, originally developed for evaluating the impact of public health research, makes evident an underlaying hierarchy that favours quantitative methods:

131 Table 3.11 Design approaches, variants and causal inference (Stern, 2015)

However useful these approaches may be in trying to differentiate the value of diverse types of evidence and the appropriateness of various research designs, they clearly present qualitative approaches as ‘less reliable’ or less ‘scientific’. Alternatively, emphasising that instead of better evidence these different approaches generate different evidence, and that thinking in terms of how quantitative and qualitative approaches complement each other is more productive. Along these lines, regarding health and health service research, Pope & Mays (1995) suggest that while

quantitative methods aim for reliability (that is, consistency on retesting) through the use of tools such as standardised questionnaires, qualitative methods score more highly on validity, by getting at how people really behave and what people actually mean when they describe their experiences, attitudes, and behaviours. In addition, the reasoning implicit in qualitative work is held to be inductive (moving from observation to hypothesis) rather than hypothesis testing or deductive.

Qualitative research is unlike most quantitative research where the aim is to test hypotheses and generate theories with the intention to generalise findings. Action research looks beyond defining or describing, ultimately aiming at changing reality; its enquiry attempts to answer how things could or should be better than they are in the present. Instead of articulating abstract ideas to predict and control practice, the improvement focus of PAR aims to transform practice ‘from one state to an improved state’. (McNiff & Whitehead, 2012). Thus, PAR is oriented towards improving

132 situations, which are specific to the context and people in these situations – its findings are commonly not generalisable, but they can be shared, compared and transferred to other situations (McNiff & Whitehead, 2012).

Nevertheless, acknowledging these differences does not make the problem go away. In healthcare, qualitative research needs to go the extra mile so that the evidence it generates and the claims it supports are heard and considered (Bowen, 2015). Even if the ‘golden standard’ embodied by RCTs simply does not apply to design research it still ‘conditions the culture in which design decisions are made’ (Jones, 2013). The validity of qualitative research – which is related to its accuracy, correctness and trustworthiness (Robson & McCartan, 2016) – is often criticised by academics of a more positivist viewpoint (that dominates clinical research). To confront such perspectives that have a derogatory effect and can often be used to disregard or discount the contributions of qualitative research, a variety of criteria and strategies have been proposed to support qualitative research approaches. Some of these are summarised in checklists and guidelines, such as the Consolidated Criteria for Reporting Qualitative Studies, COREQ (Tong, Sainsbury & Craig, 2007) and the Standards for Reporting Qualitative Research, SRQR (O’Brien et al., 2014).

The COREQ determines thirty-two items, distributed into three domains (research team and reflexivity, study design, and analysis and findings); while the SRQR proposes

twenty-one items across six categories (title and abstract, introduction, methods, results/findings, discussion, and other). Both these guidelines address key aspects of research design and governance, to help qualitative researchers be clearer about what they have done, why and how. Devising guidelines that facilitate making the research material explicit and more organised also enables third-parties to better analyse and criticise research choices, procedures and results.

Other authors have also highlighted measures to make sure that qualitative research is rigorous and presented objectively. Robson & McCartan (2016), after a review of Cresswell (1998), list eight items that are characteristic of ‘good’, flexible designs, including: 1) using multiple data collection techniques; 2) framing the study within a flexible design (evolving design, focus on participants’ views etc); 3) being informed by an understanding of research traditions; 4) bringing together research traditions so long theoretical and methodological rigour are maintained; 5) starting the research from a specific problem, rather than a hypothesis about causality or comparison; 6) showing rigour in data collection, data analysis, and report writing; 7) analysing data on multiple levels of abstraction; and 8) utilising clear writing to support a believable, realistic

133 narrative, accounting for the complexities of real life. These directives are perhaps less constraining than the guidelines discussed above but they, nonetheless, call attention to similar topics pertaining to the structure of the proposed research, the appropriateness of its methodological procedures and its attention to clear reporting of processes, results and conclusions.

Similarly, Green & Thorogood (2014) offer some strategies to help qualitative research to be perceived as more analytical, and to present its results as the outcomes of a valuable scientific endeavour – accounts that go beyond ‘reproducing anecdotes or colourful examples’. Their proposed strategies include: 1) an attention to evidence, as descriptive and interpretative accounts of phenomena; 2) a critical approach to subjective accounts, acknowledging that these are not ‘truths’, but rather situated, contextual accounts; 3) a critical approach to analytical accounts, entailing a continuous process of revaluation of assumptions as new data emerges and analysis progresses; and 4) a careful and rigorous analysis process, protecting it against cherry-picking, preconceptions, and built-in biases.

Despite perceptible differences in what elements are emphasised and how they should be addressed and presented, there is a common thread permeating all these

approaches⁴⁵: the concern to present qualitative research as something rigorous, methodical, analytical and, despite interpretative, not unrestrained or simply opinionative.

This research strived to adopt measures to improve organisation, internal validity, and auditability. Instances were dated, documented and archived; materials generated were photographed and stored; interviews were audio-recorded and transcribed verbatim; notes were taken during or after unrecorded meetings and conversations;

plans and procedures were updated to account for organic changes, adaptations and unpredictable events. In addition to observing good reporting and organisation of data, three complementary strategies to assist in enhancing the validity of qualitative

research (Robson & McCartan, 2016) were particularly embraced in this thesis: data triangulation, convergence of evidence and member checking.

45 Although helpful in guiding the underlying approach to research design and governance and data managing, these approaches are largely unheard of (or not mentioned) in the (participatory) design specific literature. So, in lieu of following one of these proposed checklists or guidelines, when describing and presenting the data in detail, a dialogue with more designerly approaches was preferred. This process will be properly addressed in subsection 4.3 of the following chapter.

134

3.5.1 Data Triangulation and Convergence of Evidence

Data triangulation is the deliberate use of multiple methods of data collection to obtain a more substantial and layered account of the circumstances being studied (Ritchie, 2003; Robson & McCartan, 2016). Triangulation also provides greater confirmation, clarity and precision about research findings (Lewis & Ritchie, 2003). Notwithstanding that triangulation adds complexity to how the data set will be handled (as using multiple methods may require different forms of analysis), it also provides an improved, deeper understanding of phenomena, allowing for cross-checking the validity of different findings (Green & Thorogood, 2014).

Within data triangulation, findings can be derived via a separate process of analysis – i.e. each data set leading to a subset of independent findings – or via a process of convergence of evidence whereby multiple data sources and methods converge

towards a unified body of findings (Yin, 2014). Convergence seems more appropriate in single case studies, leading to less fragmented findings focused on the situated context, which further helps making reporting and discussion clearer.

The analysis process for this research has drawn from different data sets to construct a unified narrative about healthcare staff participation, and the integration of

participatory design and behaviour change approaches to quality improvement. Figure 3.9 illustrates the wealth of data sources that were collected utilising multiple methods throughout the study⁴⁶. Around each type of data (as per Yin, 2014), the specific data sources and methods employed in this research are included and highlighted in red:

46 As described in section 3.3 above and further detailed in Chapter 04.

135 Fig. 3.9 Convergence of evidence in single case study, emphasising the data sources from the

empirical studies with LRI staff (adapted from Yin, 2014)

3.5.2 Member Checking

Member checking aims to improve validity by diminishing the presence or weight of researcher bias by making research data open to participants’ scrutiny (Robson &

McCartan, 2016). This entails utilising appropriate and accessible communication channels and language that permit participants to effectively criticise and contribute to the directions taken and the story being told by the research.

Through the present study, efforts were made to maintain an open and ongoing process of dialogue with the broad community of stakeholders, aiming at two key aspects relevant to validity: 1) making sure the stages and plans for all activities were known and agreed-upon by all or the majority of stakeholders; and 2) sharing the results and related interpretations of all activities conducted with participants, assuring that criticism, disputes and inputs were welcome without any prejudice or negative consequences to stakeholders.

136 The main channels utilised to enable member checking were email exchange and WhatsApp messaging, in addition to group discussions during in-person sessions. Email exchange entailed both generic communications about dates, times, upcoming

activities, as well as more directed communications regarding the sharing of activities results or interpretations deriving from the analysis of the data. All communication regarding the schedule and organisation of the research was directed at the extended group of stakeholders, encompassing all participants of the study that the researcher had email access to. Messages concerning the sharing of results or findings were usually limited to the people taking part in the activities in question, with the addition of a handful of other stakeholders that would likely partake in subsequent activities, requiring knowledge of past events.

Participation in member checking was diffused throughout the course of the study. A few participants were quite active in responding to materials shared, while the majority did not engage as frequently.

As it will be presented in the next subsection, enabling opportunities for member checking is of fundamental importance to participatory research on another level. In addition to augmenting the reliability of the data and the findings, it further contributes to making sure those affected by the research (also the most interested in its results) have a say in what things are emphasised and how they are presented to ‘outsiders’.

3.6 Research Ethics