Three methodological challenges to research in naturalistic, real-world environments

Prior to a more detailed critique of the chosen methodology for this thesis, it is important to firstly outline three methodological challenges that are characteristic of research in real-world settings. These conclusions are based on lessons learnt from the literature in culmination with the experience of collecting and analysing data for this thesis. Challenges have been themed into three areas: (i) gaining access to real- world data (i.e. building and sustaining practitioner-researcher relationships); (ii) designing and collecting real-world data (i.e. distinguishing between researcher and practitioner goals); and (iii) analysing real-world data (i.e. ensuring real-world value of findings). Although these three challenges are described separately, they are by no means discrete and can interact with one another. The advice presented in this section to overcome these challenges helped to facilitate data collection for this thesis and offers practical guidance for other real-world researchers.

2.2.1 Challenges with data access: building and sustaining trusted relationships with practitioners

Research in real-world settings is contingent upon having access to data and/or participants who work within the domain of interest (e.g. emergency service commanders). Unlike traditional psychological research, which derives its strength from its predictability and the capacity to extend findings to the wider population, research in specialised, real-world domains prides itself on its ability to provide detailed and rich descriptions about the psychology of a specific group of people with common interests, experience, skills or traits. Traditional psychology seeks to collect data from large sample sizes to increase the ‘power’ and ‘representativeness’ of findings to the general population; although as most research tends to be conducted in universities, and is thus often weighted heavily towards student samples, the realistic generalisation of such findings has been questioned (Demerouti & Rispens, 2014). Comparatively, real-world research seeks to collect data from small and select samples to generate rich and in-depth conclusions. Real-world research is contingent upon access to specific individuals and so there is an imperative need to build, establish and sustain trusted and accessible relationships

29 between researchers and practitioners; an initial challenge to real-world research projects.

The data collected for this thesis required privileged access to Police, Fire and Rescue and Ambulance Service commanders who were recruited via an existing relationship between the University of Liverpool and the emergency services in the local area. Many real-world research teams involve practitioners who are interested in studying their own work domain. For example, clinical psychologists interested in research may hold dual roles acting as both a physician and a researcher when treating patients (Thompson & Russo, 2012). Likewise, educators interested in trialling new pedagogic practices may engage in ‘active research’ with their students (Locke, Alcorn & O’Neill, 2013). Researching in one’s own work domain makes access to participants easier as the researchers are already experts in their research- domain with ready and available participants; however it can also create a number of ethical problems. For example, research in clinical settings risks a diffusion of priorities between patient care and the collection of scientifically interesting data (Thompson & Russo, 2012), and the ‘consent’ of students in pedagogic research is threated as they are often unaware of the manipulation of their teaching (Locke, et al., 2013).

An alternative way to conduct real-world research, which was used for this thesis, involves collaboration between practitioners and external academics. Collaborative teams are useful as they help to better distinguish between research goals and general work goals of ‘the practitioner’. This can help to reduce some of the ethical concerns associated with purely practitioner-based real-world research, as it helps to clarify the difference between normal work behaviour and activity for research. Having both practitioners and researchers involved in a project can clarify research goals by separating academic aims (e.g. to extend scientific knowledge), practitioner aims (e.g. to provide training for practitioners) and mutually beneficial aims (e.g. to generate scientifically grounded, useful recommendations to facilitate performance).

Despite the reciprocal benefits of such relationships, academic-practitioner relationships are difficult to develop. They can take a very long time to build over a course of weeks, months and years. As such, a lot of preliminary work is required prior to data collection in order to test and build connections (Steinheider, Wuestewald, Boyatzis & Kroutter, 2012). Collaborative research is the result of trust

30 building between both researchers and practitioners, often based upon past successful collaborations (Tillyer, Tillyer, McCluskey, Cancino, Todaro & McKinnon, 2014). For example, prior to data collection for this thesis, many hours were spent working closely with practitioners from the three blue lights services; attending training events and ‘ride-alongs’ and chatting informally during downtime. There was also an existing relationship between the emergency services and the University of Liverpool as a result of past successful research collaborations (e.g. van den Heuvel, Alison & Crego, 2012; van den Heuvel, Alison & Power, 2014) and the input that psychologists within the department had provided at various strategic workshops and training events. The existence of an evidenced, mutually beneficial relationship enabled privileged access to an incredibly rare sample of participants. For example, 31 command level participants agreed to take time out of their demanding schedules to participate in 2 hour long cognitive interviews. The participants’ themselves did not gain any obvious direct benefits from participating (e.g. payment, work credit), but based on their experiences in the past (either vicariously or directly), they were enthusiastic to participate in work that can benefit working practices within their organisation.

In addition to facilitating access to data, relationship building can also help to develop a more informed understanding of the context of real-world data. Researchers who immerse themselves in the research domain of interest whilst building relationships can improve their understanding of the work domain, which may help them to develop more informed research questions and make useful changes in their data collection design (Crandall, Klein & Hoffman, 2008). For example, during relationship building for this thesis, it was noted that participants used a lot of agency-specific terminology. The researcher realised that they needed to develop a greater understanding of this language prior to data collection. This was in order to feed relevant terminology into the design of interviews by, for example, using agency-specific terminology when conducting interviews with different agency representatives (e.g., ‘NDM’ when describing the Police ‘National Decision-making Model’; ‘SRT’ to describe the ‘Search and Rescue Team’ to the Fire and Rescue Service; ‘HART’ when referring to the ‘Hazardous Area Response Team’ members with the Ambulance Service). The use of domain-specific language during interviews helped to establish a relationship with practitioners, as they did not feel the need to restrict their discussion to non-specialist vocabulary and thus described

31 incidents more freely (Pfadenhauer, 2009). An awareness of terminology further helped during the semantic coding of interviews during data analysis. Therefore, an integral element to research in real-world settings is to immerse oneself in the world of the practitioner, in order to build trusted relationships with practitioners to facilitate greater access to data and a more informed understanding.

2.2.2 Challenges to designing and collecting data: distinguishing between practitioner needs, academic aims and collective goals

Data collection in real-world settings has both great strengths and inherent weaknesses. Positively, the ability to collect data from real-world incidents or during highly immersive live or simulated training exercises means that the ecological validity of findings is incredibly powerful. The use of detailed interviews with experts and the ability for the researcher to immerse themselves in the practitioner’s world facilitates the development of quasi-expertise in the domain of interest (Pfadenhauer, 2009). This allows findings to contribute to psychological research whilst maintaining a useful real-world impact. For example, feedback from the data collected in this thesis will be fed into a presentation, which will be presented to the emergency services with detailed recommendations on the lessons that have been learnt.

Although collaborative research may overcome some the ethical challenges associated to being both a researcher and a practitioner (Locke et al., 2013; Thompson & Russo, 2012), confusion may still arise in identifying ‘research goals’ (i.e. academic findings), ‘practitioner goals’ (i.e. applied recommendations and/or training) and ‘collective goals’ (i.e. mutual benefits). When researchers wear ‘too many hats’ in trying to achieve multiple research goals it can confuse the focus of research (Seider, Davis & Gardner, 2007). For example, the ‘Hydra’ simulation that was developed for this thesis was designed to facilitate training of emergency response commanders by exposing them to multi-agency decision making in response to a ‘Maundering Terrorist Firearms Attack’ (MTFA). The scenario was developed through collaboration between the researcher for this thesis and training facilitators from each agency. Although this increased the realism and ability to input useful decision problems as identified by Subject Matter Experts (SMEs), it also created difficulties in trying to meet both research and training needs. For example, research goals involved the desire to keep injects relatively consistent and stable

32 across groups in order to facilitate quantitative statistical comparisons, whereas training goals sought flexibility in the provision of information in response to individual training needs. This can cause tension during the design and facilitation of data collection as there is no clear dominant goal. If these tensions are not addressed during the initial design of a research study, practitioners will feel frustrated as they perceive little benefit from engaging in research, whilst researchers feel that their recommendations are being ignored (Rosenbaum, 2010).

To reduce goal conflict during real-world data collection, it is important to identify common goals between practitioners and researchers. This is an important first step during data collection design as it provides an opportunity to input mechanisms to overcome any future issues that may arise. Early and clear communication about collective goals can help to close the gap between academic and practitioner perspectives, and even help to shape conclusions through collaborative theorising (Tillyer et al., 2014). Indeed, the simulation presented in this thesis was designed following close collaboration with training representatives from the three blue lights services. Prior to data collection, the research team identified that the scenario needed to remain relatively consistent across training groups in order to meet research requirements, whilst it also needed to offer a level of flexibility to challenge the training requirements of different individuals. The communication of these goals allowed a compromise to be reached, whereby nine key injects in the scenario remained linear across groups, however SMEs from each agency attended each simulation event to provide additional, dynamic information in response to training needs during the course of the simulation (see Chapters 6 and 7 for more detail). This enabled research to meet both practitioner and research aims by anticipating and adapting to potential conflicts of interest ahead of time, in order to ensure that ‘quality’ data was collected (Glaser & Laudel, 2009).

2.2.3 Challenges to data analysis: deriving informed and useful conclusions from ‘messy’ data

Data analysis can also be challenging for real-world researchers as data sets are far more diverse than those collected from traditional lab-based settings. Data collected in real-world settings is varied and diverse, including field notes, audio and video recordings, questionnaires and interviews. It can be derived from a variety of settings, including live exercises, simulations, interviews, and at times, during real-

33 world operations. This enhances the depth of data and further offers the opportunity to triangulate findings using flexible research tools; however it also means that initial data sets are often ‘messy’ and diverse as tight experimental control is not possible, nor desirable. Thus, a final key challenge to conducting real-world research relates to problems associated with analysing complex and large data sets.

Qualitative analyses are commonly utilised to make sense of data collected from real-world settings. It can be used to analyse spoken and/or written text, along with observations and field notes collected from naturalistic settings (Reavey, 2011). A more detailed discussion of the strengths and weaknesses of qualitative analyses will be outlined later in this chapter, but in short, although it has strengths in providing depth and meaning to findings, it is, rightly or wrong, often perceived as less scientific than more objective, quantitative techniques (Malterud, 2001). Although a number of methodological papers have sought to highlight the strengths of qualitative analyses by outlining criteria (e.g. rigor, sincerity, credibility etc.) for ‘excellent’ qualitative research (Tracy, 2010), there continues to be a lack of explicit and coherent information on how to analyse qualitative data sets (Malterud, 2001). Indeed the consideration of data analysis is often disregarded until large and voluminous data sets have been acquired, leaving researchers confused and overwhelmed with how to make sense of the data (Liamputtong, 2009). Furthermore, the qualitative methods that are available are often poorly or wrongly described and rarely distinguished between (Vaismoradi, et al., 2013). More recently, there has been an increase in the publication of qualitative research (Carrera-Fernandez, et al., 2014) and increased criticism of quantitative research for creating artificial objectivity that ignores the subjective influence of the researcher’s interest and prior knowledge (Parker, 2004). However, as the methodological focus on qualitative data analysis has only gained momentum over the past few years, the challenge for how to analyse real-world data sets remains.

One way of overcoming the challenges associated with real-world data is to triangulate analyses. This involves combining data from different sources in order to strengthen conclusions (Parker, 2004). For example, this thesis collected exploratory data by conducting cognitive interviews with experienced emergency response commanders, and further supplemented findings through the use of more confirmatory analyses of data collected from a controlled MTFA simulation exercise. This allowed for triangulation of data as it was possible to test the findings

34 generated from the qualitative analyses of interviews, through a more controlled quantitative analysis of data collected from the simulation questionnaires. The key themes that emerged from the interviews relating to goal orientations and uncertainty were analysed in more statistical detail via the distribution of questionnaires following the simulation. This helped to strengthen conclusions by offering a way to test interaction effects (e.g. how did goal orientations interact with delaying behaviour?). The convergence of these different approaches to analysis is useful for enabling the generation of theoretical conclusions from real-world data (Hammond, 1996) and represents a shift in methodological focus in favour of progressive and hybrid practices to improve the understanding of psychological phenomena (Katsikopoulos & Lan, 2011). Furthermore, using ‘messy’ data may actually strengthen the impact of findings as the researcher is able to more fully immerse themselves and understand the myriad of variables that influence human behaviour in the real-world, using diverse, reflexive, meaningful and specific analysis techniques (Harre, 2004). Thus, although the analysis of real-world data may be challenging, providing that the researcher makes informed considerations on how to analyse data during the early design and collection phases, then it is possible to derive great strengths from researching within rich and naturalistic settings (Liamputtong, 2009).

2.2.4 Conclusions on the typical challenges to applied research

This chapter has outlined some of the key methodological challenges associated with real-world research related to data access, data design and collection and data analysis. By providing worked examples from the data that was collected in this thesis, it has provided an indication of how these methodological challenges may be overcome. In order to address these challenges more fully, the next section of this methodology will focus more specifically on the methodological paradigm upon which this thesis was based: Naturalistic Decision Making (NDM).