Validation and generalisability

The research strategy employed in this study departs from the single case study approach in order to attain greater validity and generalisability for the emerging framework than could be achieved from a single case study. In particular, three strategies were used at the level of the overall research:

1. Replication of the same type of case study design; 2. Methodological triangulation;

Chapter 4. Research strategy

49

1. Replication of the same type of case study design. Replication of the same type of case study design in two separate, sequential cases (Studies 1 and 2) ensured that the methodological insights and lessons of the first case (Study 1) could be carried forward and applied to the second case (Study 2)—which could in turn be executed in a more controlled way. Furthermore, replication was also functional in exploiting synergies between the two studies: the MSAW has been invented in the US and later deployed in Europe; therefore, it was envisaged that looking at the MSAW in both the US and the EU ATM systems would provide a more comprehensive view of the system’s historical background. Finally, the use of two case studies safeguarded against the risk of the failure of one case; a tangible risk in an exploratory investigation like the present study. It is in fact acknowledged that although the exploratory mode of research involves “pushing the frontiers of knowledge” (Phillips & Derek, 2010, p. 59), it also does not guarantee that a particular research project will produce the desired outcome.

2. Triangulation. Study 3 was executed to achieve triangulation at the level of the research design. This study made it possible to cross-check the quality and validity of the initial version of the OPHAII framework by using an alternative, independent approach than initially used in the first two cases. The chosen format of this study, a SME study, consists of getting information from key informants or experts whose opinions carry weight or plausibility in a given domain. Because of their unique knowledge and understanding, relevant experts can provide extensive insights into a specific topic in the form of additional supporting or disconfirming comments and/or examples. The use of this kind of study is adequate to the qualitative, or interpretive orientation, of this research4_{. In this} type of research, the focus is usually on studying small numbers of subjects, often single individuals, single groups, or single organisations, so that truth and reliability are especially important in relation to the viewpoints of the people engaged in the context being studied5_{. Therefore, one important way in which one can achieve validity is by}

4 _{The rationale for grounding this work in the interpretive mode of research is provided under}

Appendix B.

5_{This is not to say that generalisability is neglected in interpretive research. Indeed, generalisability}

can remain a desirable property also in this mode of research. However, rather than statistical generalisability, a more plausible way for interpretive research to achieve generalisability is to rest on the argumentative logic of the researcher—argumentative logic about the extent to which research results can be applied also to contexts other than the investigated one(s). Specifically to this research, such an argument for generalisability is initiated by the reminder of this section,

seeking confirmations, refutations, and reformulations of the interpretations that develop along the course of the research (Morgan, 2006, 1997). This corresponds to searching for interpretive validity, i.e. ensuring that the research adequately represents the viewpoints, thoughts, intentions and experiences of the study’s participants (Johnson, 1994; Maxwell, 1992). Striving for interpretive validity implies protecting the research against the risk of misconstruction, i.e., the risk of failing to understand and/or misrepresent the meaning of events as understood from an insider’s perspective. It is in light of these considerations that the use of an SME study was considered appropriate for the present research. (Note that other strategies were used within each study to increase their validity, as it will be explained in § 4.6).

3. Consideration of different organisational contexts. The cases studies found in the OS literature are usually based on the consideration of either single negative cases, i.e., accidents (e.g., Snook, 2002; Turner & Pidgeon 1997; Vaughan, 1997; Weick, 1993), or single successful cases, i.e., the HRO (e.g., La Porte, 1996; LaPorte & Consolini, 1991; Roe & Schulman, 2008). Such a single case study approach would have encountered issues regarding the generalisability of its findings, as these would have been relevant only for the individual studied context. The chosen research methodology addresses this concern by maximising the number of organisational contexts included in the research. In particular, the first two case studies explicitly consider a total of three organisational contexts that have been analysed in depth. Two of these contexts (Study 2) addressed a successful and a less successful case. Finally, the third study is informed by the knowledge of different organisational contexts found in European ANSPs, as reflected in the professional backgrounds of the safety net experts involved in the study.

Having covered the aspects of validity and generalisability, this section has completed the description of the strategic aspects of the research and has set the stage for the description of the lower-level, methodological aspects of the research, which will be addressed in the following two sections.

DATA AND DATA COLLECTION

Considering the qualitative focus on the history of the MSAW, it was considered as appropriate data any key documents or insider accounts reporting on activities, decisions, and events influencing the adoption, implementation, management, human factors and safety assurance and use of the MSAW in the studied organisations. In investigations of organisational sources of failure it is normal to rely on a variety of data sources, such as:

Chapter 4. Research strategy

51

official accident reports, internal company documents (e.g., Ocasio, 2005; Turner & Pidgeon, 1997); field notes from ethnographic observations of operators at work, and transcripts from single or group interviews (e.g., Rochlin, 2011, 1996; Roe & Schulman, 2008; Roberts, 1990); a combination of both official documentation and in-depth interviews (e.g., Snook, 2002; Vaughan, 1997); and, also, on secondary data sources (e.g., Weick, 1993). In exploratory research like the present thesis, flexibility in data collection is desirable because it maximises opportunities for discovery—i.e., it maximises opportunities for finding possible organisational precursors to HAI issues wherever they can be found.

Furthermore, the data had to be representative of the viewpoints of both (i) the air traffic controllers at the sharp end, i.e., the users of the technology; and (ii) the other stakeholders at the blunt end, such as supervisors, managers, R&D directors, safety experts, and international and national regulators involved in the management of the MSAW. The former viewpoint provided insights into the actual use of the alarm in context, and the accompanying HAI issues. The latter viewpoint was instrumental to provide insights into the rationales, interests, and frames behind the (normal) decisions and conditions influencing alarm development, adoption, operation, and optimisation. Overall, these considerations justify the choice of the type of data that have been employed in this research. The data are described in the next sub-sections.