Reliability of Research Process Design

According to Yin (2009) there are four main heads of consideration for the design of any research process, especially that involving the Type 3 or multiple case studies. These heads comprise:

• Construct validity

• Internal validity

• External validity

• Reliability

Construct validity

One problem associated with case studies involving mixed methods is that the operational procedures for gathering data may be seen to be based on subjective judgements could be used to collect the data. Correct operational procedures therefore need to be adopted for the concepts being measured (Yin, 2009) e.g. structured content analysis for deriving data from media. Correct operational procedures apply especially to the survey, observation and recording of the trial evacuations so that the contextual and performance issues are comparable between buildings, otherwise generalisations cannot be made. The procedures for each case study were designed with this in mind including the Exploratory case study. The construct validity of the overall PhD Case Study is

supported by multiple sources of evidence97. The procedures are described in Section 3.4

Internal validity

Internal validity in case study design is concerned with the creation of the ability available in the analysis of the data to establish causal relationships where one condition can lead to another (Yin, 2009) e.g. relationship between obesity and stair pitch to contribute to falling. It is not normally of concern in studies such as those of the trial evacuations which are just standard case studies but it can be of use in the analysis of descriptive statistical data for each trial evacuation. Patterns or trends may emerge. Pattern matching is a technique that needs to be available (Hak and Dul, 2009) here where “trends” or “directions” implied by the data can be matched between cases so that generalisations can be made (Yin, 2009). Spurious relationships can be dispensed with being one of the purposes of the inclusion of the Explanatory case studies to explain data from the main 2008-2010 case study.

External validity

One of the main reasons for selecting the case study method besides its flexibility is knowing whether or not the findings are generalisable beyond the immediate case study in question. Replication logic can be used to support this type of validity (Yin, 2009). Also when a finding appears to be generalisable such as the causes of falling the finding that may be generalised is the individual is hurrying (Mademli et al, 2008) which can represent a group of factors.

97 _{One of the main major strengths of mixed methods is the use of triangulation to tie the} evidence together so that reliable conclusions can be drawn. This also includes the integration of data from the Explanatory case studies which form part of the 2008-2010 case study in

association with the trial evacuations. The chains of evidence also need to be clear. The reliability is therefore drawn form rigorous properly applied operational procedures.

113

Replicating can be achieved across a number of cases and is one of the reasons for the selection of a range of buildings and stair types in the 2008-2010 trial evacuation study.

Reliability

Reliability is most likely the one that is the most familiar in research design. This relates to the replication of protocols between cases (Yin, 2009). For example the content analysis procedure followed between the WTC9/11 incident study and the New York Times Blog study used a common information classification framework and then axial coding to populate the classifications (Strauss and Corbin, 1998). The classifications comprise the context made up of extrinsic and intrinsic factors associated with descent of multiple flights of stairs. Reliability is directly improved by the use of RCA Analysis across all the case study types (Portwood and Reising, 2007) and the associated Ishikawa Chart (Ishikawa, 1982).

Triangulation

Triangulation of the data in the PhD Case Study mainly applies to the 2008-2010 trial evacuation case studies (part of the 2008-2010 case study). There are three sets of data from the trial evacuation studies being:

• Survey based i.e. survey of the office workers completing the trial evacuations copies of which may be found for each cycle in the Appendix A3.

• Observations by observers in accordance with a written set of procedures from Dictaphone sound files where the observers descended the stairs with the office workers from each trial evacuation and recorded their progress.

• Observations of video captured visual images of evacuating office workers where their progress, pattern of movement and intrinsic characteristics are recorded to a time based stair descent spread sheet using Excel

. The x-axis would represent the time at entry of the first evacuee into the stairs extending to the time that last person passed through the final exit to that stair. The y-axis represents the number of levels in the building.

The process of triangulation will be in accordance with the guidelines set down by Hales (2010). Triangulation relates to evidence and to its reliability. It is summarised and explained in Figure 3-7

Figure 3-7: Check for convergence of evidence in PhD Study (Derived from Yin (2009)) – Green highlighted boxes indicate techniques used

The collection from multiple sources places a burden on the researcher, but can be extremely useful in checking evidence i.e. showing that evidence converges. A simple example of this is the formation of groups. The survey respondent indicates that they entered a stair with a friend or in a group as a direct answer to

115

a question in the survey questionnaire. The observer descending the stairs as part of a group can confirm this as can an observer transcribing video captured evidence to a spread sheet (Boxes 2 and 4). The range of group behaviours that could be expected could also be triangulated (Boxes 1, 2 and 5). The classification of the context could also be checked (Boxes 1, 5 and 6). Where the findings confirm one another then the issue being checked is successfully triangulated. Even when one piece of evidence does not “converge” with the other, they may still be used to explain what is happening (Yin, 2009). Figure 3- 7 therefore represents an overview of the triangulation process used in the PhD Study widely used in Chapter 7 and to establish findings in Chapter 8.

Conclusions on case study design

Reliability and validity are both grounded in evidence and method protocols. The design must therefore:

Show that the analysis relied on all available evidence.

Challenge the analysis via the main rival theories e.g. obesity vs. descent speed or obesity vs. fatigue (Galea et al, 2008; Proulx et al, 2007; and Peacock et al, 2009).

Addresses the most significant aspect of each case study even if the data presented is in the form of “outlier”98 events such as a fall (Pauls, 2011). Uses the author’s prior, expert knowledge and experience to further the analysis as he was immersed (Yin, 2009) in both the exploratory and 2008-2010 case studies.

98 _{The term outlier is used here in terms of frequency where the outlier represents a very low} frequency of occurrence of a variable that is an extreme of the action of descending stairs which comprises a series of small falls from which the person recovers (stability) whereas a fall as defined in the text is where someone comes to rest on the ground and is most likely injured (Tinnetti et al, 1988). See also Pyle (1998) for definition of outlier dealing with frequency of occurrence

Finally considering the objective of the PhD Case Study (1.3.3) it is necessary to consider the concept of categorical aggregation (Tellis 1997) as a more comprehensive method of analysis to pattern matching (Hak and Dul. 2007). Multivariate regression is extremely useful when the objective of a study is to test a relationship in the context of many other contextual or explanatory variables. A great deal of the data gathered has been coded into a categorical format so that some form of categorical aggregation may be required. This would mean the use of Multivariate Regression Analysis (Liang et al, 1992). Further reading and comparison of examples put forward by Liang et al, (1992) show that Logistic Regression if properly constructed can provide results that are comparable with the Multivariate approach (Miles and Shelvin, 2001).