In order to provide guidelines to assess the validity of case study findings, Yin (2009) defines four axes on which the quality of any empirical social research is judged:
1. Construct Validity: Identifying correct operational measures for concepts studied 2. Internal Validity: Establishing causal relationships between conditions
3. External Validity: Ability to analytically generalise the findings within a larger domain of interest
4. Reliability: Demonstrating repeatability of the findings
This section discusses the steps taken during the analysis process, in order to meet all four of Yin’s research quality requirements.
3.4.1 Investigator triangulation – Improving Internal validity
As discussed in section 3.2.1, many researchers took part in data collection and all received training from the ISRG group leader, who has significant experience in conducting research in similar settings.
Researchers also spent a lot of time discussing each other’s interpretation of the data, in order to ensure no unjustified conclusions emerged from the work presented in this thesis, or any of the other research publications related to this work.
3.4.2 Methodological triangulation – Improving Internal validity
Another way to improve the validity of case study results is to combine more than one sources of evidence (Bryman, 2003; Klein and Myers, 1999). For Company A, this was done using an already available set of survey data. The surveys were conducted based on a preliminary analysis of the interview data by Dr Adam Beautement. They presented participants with examples of scenarios where security and productivity came to conflict, drawn from the interview results, and asked them to respond to those.
To capture both employee behaviour and attitude towards friction-inducing security, two different interview scenario types were used:
Behaviour scenarios: Survey participants were presented with a scenario where a conflict between security and their primary occurs and were offered four non-compliant courses of action that would allow them to resolve the conflict. They were then asked to rank the options in order of how likely they would be to follow them and also to rate how severe is the breach of policy the chosen course of action presents. The presented courses of action are based on Schwarz and Thompson’s risk models (Schwarz and Thompson, 1990) and each one corresponds to a different risk response behaviour (individualists, egalitarians, hierarchists, fatalists – short description for each in Appendix C).
Attitude scenarios: Attitude scenarios reflected different levels of maturity on the Security Behaviour Maturity Model (SBMM – an adapted version of the Carnegie Mellon Capability Maturity Model – Appendix D, Paulk et al., 1993). A security violation or non-compliant action was presented to employees, who were then given four potential actions representing attitudes towards that action, each one referring to a different level in SBMM. Again participants were required to rank the available options in terms of how likely they were to follow each of the described courses of action.
A similar survey was conducted inside Company B, with the researcher taking part in formulating the scenarios and the corresponding responses, in collaboration with a team of 3 other researchers. As discussed at the beginning of this chapter, in total 1488 employees from Company A and 641 from Company B took part in the survey. Example scenarios for both attitudes and behaviours and the available actions can be found in Appendix E. For the purpose of this research the survey data from both organisations were used to (1) strengthen interview findings by demonstrating the prevalence of identified insecure employee behaviours in both organisations on a much larger scale than the 200 interviews. The quantitative nature of survey results also (2) allowed the two organisations to prioritise solution deployment to address the more serious of the identified issues first.
3.4.3 Pattern matching – Internal validity
Matching and comparing emerging patterns is another way to improve internal validity of case study findings. Behavioural patterns identified in early interview analyses were matched to behaviours observed in subsequent interview analyses across both organisations. This led to improved validity of the findings, together with improved applicability and generalisability to a wider domain (Yin, 2009;
Flyvbjerg, 2011).
3.4.4 Data Triangulation – Improving construct validity
As discussed at the beginning of this chapter, the primary data collection method used to devise the paradigms presented in this thesis is interviews. Investigating potential validity compromises, two factors were identified in the use of interviews as a research tool that could negatively affect the validity of the outcomes:
1. Non face-to-face interviews: Despite the majority of interviews being conducted face to face, some employees were located at remote locations or working from home. In order to include them in the sampling and improve the representativeness of the research sample, they had to be interviewed over the phone (this was a challenge present in both companies, but more prevalent in Company B). This initially seemed as a limitation, but over time it became obvious that, in both the organisations studied, employees spend a significant part of their time working remotely, and are accustomed to discussing business issues over the phone. Ignoring this element of the organisational setting by removing them from the sample would narrow the focus of the study to the few office locations where physical access was possible. This would exclude employees in smaller offices around the country or working from home, missing a large part of the organisational structure. Combining face to face and phone interviews allowed for improved data collection and better company representation.
2. Selection bias: Interviews can also result in overstating or understating the importance of identified problems due to selection bias. In addition they may provide weak insights on the seriousness of phenomena of interest (Flyvbjerg, 2011). Despite this, as previously reported, statistical generalisability was not a goal for this research; the focus was on improving current understanding of employee responses to friction-inducing security mechanisms.
Despite not aiming for statistical generalisability, a number of steps were taken to improve the validity of the emerging case study constructs:
1. Employees across different divisions were interviewed, which ensured the emerging constructs were valid across various organisational divisions. In addition the number of interviews was
Pattern matching example
Paradigms emerging from early analysis: When security policy or mechanisms impose significant overheads on employee primary tasks, employees restructure those tasks or find alternatives that they perceive as preserving security, but are less taxing on their productivity.
Patterns identified in follow-up analyses:
1. Company A: Company provided encrypted USB drives were described as “slow and problematic” for file transfers within the office. Employees used their own unencrypted ones and then deleted the data, assuming their actions preserve security.
2. Company B: Internal file sharing mechanisms were slow to setup and fine-graining access control was slow and hard to get right. Employees then used third-party file sharing facilities, ensuring only those who needed access to that information could see it.
high compared to past case studies, which also improves the validity of the constructs and demonstrates their prevalence in the examined settings (Yin, 2009).
2. The use of pattern matching to confirm findings in more than one organisations and improve internal validity also improves construct validity (Trochim, 2006).
3. Inferences were never made unless cause and effect could be identified in employee interviews.
The chain of evidence that led to the knowledge emerging from this work was purely based on what employees reported as their response to non-compliance (no pre-conceived notions existed, no hypotheses to be tested), which was then confirmed by the surveys (Yin, 2009).
4. The findings of the case studies were reviewed by the organisations involved to ensure reported outcomes were consistent and valid within the environment in which they were observed, which again improves construct validity (Yin, 2009).
3.4.5 External validity
Improving the external validity of the findings was also important, as research lacking external validity cannot be applied to settings differing from the one in which the theories were developed. This significantly narrows the scope and applicability of emerging knowledge, also reducing their usefulness to other researchers or practitioners operating in different contexts. The combination of method and investigator triangulation, pattern matching and data triangulation from different case studies aimed to provide external validity for the findings (Flyvbjerg, 2011; Gerring, 2004), which improves their generalisability (Eisenhardt, 1989).
3.4.6 Reliability
Attempts to improve the reliability of the analysis process focused on two areas:
1. The coding process and the findings were reviewed by and compared amongst the four researchers that analysed the interview data, to ensure emerging themes were consistent with the behaviours represented in those.
2. The iterative approach of the analysis (revisiting interview sets after early identification of the concepts reported in this thesis), ensured the process managed to capture a rich understanding of the examined environment and corresponding behaviours (Figure 6).
Figure 6: Iterative case study approach (Yin, 2009, p1) Prepare
Collect
Analyse Share
Design Plan