The exploratory nature of the research problem made standard statistical methods an unsuitable research approach. It required an exploratory approach that would allow improving the current understanding of both practitioners and researchers on how security behaviours in organisations are affected by various elements of an existing security implementation. As a result, a case study appeared to be the most
suitable method to use. Thomas (2011) defines a case study as "…analyses of persons, events, decisions, periods, projects, policies, institutions, or other systems that are studied holistically by one or more method”. It is focused on understanding the dynamics presented within single settings, investigating a phenomenon within its real-life context, revealing “Not only what, but also why and how” (Yin, 2009). A case study is a form of interpretive research (Lapke 2008, p68): it does not predefine independent and dependent variables, focusing on the complexity of human sense as the situation in question emerges, attempting to understand phenomena based on the meanings people assign to them. It can be both descriptive and explanatory (Eisenhardt, 1989), and is good for subjects where existing knowledge is fundamentally flawed or non-existent: case study outcomes can enrich the available understanding of the research issues of interest (Gerring, 2004). The case that is the subject of the inquiry is an instance of a class of phenomena, providing an analytical frame, an object, within which the study is conducted and which the case illuminates and explicates. A case study also has “…an aim to generalise across a larger set of units” (Gerring, 2004). Essentially a case study’s purpose is “not hoping to prove anything but learn something”, on a problem where lack of existing understanding prevents the use of other research methods (Eysenck, 2013).
3.1.1 Merits of case study
The use of case studies in research provides a number of advantages compared to statistical approaches (Table 4):
1. Identification of difficult to capture insights: Case studies offer insights, observations and examples not easily revealed using other research approaches. The depth of investigation allows for improved understanding of situations of interest where pre-existing knowledge is limited (Eisenhardt, 1989).
2. Providing causal explanation of phenomena. In-depth description of phenomena during a case study allows identification of causal explanations for phenomena of interest (Yin, 2009).
3. Biased towards falsifying theories. Case study research is not more biased towards verification than statistical methods. Instead there is greater bias towards falsification, as single case studies can disprove a theory by showing that a current situation is problematic; in such cases one proof of evidence is adequate for falsification of a theory (Flyvbjerg, 2006).
4. Avoids removing extreme cases. Case studies also avoid the problem of having rare, but potentially critical, situations being treated as outliers by other statistical methods. They also prevent the potential for the effect of such rare events on the phenomenon of interest being lost in normalisation (Yin, 2009).
5. Useful for theory generation. The improved situational understanding of the phenomena of interest, emerging from case studies, can be used to provide justification for generation of both theories and testable hypotheses that can drive further research (Gersick, 1988).
Case studies Statistical methods
Table 4: Case study vs Statistical methods (adapted from Yin, 2009 - p8)
3.1.2 Criticism and defence of case study as a research tool
The main criticism against the use of case studies in research is that they lack generalisability. This argument is based on the lack of statistical testing or statistical proof of causal relationships in phenomena that occur in the context examined, which often leads to researchers immediately dismissing case study findings as situation specific (Lee and Baskerville, 2003). But the truth is that case study research has no need or desire to be statistically generalisable (Lapke and Dhillon, 2008). Instead of looking for numbers to perform statistical analyses on, a case study aims to interpret a situation of interest as much as possible to: (1) provide holistic and meaningful understanding and description of real events within a problematic situation (Yin, 2009), then using that understanding to (2) generate knowledge that can be useful in tackling similar problems in the future. Consequently, generalisability of case study findings is not achieved through statistics, but by applying the acquired knowledge to similar contexts when studying new cases. In addition, emerging knowledge can be usefully applied without having to keep all other parameters constant, as traditional statistical methods strongly require (Guba et al., 1994). As a result, despite the lack of statistical generalisability, findings and knowledge generated from case studies are much easier to transfer across different complex environments than those generated by statistical methods. The case study definition of generalisability ties well with its dictionary definition: “forming general notions by abstraction from particular instances” (Oxford Dictionary of English, 2010). As Walsham (1993) puts it: there are no correct or incorrect theories, just interesting and less interesting ways to view the world and case studies can provide rich representation of such worldviews.
3.1.3 Why it was chosen
The exploratory nature of the research topic and the complexity of the research problem (insufficient existing understanding of employee security behaviours) required a research method that would focus on improving existing knowledge, instead of testing specific theories. This made a case study the most suitable approach. Case studies currently amount for a large proportion of published books and articles in
psychology, anthropology, sociology, history, political science, economics and medicine (Flyvbjerg 2011) due to their exploratory and learning-oriented nature. They have already been reported as a good research method to use in software engineering and information systems, when the boundaries between phenomena and their context is unclear, offering insights impossible otherwise; examples can be found in Runeson and Höst (2008) and Walsham (1993). Despite that, there are few reports of case studies being used in studying the application of information security in organisational environments. A notable exception is Lapke (2008), who used case studies to demonstrate that organisational power relationships significantly influence the formulation and implementation of information security policies, and Moore et al. (2011), who used those to characterise the drivers and nature of insider attacks in US-based organisations. This section outlines the reasons for which case study was chosen as a suitable methodology for the research presented in the remainder of this thesis.
3.1.3.1 Proving the existence of a problematic situation
The case studies conducted aimed to demonstrate the ineffectiveness of current information security approaches, by providing organisations with tangible examples of failures of their existing processes and mechanisms to tackle current security challenges. This proved to be a useful tool for the author (and other ISRG researchers) in attempts to provide relevant and practical suggestions for improvements to partner organisations.
3.1.3.2 Deep understanding of a problem where knowledge is limited
As previously explained, the aim of this research was to better understand and characterise how existing security controls fail to deliver compliance and how employees respond to those. A case study makes this possible by identifying causal explanations for non-compliance by revealing missing or incorrect elements of the security implementation that negatively influence employee behaviour. In addition, to the best of the researcher’s knowledge, no research has been published to date on employee security behaviours that had access to such a large dataset from more than one organisation. When access to such a dataset was secured, a case study approach would provide insights previously inaccessible and unattainable.
Directly interacting with employees during the interview phase allowed for better understanding of employee priorities and the effect of current security implementations on the organisations’ primary task-focused functions. This allowed identifying causal links between friction-inducing security and corresponding employee behaviours. The emerging knowledge proved to be useful when engaging with corporate partners: it attracted attention from industry by providing tangible evidence that current security approaches are ineffective and require improvements. In addition, the combination of two studies from different organisations allowed for widening the scope of the emerging paradigms on information security compliance, also improving the validity of the outcomes and transferability of those to solve security behaviour challenges in other organisational settings (Yin, 2009; Flyvbjerg, 2011).
3.1.3.3 Looking for rare events
Unless a security implementation suffers from severe drawbacks, information security behaviours in many organisations may appear “good enough” on a macro scale, when average-case behaviours are examined. This means that statistical testing can potentially miss infrequent events, by dismissing those
as outliers or losing those in normalisation. The rarity and “extreme case” nature of many high-risk behaviours (security breaches are often rare events, with non-compliance affected by many different contextual and individual factors), made statistical testing unsuitable for this research and case studies a more suitable method to use.
3.1.3.4 Hypothesis testing unsuitable to use
Hypothesis testing calls for a controlled environment where control of other influences to the phenomena examined exists. In addition it requires possession of sufficient knowledge on that phenomena for statistical hypotheses to be devised before the research commences. This made hypothesis testing unsuitable for this research for three reasons:
1. As mentioned in section 3.1.3.2, existing knowledge on employee behaviour by security researchers, designers or decision makers is not specific enough to devise hypotheses that accurately capture the nature of insecure behaviours and their causes.
2. Attempting to only verify a set of statistically testable hypotheses ends up focusing only on specific, predefined factors that a researcher assumes to influence security behaviours. This carries the risk of missing the bigger picture, failing to identify previously unknown drivers behind employee behaviours (Flyvbjerg, 2011; Yin, 2009).
3. It is impossible to control elements of organisational environments to avoid damaging the validity of hypothesis testing research. In organisations changes in the environment happen very quickly and are often unpredictable (this became even more obvious when research commenced and during the subsequent fieldwork): organisations have projects, audit deliverables, annual targets, changes in their target markets they may need to adapt to, even mergers, acquisitions and outsourcing of key divisions happening constantly, often at very short notice. Any attempt to engage with them to improve their security should pose a minimal impact on their production related tasks, but also be adaptable to the pace with which the above changes happen in the corporate world. This makes controlling organisational elements for statistical testing impossible, as all the above changes would invalidate the results.
3.1.3.5 Creates potential for future research
The improved knowledge on employee security behaviours that can emerge from case studies provides significant ground for future research. It allows the formulation of future testable hypotheses, grounded on real-world problems identified during the case study (and clearly outlined in case study reporting), leading to wider applicability of the findings. This proved to substantially address the “narrow focus”
problem of past information security research discussed in section 2.8 of the literature review.