General Factors Affecting Methodological Selection

In the preceding section some candidate data gathering methods were listed, those less suitable for epistemological or practical reasons identified and those with most potential noted. Before a selection can be made however some further factors must be taken into consideration:

methodological and practical. The former ensure that the research output can be accepted according to the conventions and standards of the applicable literature, whilst the latter ensure that no undue risks are accepted which might prevent the project from being completed successfully.

4.3.1 Quality Considerations

Three areas from methodological theory require some discussion in the context of an interpretative, qualitative study to ensure that the research is of publishable quality, which introduces some concepts which will shape the research design.

Three aspects of research quality are routinely assessed for quantitative work in the positivist tradition:

 Validity: the degree to which the research correctly captures and reports the reality of the phenomenon being studied.

 Reliability: whether the results of the study would be consistent if repeated elsewhere.

 Generalisability: whether the findings apply to any group wider than the subjects of the study themselves.

(paraphrased from Gibbs 2007, p.91) Applicability of these quality criteria to qualitative work is controversial. Assessing reliability through consistency of results for interview data for example is problematic, since each subject will be explaining their subjective viewpoint, thus comparing responses will always be inexact.

Exact repetition between respondents might even be deeply suspicious and be suggestive of learned or somehow replicated responses (Kirk and Miller, 1986 cited in Flick 2009, p.385).

Procedural reliability can however be assessed to ensure that gathering and coding is done consistently and validity increased by referring the interview back to the respondent for confirmation (Flick 2009, pp.386–9).

Frameworks have been suggested for assessing quality in qualitative research, adopting functionalist [positivist] (Lee, 1989), interpretive (Cepeda and Martin, 2005; Klein and Myers, 1999) and Critical (Myers and Klein, 2011; Riege, 2003) paradigms, although these sources

focus particularly on case studies. Studies taking an interpretative approach using primarily qualitative data for theory building (in other words those not looking to prove a hypothesis) can arguably not be assessed on the criteria above and instead these concepts should map to more suitable alternatives, shown in Table 2.

Positivist Term Meaning Anti-Positivist Equivalent

Meaning

Validity Accuracy of capture Credibility Conclusions coherent and justifiable from the data

Table 2: Comparison of positivist concepts with anti-positivist analogues (from Remenyi, 2012 p.21; Flick, 2009; Riege, 2003).

The requirements to produce credible, dependable and transferable data are noted and taken forward into the final selection, however before proceeding, some more detailed consideration is needed of triangulation, sometimes suggested for increasing credibility, and generalisation when discussing populations in interpretative work.

4.3.1.1 Triangulation

The potential of any specific method is finite, therefore some studies aim to enrich data-based theory using triangulation of multiple methods (Flick 2009, p.444). The ultimate expression of this is the pragmatic approach, where methodological purity is disregarded and the researcher uses all available resources (Creswell 2009, pp.10–11).

Denzin (1978, cited in Jick, 1979) defines triangulation as “the combination of methodologies in the study of the same phenomenon.” It is an attempt to compare data from multiple types of studies in order to improve the reliability of the conclusions or theory drawn from it. This can be differentiated as “within-method” or “between-method”. The latter refers to where discrete methods are employed but the data sets are comparable, such that multiple views of the same object produce the same conclusions and hence increase confidence in validity. In contrast, within-method triangulation seeks to examine the same output in multiple ways thus increasing the confidence in the study’s internal consistency (Jick, 1979).

Mixed methods can include combining quantitative and qualitative approaches, to validate or objectively disprove hypotheses raised from exploratory work (Flick 2009, p.30); this is

explored in the next section. Conversely, since social science work is frequently value- and motivation-driven (Silverman, 2000) and more often adopts interpretive approaches (Gibbs 2007, pp.6–7) it may be that rather than validating results, qualitative data may provide causal insights to statistical correlations. Together both approaches can contribute the nomothetic and idiographic angles to the study of a phenomenon (Gibbs 2007, p.5).

In this case, the aim is to provide data for an ANT account, which is itself the presentation of those parts of the participants’ stories, views or accounts as the author identifies as the critical stages of some observed action. The selection of those events and interactions, and the extent of the network described, are inherently subjective (Collins and Yearley, 1992). The use of multiple techniques purely to improve access to objective truth (as implied by a more Realist ontology) was therefore rejected here. No claim is made to objective determination of absolute social truth by this work, thus it was not felt useful to dilute the available resources by attempting to pursue positivist concepts of reliability.

4.3.1.2 Generalisation

The research refers to a population (the Information Security profession) and the environments in which it operates. In the interpretive paradigm the prevailing ontology denies a universal social “truth”, considering each respondent to generate their own, but as also noted above, qualitative research is expected to be useful and applicable beyond a simple description of a unique case. This creates a dilemma. As Hume is credited with proposing, induction has no basis in strict logic, a problem which requires an act of pragmatic judgement to square (Lee 2005, pp.19–23).

Generalisation can have multiple meanings, which span from a positivist statistical proof at accepted confidence levels to moderatum generalisation. The latter is “where aspects of [a specific case] can be seen to be instances of a broader recognisable set of features … then they can form the basis of theories about process or structure”, thus allowing useful inferences to be drawn without rejecting any proposition which cannot be proven to the satisfaction of the positivist (Williams, 2000). The research design must consider whether to fully make claims for a population, in which case those analytical inferences could form hypotheses to be tested in corroborative quantitative work.

This study tends towards Dhillon’s (1995) position that it would be difficult convincingly to combine positivist methods of hypothesis testing and interpretive work based on qualitative data. Whilst it is in principle valid to take the result of one piece of work, formulate a hypothesis from it and test this in a second, it is felt that to pursue this approach within a single work shows

insufficient ontological and epistemological coherence. It is surely inconsistent to present an epistemology which presents a nuanced report of its subjects’ constructed truth but then immediately attempt to test some simplified reduction of this output to prove its universal correctness. Both parts of the study would have internal consistency and each would be valid, however that which is carried over between them is necessarily altered and partial.

Cross-paradigm triangulation was therefore rejected to ensure coherence of the single study. It was resolved instead to note from a succession of cases what can usefully be learned from them individually and also abstract what can be generally seen. The usefulness of the study therefore rests on the quality and coherency of the narrative produced.

“… from an interpretive position, the validity of an extrapolation from an individual case or cases depends not on the representativeness of such cases in a statistical sense, but on the plausibility and cogency of the logical reasoning used in describing the results from the cases, and in drawing conclusions from them.”

(Walsham, 1993 cited in Dhillon, 1995)

4.3.2 Practical Considerations

Alongside those from theory, further restrictions on method arise from proper assessment of risk and resource to ensure that the chosen strategy is practical and achievable.

4.3.2.1 Sensitivity of Data

Security presents particular challenges for participant-based research. For example, the artificial situation and lack of trust inherent in the execution of an interview might translate to reticence on the part of the interviewee to discuss sensitive issues (Myers and Newman, 2007). Kotulic and Clark (2004) attempted to validate a risk management model by survey, which attracted a response rate of less than one per cent; they concluded this to be a particularly intrusive subject area where approaches with no prior introduction are unlikely to be fruitful. Fulford and Doherty (2003) surveyed on security policy and achieved a slightly higher response rate of just over seven per cent however the design factor which increased the response is not clear.

Access to data was therefore considered a significant challenge for discourse analysis, since access to natural exchanges between a security manager and their clients would be difficult to obtain due to its sensitivity and confidentiality. Analysing the discourse of security managers talking amongst peers would be far less controversial as the conversation would be general rather than specific, however problems have been experienced here also. Bowen-Schrire et al.

(2004) conducted such research however their analysed text was eventually derived from

interviews rather than genuine inter-subject dialogue due to a paucity of suitable data. The authors noted this as a limitation and the potential for inter-subject data enhancing the study.

Ethnographies would similarly have been particularly challenging. It is difficult to imagine how it would be possible to engage in (or even simply observe) a meeting where security managers were at work without being exposed to confidential or sensitive information. In addition, ethnographic approaches frequently cannot specify at design time exactly what they aim to achieve, which can be unhelpful during negotiations with managers who will usually wish to know the boundaries of the work to which they are agreeing (Harvey and Myers, 2002).

Al-Awadi (2009) found whilst researching security policy in Oman that she did not gain the permission of any of her participants to tape-record their interviews but was able to use the

“blessing” of the Omani governmental technology agency to achieve an 81% response rate to a related questionnaire. Conversely McGee (2006) in a study of physical security managers found few problems with recording interviews and reported widespread compliance, which he partially attributed to the novelty of being personally worthy of study. McGee however similarly benefited from support from the industry body which translated to contacts who were particularly interested in professionalisation, something which he acknowledged was a potential factor. Therefore if analysis of a verbatim transcript is required, this principle should be validated during a pilot to confirm likely participant compliance.

4.3.2.2 Risks of Extended Duration of Access to Data

This project was conducted in the part-time learning mode alongside permanent employment.

This doubled the total elapsed time between entering and leaving the field and hence enabled more serious consideration of longitudinal studies in terms of the opportunity for subject change. Case or ethnographic studies embedded onsite into an organisation were however impractical unless a very infrequent access protocol could have been agreed, both with the participants and the student’s own employer.

Alternatively an ethnographic study might have been undertaken within that enterprise, which would have presented fewer issues of time and access, however this raised its own challenges.

Firstly, the fieldwork stage could potentially have lasted several years, presenting a risk that the employment might be terminated for uncontrollable personal or professional reasons (see Harvey and Myers, 2002). Secondly, changes in management attitude, trading climate or other factors could have led to the co-operation being withdrawn. As there would be no comparison study possible unless the change happened at a convenient time and the new employer was amenable, this represented a wholly unacceptable risk that the project would have been

abandoned with a total loss of data. As noted in previous sections, ethnographic and observational approaches were therefore discounted on practical grounds. For similar reasons, longitudinal studies were felt to be impractical along with other approaches which were unacceptably exposed to the risk of losing a long-term commitment to the project, such as participative action research.

4.3.2.3 Finance

As this project was privately financed, this acted as an additional filter on those methods which might be considered, although it was important that this did not prevent the project from answering the research question convincingly and defensibly. A strategy which would have required regular long distance travel would have been problematic. In addition, subject engagements could not be compensated and needed to rely purely on the altruism of the respondent. The risk to the quality of the output cannot easily be mitigated thus must be accepted and acknowledged as a potential deficiency of the overall work.

4.3.2.4 Group Activity

Again, as the study had to rely purely on the altruism of its subjects for co-operation, the selection of group activities such as a focus group or multiple-interview case study represented a considerable resource challenge. Even dyadic interviews were considered challenging; while a security manager may sanction a single hour of their own time, to commit several person-hours’

resource to such a session was unlikely to be justifiable to management. This method would therefore have required the demonstration of a very clear benefit to be considered in a business environment.

Online forums (suggested by Flick 2009, p.269) were considered more practical than in-person focus groups since the subjects would be participating in their own time. Since the human interaction would be reduced, which is the key differentiator of the method, problems of arranging access across multiple subjects simultaneously were not felt to be outweighed by the minor benefit of allowing the participants to compare their own experiences directly.