Validity and Reliability

4.8.1 Criteria for evaluating the Validity and Reliability of the Research Validity and Reliability are twin concepts for evaluating the effectiveness of the research. Validity is concerned with the extent to which the research method and instruments actually measure what they purport to measure; reliability is concerned with how consistent the research method is and whether the same findings would be replicated in a similar study conducted under similar circumstances. The traditional approach has been to check the findings for internal and external validity, reliability and objectivity. However, these criteria are more applicable to quantitative methods, though they can be extended to qualitative methods. Criteria more appropriate for evaluating qualitative research methods have been proposed by Lincoln and Guba (1985) as an alternative to more traditional methods which tended to be quantitatively-orientated. These are set out in Table 4.5

182

Table 4.5 Alternative Criteria for evaluating Qualitative Research Traditional Criteria for evaluating

Quantitative Research

Alternative Criteria for evaluating Qualitative Research

Internal Validity Credibility

External Validity Transferability

Reliability Dependability

Objectivity Conformability

Table adapted from Lincoln and Guba (1985)

The researcher used these four criteria adapted from Lincoln and Guba (1995) to assess the validity of the research method and findings. Each of these criteria is now discussed briefly.

4.8.1.1Credibility

The credibility criterion is concerned with assessing how credible or believable the results of the research are from the participants’ perspectives. It must be clear that since the purpose of this qualitative research is to investigate and understand the phenomenon of IT project planning as perceived by key project managers, that only those project managers who participated in the research are in a position to judge the credibility of the results. Accordingly, once the results have been analysed and written up, the researcher will present the draft report to the participants to establish its credibility.

183 4.8.1.2 Transferability

This refers to the extent to which the results of a qualitative research can be generalized or transferred to other contexts. Here, the primary responsibility for appropriate transferability of the findings of this research to other situations lies with the person who is generalising from these findings. For this reason, it was important to give a full description of the context of the study which has been largely achieved by means of the Case Study in Chapter 3. This would also highlight the basic assumptions underlying the research. This plenitude of information should enable the person who is considering transferring the results to another context, to be able to judge whether it would be prudent to do so.

Thus, transferability would strengthen the findings of this research but is the responsibility of other researchers who must decide on the appropriateness of such a transfer.

4.8.1.3 Dependability

The traditional view of reliability within quantitative studies relies on replication of results and findings. Thus, a reliable study would be one, which if repeated under similar conditions would yield the same results.

By contrast, dependability stresses the ever-changing context within which the research occurs. The researcher notes how circumstances change in the course of the research and how these changes have affected the way in which the researcher approached the study.

4.8.1.4 Conformability

The goal to be achieved in quantitative research is for the researcher to be as objective as possible and to remain somewhat distant from the data being

184

collected in order to avoid bias. By contrast, the researcher is very much involved with the data in qualitative research. In fact, the researcher can be considered as an instrument of data collection. In fact, much qualitative research assumes that each researcher brings a unique perspective to the study. The current researcher is directly involved in the research, particularly in that they belong to the organisation being researched. Confirmability refers to the degree to which the results could be confirmed or corroborated by others.

Conformability can be established in a number of ways. The researcher can document the procedures for checking and rechecking the data throughout the study. Another researcher can take a "devil's advocate" role with respect to the results, and this process can be documented. The researcher can actively search for and negative responses that contradict prior observations. Following the research analysis and findings, the researcher intends to conduct a data audit to assess the data collection and analysis procedures in order to identify any instances where there might have been some potential for bias or distortion in the data.

4.8.2 Implications

The underlying assumptions in this research about the nature of knowledge and how it can be obtained is that of interpetivism. This implies that the researcher will play a key role in ascertaining the meaning of the data which emerges from the semi structured interviews. This raises a number of serious problems relating to the validity and reliability of the findings.

185

The first of these problems relates to the researcher himself, who as an IT manager within the NGHA, is already a key player within the phenomenon under investigation. It could be argued that an independent “outsider” as a researcher might remove possible biases in interpreting the data. However, the culture of the NGHA, and even within distinct departments of the NGHA, is extremely complex and such an outsider might also be biased due to their lack of appreciation of the subtleties in how key players construct, through their interaction, the very culture under investigation. Moreover, as has been discussed in the previous section, in a qualitative study, the interviewer becomes involved in interpreting the meaning of the data that is collected.

Indeed, the researcher is an instrument of the research itself rather than a detached observer as is often the case in quantitative studies in the interests of objectivity.

In paying attention to the removal of bias, the researcher was greatly influenced by the “epoche” method used in phenomenological inquiries. By this method, the researcher “brackets out” their prior knowledge and presuppositions in interpreting the data obtained from interviews (Moustakas, C. 1994 pp. 84).

This is not to say that the researcher did not still have their preconceived ideas and prejudices, but rather that the researcher made a conscious and conscientious effort to bracket these out of the interpretation to allow for the interviewees’ rich understanding of the phenomena under investigation to emerge simply and non-judgementally. How this is accomplished in the analysis stage has been related by Giorgi (1979 p.83):

“(1) The researcher reads the entire description straight through....to get a sense of the whole.

186

(2) Next the researcher reads the same description more slowly and delineates each time that a transition in meaning is perceived with respect to the intention of discovering the meaning.”

The researcher, then, intended to adopt the “epoche” approach outlined in Moustakas (op. cit.) and kept in mind the steps suggested in Giorgi (op. cit.) in discovering the meaning of what was being read in the interview transcripts.

The researcher has reflected on the matter of his position within the hierarchy of power within the NGHA and how this might discourage honest disclosure on the part of his colleagues. He intends to address this by reassuring the interviewees of the purpose of the inquiry which is to obtain honest views regarding how IT projects are planned and implemented as well as their views on factors that militate against their effectiveness. The interviewees will be given assurances that their responses will be generalised and only reported or commented on in an anonymous manner. Respondents will be assured that by being open and honest they will contribute to the research and will help making recommendations for enhancing their project management. Ultimately, their honest disclosure will result in a better understanding of IT project management in which they, too, would be beneficiaries. Assurances will also be given about the security of their data and responses which will only be accessible by the researcher alone. The generalisation of the findings will make it impossible to identify any individual’s contribution; thus there should be no fears of any adverse repercussions arising from their contributions. Additionally, a time period will be allowed for any respondent to have their data removed from the research if they so desired.

187

The researcher is also aware of the many theories about meanings that can be attributed to interview data. A number of these theories, based on semantics, are concerned with the truth or falsity of statements and propositions (e.g.

Frege (1919) and Russell (1914). The researcher has been very much inspired by the viewpoint taken by Chomsky, despite his many scholarly critics Chomsky (1965). Following Chomsky, the researcher does not intend to become hampered in his interpretation of data by focusing on propositional truth or falsity. Rather in agreement with Chomsky, the data of interviews should not be subjected to the categories of “true” or “false”. Statements from an interview should not be viewed in the same way that a logician views propositions. With Chomsky, the researcher does not believe that statements from interviews should not be measured by the extent to which they are purported to correspond to external realities. In an interview, there are really no “true” or

“false” responses. What really matters is the meaning which the researcher attributes to the statements made by the respondent. These internal meanings can be picked up by the researcher because the researcher has a holistic view based on tone of voice, nonverbal cues and the overall context within which the interview is being conducted: “meanings are content intrinsic to expression”

and “are defined and individuated by syntax, broadly conceived” (McGilvray 1998 p. 225). Chomsky’s claim of a universal innate language which underlies all human languages, despite their diversity led the researcher to conclude that the situational specificity of this piece of research can be transcended with reference to its universal applicability.

188 4.9 Conclusion

This chapter has presented the Research Design underlying this study with special reference to a framework for the design. The method is mainly of a qualitative data although some use of statistics is employed to support the more qualitative statements. Issues of epistemology were addressed and a thorough discussion was given to underlying philosophical assumptions in the research.

Particular attention was given to case studies and semi-structured interviews as instruments for generating data.

The analysis of the data was outlined and ethical considerations were also discussed. Some attention was given to aspects of the interpetivist paradigm and issues of validity and reliability were outlined with special reference to the work of Lincoln and Guba (1985) who provided an alternative set of criteria than those applicable to more quantitative methods.

The following chapter will present the analysis of the data collected through the Case Study of the NGHA.