Rigor Reliability and Validity

Traditional quality criteria (in case study method or otherwise) of rigor are the validity and reliability of the data and the conclusions achieved by the research (Bryman, 2012; Yin, 2013). Establishing tests to check reliability and validity of quantitative data are well documented but in the case of qualitative data, examining reliability and validity of

data can be more challenging (Whittemore, Chase, & Mandle, 2011). Weiner (2007) defines reliability as “the degree to which a measurement technique can be depended upon to secure consistent results upon repeated applications”; Weiner defines validity as the “degree to which any measurement approach or instrument succeeds in describing or quantifying what it is designed to measure”. Yin (2013) gives four common tests on validity: construct validity, internal validity, external validity, and reliability. The final item reliability is a necessary but not a sufficient condition for validity (Bryman, 2012; Weiner, 2007).

3.4.2.1 Construct Validity

Construct validity identifies if the research study has used correct operational measures to measure what should actually have been measured in the study (Yin, 2013). In general, social research involves measurement of concepts (constructs) that require multiple measures to capture the richness of the concept (Hoyle, Harris, & Judd, 2002). Within a case study framework, construct validity can mean “multiple source of evidences” that provide a “chain of evidence” and/or when the key informants have reviewed a draft case study write-up for factual accuracy of information provided by the informants (Yin, 2013).

Constructs in this study are the specific concepts that the researcher attempts to capture using multiple measures. In social research constructs are the basic elements that a researcher uses to explain a social phenomenon (Bryman, 2012; Hoyle et al., 2002). Examples from the IT discipline could be “Agile maturity”, “intensity of IT use” and similar concepts that are not directly observable. Thus a social researcher has to identify appropriate operational measures that represent the relevant constructs (concepts) of their study. For this study an example of operational measures—for the construct “conciseness of documents” related to RQ2—include relevant records which can be presented in the documents in a Scrum project to gain ISO 9001:2008 certification. Yin (2013) asserts that constructs and their operational measures should be defined explicitly prior to the commencement of data collection, along with a relevant explanation on what is expected from the audience. This protocol was followed by the researcher.

Yin (2013) asserted that the primary strategy a case study researcher should adopt to addresses construct validity is to use “multiple sources of evidence”, which is also known as triangulation in social research (Gibbert, Ruigrok, & Wicki, 2008). Multiple sources of evidence could be data collected at different times, locations, and from different people; use of more than one investigator; and the use of multiple methods (Lindgreen, Hingley, Stavros, & Westberg, 2009; Mathison, 1988).

In this study data were collected using focus groups, mainly through a semi-structured survey. This included multiple rounds of interviews targeting best suited participants for each round. Each round was carried out leaving a 4-6-week gap between each round. Thus data collection was accomplished using different people at different times using different data collection methods—surveys, interviews, focus group briefings, brainstorming—filling the multiple sources of evidence (on the same phenomenon) requirement for construct validity. The reader should note that use of different locations and investigators was not possible due to resource constraints and academic reasons respectively.

The second strategy that Yin (2013) prescribed to strengthen construct validity in case- study research, “establishing a chain of evidence”, was implemented by formulating logically related (sequential) research questions and marshalling evidence in a sequential fashion (the 3-step process is described later).

The third strategy that Yin (2013) prescribed to strengthen construct validity in case- study research, to review the case study report by key informants (respondents), was implemented as follows. In this study data were gathered via questionnaires using multiple rounds and prior to commencement of the next round (and also at the end of the final round) the researcher briefed the findings of the previous round to each participant via the questionnaire. For example, the introduction and background mentioned in the questionnaire used in the second round of data collection was the key findings of the previous round of data collection.

3.4.2.2 Internal Validity

(Yin, 2013). Internal validity refers to the ability of the study’s research design to exclude any rival explanations (confounding effects) of the phenomenon being studied (Brewer, Reis, & Judd, 2000; Bryman, 2012). Since this study is an exploratory study, the necessity to implement specific strategies to enhance internal validity did not arise.

3.4.2.3 External Validly

The third type of validity test is external validity. External validity refers to the domain within which the findings of a research can be generalized (Yin, 2013). In social research, there are two types of generalizations: statistical generalization and analytical generalization (Yin, 2013). Statistical generalization refers to making a statistical inference on the population based on the sample. The cases (units of analysis) in a case study are not sampling units drawn from a population, and hence statistical generalization does not apply in case study research; what is applicable in case study research is analytical generalizability, where a researcher provides a theoretical basis for the case study findings (Yin, 2013). This theoretical basis is in turn used to explain events similar to what was observed in the case study. In this research, the researcher has argued that the results produced can be reproduced in a context similar to that of the case study, namely a Scrum-based software development environment in which ISO 9001:2008 accreditation has become necessary.

3.4.2.4 Reliability

An important facet of validity (more technically construct validity) is reliability. In a strict sense, reliability is a positivistic notion related to the measures used to operationalize a social concept (construct). In positivism reliability refers to the extent to which a measure used to operationalize construct is “free from random error” (Voss et al., 2002). Thus reliability refers to extent to which repeated measurements (by the same person or a different person) produce similar results (Voss et al., 2002; Yin, 2013). In a case study context, reliability can be established by a clear case study protocol and maintaining records of the evidences such as a case study database (Yin, 2013).

Protocol used in this research is the documented procedures, which can be repeated throughout the research; also these can be used by another researcher. In this research

data collection is done by using sets of questionnaires, which enable all the participants to provide their feedback using a unified data collection framework. Semi-structured focus group interviews were carried out iteratively in this research by following same pattern and guidelines throughout the research.

Also, records of all the responses received were maintained in a database file, among other things, for verification purposes. The database contained all the relevant journals, articles, questionnaires, recording of the results, and notes taken via the interviews. Moreover, the researcher has provided a clear description of the data sources and how the findings follow from the data, which is an important aspect of reliability and validity of the findings (Benbasat et al., 1987).