Consideration was given to the reliability and validity of the content analysis performed. Where applicable, remedial actions were identified and implemented to mitigate any shortfalls.
3.6.1 Reliability in content analysis
Reliability is defined as the degree to which the data collection can be reproduced while validity refers to how accurate the measure assesses the true phenomenon (Krippendorff, 2004). Three types of reliability designs are applicable for content analysis methodology. These are stability, reproducibility and accuracy (Krippendorff, 2004). In this research, reliability is aimed at achieving the stability and reproducibility aspects in establishing weak and medium strength reliability forms (Krippendorff, 2004).
Stability is defined as the state where the analysis process and results produced remain the same over time (Krippendorff, 2004). To achieve this aspect, the researcher recorded inducement and expectation sub-constructs on the sample company websites using the consolidated codebook. The research supervisor then reviewed these codes and either agreed or disagreed with the coding recorded.
Disagreements were discussed until agreement was achieved on the correct inducements and expectations recordings performed on sample companies.
Through the process of disagreement and working towards agreement, justifications for coding applied were provided by both researcher and research supervisor. This process fulfils the requirement for reproducibility. Reproducibility in content analysis reliability, refers to the degree that two separate researchers can apply the same measures and achieve the same results (Krippendorff, 2004).
Although disagreement on some coding was evident, much of the coding given was the same for both researcher and research supervisor. Applying the Krippendorff α calculation, reliabilities above α = 0.800 were achieved, indicating an 80% or greater level of agreement between the researcher and supervisor, thus signifying that reliable data was produced (Krippendorff, 2004).
Accuracy, as the highest form of reliability in content analysis, refers to the degree that current research compares to previous research (Krippendorff, 2004). No previous studies were found during the literature review against which the consolidated codebook, as used in this study, could be validated. By creating the codebook from a wide variety of sources and theoretical foundations, reliability of the measure was enhanced as far as possible.
Adhering to the three principles in establishing reliability as proposed by Krippendorff (2004), the following requirements were considered while creating the consolidated codebook and describing the relevant research method:
• The coding instructions are clearly indicated as definitions in the consolidated codebook.
• Coders need only apply the definitions per sub-construct, as defined in the consolidated codebook when performing the content analysis. For this reason, anybody in the research field should be able to perform the content analysis as instructed.
• To work independently would not be an issue, as the sampling unit, of websites of the Business times top 100 companies (Business Times, 2015) used, is freely available to all researchers.
3.6.2 Validity in content analysis
Determining validity in content analysis may be problematic since what is measured, may not have historical data or previous studies that can be used as reference (Babbie, 2007). Three obstacles to validity in content analysis were identified. These three obstacles are substantive, conceptual and methodological obstacles (Krippendorff, 2004).
Definitions for the three obstacles together with relevant mitigating strategies applied in this research are as follows:
• Substantive obstacles refer to the interpretation a specific audience places on texts (Krippendorff, 2004). Defining clear definitions for each construct and sub-construct can work to overcome this obstacle.
• The conceptual obstacle refers to the need for reliability in the conceptualisations of content from text (Krippendorff, 2004). Again, having clear definitions that distinguish constructs and sub-constructs, aid in mitigating this obstacle. As discussed under the reliability section, taking actions to increase reliability, mitigates this obstacle.
• Methodological obstacles refer to the absence of comparative evidence which makes validating results difficult (Krippendorff, 2004). In the absence of similar literature, following a detailed and reliable approach in research design should produce evidence which, in its own right, may prove the validity of the measure (Babbie, 2007).
Empirical validity is defined as the degree that existing theory and evidence support the research process and findings (Krippendorff, 2004). In the current research, the notion of empirical validity is sought by employing a detailed and reliable research design together with distinct theoretical based definitions for constructs and sub-constructs. Existing theories are contrasted and consolidated to produce a consolidated codebook with distinct definitions for each inducement and expectation construct and sub-constructs.
To ensure that empirical validity is achieved, evidence is required in the various parts of the research design and research method. The three main areas where validity should appear are in the content, internal structure and the relation to other variables (Krippendorff, 2004). The three parts together with considerations to the research design are described as follows:
• To establish content validity in the research refers to ensuring sample and semantic validity (Krippendorff, 2004). Sample validity implies choosing a representative sample from the population while semantic validity refers to the accuracy of analytical units used in the content analysis (Krippendorff, 2004).
These considerations are made as described in the samples and measures sections that follow.
• Internal structure validity implies structural and functional validity (Krippendorff, 2004).This refers to how sub-constructs used are defined and clustered together into constructs. Using existing theories as the basis from which the consolidated codebook is created, seek to establish both structure and functional validity of the constructs and sub-constructs used in the content analysis.
• The final area where validity is required is in the relation to other variables. This requires correlative and predictive validity to indicate how the current study ties to existing theories, literature and other studies (Krippendorff, 2004). By creating the codebook from existing theories, the sub-constructs and constructs used in the content analysis relate directly to the existing body of literature.