3.3 CALCULATING INTER-CODER RELIABILITY
According to Poindexter & McCombs (2000), Inter-coder Reliability measures the consistency of coders in coding the content. There are many ways of calculating inter-coder reliability. The widely used formula computes a coefficient of reliability by stating the ratio of decisions that coders agreed on to the total number of decisions made by each coder (Holsti, 1969).
C.R. = Where:
CR = Coefficient of Reliability
M = Number of Coding decisions agreed on
157 | P a g e
N = Total Number of Coding Decisions made by Each Coder
Note:
Coefficient of Reliability is reported as a percentage. To determine the M, or the number of decisions that coders agree on, the Pretest coding must be compared code by code. Any discrepancies that are found should be flagged for follow-up discussion. The number of coding decisions that match will be used in the formula.
Although M represents the number of decisions that the coders agreed on, the formula doubles this number to represent the agreed upon decisions of two coders.
In the denominator, the total number of decisions, N, made by each coder is summed (Poindexter & McCombs, 2000).
For example, two coders matched on 180 coding decisions. Each coder made 240 coding decisions. The inter-coder reliability is
C.R. =
A rule of thumb for an acceptable Coefficient of Reliability is 80% or above. If the Coefficient of Reliability is less than 80%, the Researcher should find means to increase it before proceeding with the Content Analysis coding proper (Poindexter
& McCombs, 2000).
3.4 IMPROVING INTER-CODER RELIABILITY
According to Poindexter & McCombs (2000), the underlisted points have been tested as valid ways to increase intercoder reliability:
1. Regardless of the percentages to Coefficient of Reliability, the Researcher should discuss the discrepancies in coding with the coders.
2. The Researcher should also identify the coder who is consistent in making coding errors and replace him/her.
3. If errors consistently appear because of the coding categories or descriptions that are not mutually exclusive, the Researcher must revise the codebook.
4. Once the codebook has been revised, the researcher will then ask the coders to code another sample of content. After this, the Coefficient of Reliability will be
158 | P a g e
calculated again. Hopefully, this should increase the Coefficient of Reliability to at least 80%.
3.5 ANALYZING THE CONTENT ANALYSIS DATA
According to Poindexter & McCombs (2000), since Content Analysis is quantitative in nature, it takes after Survey data analysis in many respects. Some are itemized here:
1. Content Analysis data are analyzed the same way the survey data are analyzed.
2. However, before focusing on the specific research question, the research expert would run a frequency print out to produce frequency distributions of all variables. The researchers must verify that the variables are being read in the correct columns and then check for and clean any dirty data or out-of-range codes.
3. Once the researcher confirms that the data is clean, focus can shift to answering the Research Questions.
4. Descriptive statistics such as percentages, means, modes, and medians, are also used to analyze Content Analysis. However, as noted by Wimmer & Dominick (2011), if hypothesis tests are planned, then common inferential statistics are acceptable. Usually, the Chi-square test is the most commonly used in content analysis because content analysis data tend to be nominal in form. If the content analysis data meet the requirements of interval or ratio levels, then a t-test, or ANOVA, or Pearson r may be used.
3.6 WRITING THE CONTENT ANALYSIS DATA AND REPORT
1. The report writing stage of Content Analysis follows the same process as survey.
2. Check the draft for organization and transitions as well as correct grammar, sentence structure, and punctuation.
3. The tone should be professional and objective
4. Persuasion is permitted in the recommendation section only.
159 | P a g e
5. Proof all numbers in the text and tables to make sure they match the original printouts.
6. Ethics must be observed. Data should not be omitted or buried because they do not conform to expectations.
4.0 CONCLUSION
Content Analysis is a popular and frequently used research methodology in mass communication. It also follows that some of the steps in a typical survey are applicable in Content Analysis. Since this is a popular research methodology, the Expert Researcher should make sure his or her coders are adequately trained and mentally focused to fulfill that mandate. In fact, the entire research design must be thoroughly evaluated to achieve a valid Content Analysis.
5.0 SUMMARY
This unit focused on the concluding part of the Content Analysis discourse. We considered reliability and validity in Content Analysis as well as how the data gotten from such a research method are analyzed and reported.
6.0 TUTOR MARKED ASSIGNMENT
1. Discuss reliability and validity in Content Analysis.
2. Two different Coders coded 250 items and agreed on 210
. Calculate the
Coefficient of Reliability.3. Discuss how to practically analyze data gotten from content analysis.
7.0 REFERENCES/FURTHER READINGS
1. Poindexter, P. M. and McCombs, M. E. (2000). Research in Mass Communication (A Practical Guide). Boston, USA: Bedford/St. Martin’s
2. Sobowale, I. A. (2009). Scientific Journalism (2nd Ed.). Lagos: Idosa Konsult.
160 | P a g e
3. Wimmer, R. D. and Dominick, J. R. (2011, 2003). Mass Media Research. An Introduction. Belmont, CA: Thomson/Wadsworth
4. Lucey, T. (1998). Quantitative Techniques-An Instructional Manual. London: Dp Publications Ltd.
5. Obikeze, D. O. (1990). Methods of Data Analysis in the Social and Behavioural Sciences. Enugu: Auto-Century Publishing Company Ltd.
6. Maner, M. (2000). The Research Process: A Complete Guide and Reference for Writers. Boston: McGraw Hill
7. Otokiti, S. Olateju, O. I. and Adejumo, O. (2007). Contemporary Statistical Methods. Lagos: Vantage Publication Company
8. Wilson, D., Esiri, M., & Onwubere, C. H. (2008). Communication Research.
Unpublished Lecture Note developed for the National Open University of Nigeria (NOUN).
161 | P a g e