Existing Credibility Instruments

Most credibility instruments were created in the fields of communication and education. In the discipline of communication, the media-level (e.g., TV, radio, and newspaper) credibility instruments were developed in the early period, and the source-level (e.g., broadcasting station, newspaper, or manager) credibility measures were developed later. In the discipline of education, credibility instruments were mainly developed to measure the credibility of teachers. Gaziano and McGrath (1986) developed a credibility instrument that is comprised of 12 items using principal component analysis to identify a single factor of news credibility. However, Meyer (1988) argued that Gaziano and McGrath's credibility instrument is a by-product of the semantic difference (e.g., believable and not believable) in item format. According to Meyer (1988), the response effects of the polarity of the semantic differential items artificially created the

unidimensional structure of the credibility instrument by Gaziano and McGrath. Removing the polarity artifact introduced two factors of media credibility: believability and community affiliation (Meyer, 1988).

Abdulla et al. (2004) changed the semantic differential scale to a five-point Likert scale (1 = “strongly disagree” to 5 = “strongly agree”) to address the opposing results for the factor structure of news credibility. Exploratory factor analysis revealed three underlying factors of newspaper credibility: honesty, currency, and balance. The credibility instruments used in the previous research are summarized in Table 3. There are formal instruments that have undergone reliability and validity tests through a factor analysis and ad-hoc instruments for which the author designed questionnaires but did not conduct a formal test. The “questionnaire” column of the instruments that had undergone reliability and validity tests through factor analysis was marked in bold.

Table 3. Existing Credibility Instruments

Author Concept Questionnaire Scale

Gaziano & McGrath,

1986

Credibility

Fair, biased, tells the whole story, accurate, invades or respects people’s privacy, does watch after

readers’/viewers’ interest, concerned about the community’s well-being, separate fact and opinion, trusted, is concerned about the public interest, factual, well-trained, cares what audience thinks, sensationalizes, and moral

Yes/no

Meyer, 1988 Credibility Fair, unbiased, tells the whole story,

accurate, and trusted Yes/no

Johnson &

Kaye, 1998 Credibility

Believability, accuracy, bias, and depth or completeness 1 (not at all) to 5 (very) Flanagin & Metzger, 2000

Credibility Believable, accurate, trustworthy, biased, and complete 1 (not at all) to 7 (extremely) Abdulla, Garrison, Salwen, Driscoll, 2004) Credibility

Balanced, reports the whole story, objective, fair, accurate, honest, believable, trustworthy, up-to-date, currently, timely -2 (strongly disagree) to 2 (strongly agree) Rains & Karmikel, 2009

Credibility Believable, trustworthy, accurate, complete, and biased

1 (strongly disagree) to 7 (strongly agree) Kang, 2010 Credibility

Knowledgeable, influential, passionate, transparent, reliable, authentic,

insightful, informative, consistent, fair, focused, accurate, timely, and popular

1 to 7

McCroskey & Teven,

2013

Credibility

Intelligent, untrained, cares about me, honest, has my interests at heart, untrustworthy, inexpert, self-centered, concerned with me, honorable, informed, moral, incompetent, unethical, insensitive, bright, phony, and not understanding

1 to 7

Soo Young

Rieh, 2014 Credibility Trustworthy, accurate, and reliable

1 (not at all) to 7 (very)

Rieh, 2015 Credibility

Trustworthy, accurate, reliable, useful, objective, complete, clear, easy to understand, and appealing

1 (not at all) to 7 (very) Mitra & Gilbert, 2015 Credibility Accurate -2 (certainly inaccurate) to 2 (certainly accurate) Appelman & Sundar, 2016

Credibility Accurate, authentic, believable, and reputable 1 (describes very poorly) to 7 (describes very well) Castillo et

al., 2011 Credibility “Please classify these messages as”

“Almost certainly true,” “likely to be false,” “almost certainly false,” “I

can’t decide” O’Donovan, Kang, Meyer, Hollerer, & Adalii, 2012

Credibility Impression of credibility 1 to 5, “can’t answer”

Gupta & Kumaraguru,

2012

Credibility “Rate the credibility of information present”

“Definitely credible,” “seems credible,” “definitely incredible,” “I can’t decide” Kakol et al., 2017 Credibility

Site appearance, information completeness,

author expertise, and intentions 1 to 5 Ong, Day, &

Hsu, 2009 Quality

Completeness, accuracy, format, and currency

1 (strongly disagree) to 7 (strongly agree Fichman,

2011 Reliability Accuracy, completeness, and verifiability Yes/no

There are two major issues in applying most credibility instruments to this dissertation study. First, since many credibility instruments have been developed using principal component

analysis (e.g., Gaziano & McGrath, 1986; Meyer, 1988; Abdulla, Garrison, Salwen, Driscoll, 2004), it is methodologically difficult to identify the actual underlying factors of credibility. The measurement researchers (Fabrigar, Wegener, MacCallum, & Strahan, 1999; Gorsuch, 1990; Morrison, 2009) suggest that principal component analysis is the procedure that should be used for data reduction rather than looking for the most parsimonious group of variables. Second, most of these instruments did measure the credibility of the source or medium rather than the credibility of the message or content. As will be explained later, in the case of the Yahoo! Answers data, the information that can explain the characteristics of the source is extremely limited and not shared due to privacy concerns. It is not feasible to create credibility labels focused on source credibility through trained experts or MTurk workers with the Yahoo!

Answers data. Instead, it is much more reasonable and practical to assess credibility by focusing on message or content in the unit of each answer and review based on informational content.

In recent years, instruments measuring message-level credibility have also been created. Two of these instruments were examined in depth because of the similarity between contexts of those studies and the one in this dissertation study. Rains and Karmikel (2009) investigated how people judge the credibility of health-related websites, and focused on both message

characteristics (e.g., the inclusion of statistics and testimonials) and structural characteristics (e.g., the inclusion of a privacy policy statement and third-party endorsements). Their credibility instrument asked questions about believability, trustworthiness, accuracy, and completeness. These also included questions about bias but retracted this after conducting reliability and validity tests.

Appelman and Sundar (2016) developed a credibility instrument focused on message credibility, noting that there is no instrument developed to exclusively measure message

credibility despite the demands to define credibility as three concepts (media credibility, source credibility, and message credibility). They defined message credibility as “an individual’s

judgment of the veracity of the content of communication” (Appelman & Sundar, 2016, p. 63). It was found that message credibility can be measured by using three items: accurate, authentic, and believable. Even when measured with four items, including reputable, the model had an acceptable overall fit. However, the best model was measured with three items.

After comparing two credibility instruments that focus on the message credibility, the instrument by Rains and Karmikel (2009) was selected in this dissertation study for the following reasons. First, it is because of the suitability of the item. In both instruments, believable and accurate are common items. The instrument by Appelman and Sundar (2016) has authentic as an additional item, while the instrument by Rains and Karmikel (2009) has trustworthy and

complete as additional items. Authentic information can be considered credible information. However, in terms of measuring the degree of authenticity, it will be difficult for evaluators to judge whether the individual information is original or not. On the other hand, in the case of completeness of information, some heuristics such as the text informativity (Liu et al., 2013) and quantity of information (Adamic et al., 2008; Agichtein et al., 2008; Daft & Lengel, 1986) can be utilized. Also, it is relatively more plausible to judge. This difference between two instruments may be due to the fact that the study by Appelman and Sundar (2016) focused on news content (which is a traditional topic in communication), and the study by Rains and Karmikel (2009) focused on health information on the Web. Considering that the purpose of this dissertation study is to predict the credibility of health information, the instrument by Rains and Karmikel (2009) fits this dissertation study better.

Second, message credibility does not exist independently. Message credibility interacts with source credibility. For example, if the message credibility is high, the credibility of the source providing the message will be high. Moreover, the information provided by sources with high credibility is likely to have high message credibility. This study cannot focus on source credibility due to limitations on the data and research method. However, if we can focus on the message credibility and examine some parts of the source credibility, we can examine credibility that is more realistic and similar to the perceptions of the actual users. Thus, the credibility instrument by Rains and Karmikel (2009) has better potential in that it includes an item (trustworthy) associated with source credibility.