Figure 5 shows the flow of participants through the online survey-based experiment, from informed consent through debriefing. This study utilizes only a portion of the variables
measured; the full text of all survey items, as well as full randomization procedures, can be found in Appendix E. What follows is a description of the measures used in this analysis.
Figure 5. Flow of participants through online-survey based experiment.
→ indicates a new block of questions progressing linearly; | indicates a new block of questions, with the blocks randomized in their presentation order. Arrows descending from blocks indicate progression of respondents through the survey; whenever more than one arrow descends from a block, respondents are randomly assigned to one of the available paths. The double arrow between the al-Qaeda set of questions and the ISIS set of questions indicates that respondents were first asked one of the two sets, and were then asked the other set (e.g., if they first were asked about ISIS they were then asked about al-Qaeda).
Dependent Variable – Perceived Threat of the NDA
The perceived threat of the NDA to the U.S. was measured using a set of three items. I randomized the order in which these questions were displayed to participants, to avoid any ordering effects. For two of the items, respondents were asked to indicate the extent to which
they agreed or disagreed (on a seven point scale ranging from -3 = strongly disagree to 3 = strongly agree) with a statement about the organization: (a) “The [NDA] is a threat to the security of the United States” (M = 0.31, SD = 1.75); and (b) “The [Framed Organization] have the will and capability to attack the United States” (M = -0.02, SD = 1.71). In addition, respondents were asked (c) “How likely do you think it is that the [Framed Organization] will conduct a terrorist attack in the United States in the next 6 months” (M = -0.47, SD = 1.78) with responses ranging from -3 (extremely unlikely) to 3 (extremely likely). These three items had good reliability (α = .880) and were loaded onto a single latent variable (NDA Threat).
Independent Variable: Prior Perceptions of the Threat from Terrorism
The primary independent variable used in this study was individual’s prior beliefs about the threat from terrorism within the U.S. Given the dependent variable was perceived threat of the NDA to the U.S., it was important that the items used to measure the
independent variable not tip respondents off to the study’s purpose. As such, I included only one direct measure of threat prior to measurement of the dependent variable: asking
respondents to report their level of concern “that the United States might suffer another terrorist attack in the next three months” with responses options ranging from -3, strongly unconcerned, to 3, strongly concerned (M = 1.09, SD = 1.50). This question was asked at the end of the first block of questions, along with questions about news consumption and interest in politics, prior to mention of any particular terrorist actors.
In addition to this question, I indirectly measured prior beliefs by asking respondents about the perceived issue important they assign to each terrorist discussed in the survey. I expected that the same psychological processes wherein individuals reported increased
concern about the threat of terrorism would produce higher levels of perceived importance related to each terrorist actor, and as such these items would load well together onto a single latent variable. At the end of each block of questions measuring individuals perceived issue importance for each terrorist actor, I asked: “How unimportant or important would you say the topic of [terrorist actor] is to you personally,” with responses ranging from -3, not at all important, to 3, extremely important (al Qaeda: M = -0.05. SD = 1.59; ISIS: M = 0.35, SD = 1.58). The answer to these three questions was then averaged (M = -0.29, SD = 1.40), to create a single indicator of DTA importance. This measure was loaded onto a single latent variable (Pre-Threat), along with the general threat measure discussed in the previous paragraph.
Moderating Variable: Executive Control
The extent to which individuals engaged executive control over the processing of the information in the article they read – that is, the extent to which they deliberated and
elaborated on the presented information – is expected to play a fundamental role as a
moderator of framing effects. This construct has been measured a variety of ways – response latencies (e.g., Carlson, Poole, Lambert, & Lammers, 2017; Faust, Balota, Spieler, & Ferraro, 1999), self-reported prior dispositions toward engaging in effortful processing (e.g.,
Hodgkinson, Sadler-Smith, Sinclair, & Ashkanasy, 2009; Soane, Schubert, Lunn, & Pollard, 2015), overall survey response patterns (e.g., Kleiner, Lipps, & Ferrez, 2015; Krosnick, 1991), surrogate measurement of motivation and ability to engage in effortful processing (e.g., Druckman, 2004; Druckman, Peterson, & Slothuus, 2013; Grabe, Yegiyan, & Kamhawi, 2008; Hopmann, Vliegenthart, De Vreese, & Albæk, 2010; J. M. Miller &
approaches, I decided to use two approaches: (a) coding of open-ended reflection responses for cognitive elaboration, and (b) measurement of time spent reflecting, and deliberating on the NDA. The resulting three indicators had acceptable internal consistency (α = .758) and were loaded together onto a single latent factor (Executive Control).
The choice of these measurement approaches over others was driven by two
considerations: feasibility, and validity. A number of possible approaches were ruled out due to limitations of the survey platform. For example, direct measure of response latencies while simple enough to measure in a lab are not possible to obtain using the Qualtrics survey platform.35 The vast majority of alternative approaches were ruled out based on validity concerns. This primarily revolved around prior dispositions and surrogate measures of
motivation and ability, which while often used are only loosely connected to the phenomenon of interest in any given context. For example, while someone with a high Cognitive
Reflection score (i.e., Frederick, 2005; Primi, Morsanyi, Chiesi, Donati, & Hamilton, 2016) may in general be more able to engage in cognitive control, that does not mean they will do so in any particular context. Additionally, something like an individual’s interest in a particular topic, a commonly used indicator of motivation, likely has multiple drivers, e.g., motivation, ability, and prior attitudes. The selected measures were a more direct and valid approach to assessing executive control in the current context.
Executive control indicator 1: Manifest elaboration in written reflection. In keeping with previous research on media framing and the processing of mediated information, immediately following reading the manipulation, I asked respondents to “Please
35 Qualtrics does provide information on the “time to first click,” however, it was possible to take the survey
write down all the thoughts, ideas, or reflections induced by reading the news story, that is, those impressions that came to mind while reading it.” This question is based off similar research attempting to assess the level of elaboration in individuals responses to mediated content (e.g., Igartua & Cheng, 2009; Shiv, Edell Britton, & Payne, 2004; Valkenburg, Semetko, & de Vreese, 1999). To translate this qualitative measure into a quantitative
measure, responses to this question were coded by a set of four undergraduate students, blind to the experimental conditions, using the following coding scheme (M = 1.50, SD = 1.20):
0: No information and no elaboration contained in the response. Includes answers which only address emotional reactance to the article. (n = 708) 1: Single concept answers, e.g., “terrorist group,” “George Bush” (n = 250) 2: Basic description, labels, definitions, and “facts.” (n = 969)
3: Simple or abstract elaboration. This generally means the respondent puts the information in the article into a broader context, which was not explicitly mentioned in the article. (n = 254)
4: Complex elaboration. This generally means the respondent provides elaboration along multiple dimensions or displays analytical thinking. (n = 136)
If everything in the response was taken purely from the article, it could not be coded above a 2. Additionally, elaboration was only coded in relation to “information” and not in relation to emotional reactance. This coding scheme was developed following a constant comparative analysis of the responses to the same question in a pilot survey, and is similar to other approaches used to assess cognitive elaboration in media studies research (e.g., Igartua & Cheng, 2009; Igou & Bless, 2007).
coded by all four coders. The coders were unaware of which responses were being coded by multiple individuals, and the selected responses were randomly distributed throughout each coding file. Krippendorf’s alpha (Kalpha) for ordinal data was used to assess inter-coder reliability, using the SPSS macro provided by Hayes and Krippendorff (see: Hayes & Krippendorff, 2007). According to Krippendorff: “To assure that the data under
consideration are at least similarly interpretable by two or more scholars (as represented by different coders), it is customary to require [Kalpha] ≥ .800” (Krippendorff, 2004, p. 429). The resultant inter-coder reliability across the four coders was 𝛼𝑘 = .822, 95% CI [.788, .849]. This falls well within the recommended range for academic research of 𝛼𝑘 ≥ .800 (see: Krippendorff, 2004, 2012), with the probability of having failed to reach at least .800 (based on 1,000 bootstrap samples) of only 7.80%.
Executive control indicators 2 & 3: Reflection and description timing measures. I indirectly assessed executive control by tracking the time spent by each respondent on the cognitive reflection page (ln-transformed M = 3.68, SD = 0.917, Range = 0.20 – 7.30, n = 2312), and the subsequent page asking respondents to describe the NDA to someone who
had never heard of the group (ln-transformed M = 3.97, SD = 0.77, Range = 1.52 – 7.02, n = 2314). Both of these activities require some amount of deliberation (see: Diamond, 2013), and this approach to assessing deliberation has been used in previous research (e.g., Schaffner & Roche, 2017). To avoid issues with multivariate outliers, extreme outliers (±3.5𝜎) were removed and treated as MAR in the final analysis.