Experimental Task

Following the Pre-Task Stage, participants were asked to complete the

Experimental Task. This required them to interact with a purpose-built set of sites to use in answering questions that reflected either a high-structure or low-structure condition. These sites consisted of an invented, static, Google-branded news search results page (see Appendix C, Figure C.3) and nine branded sites, each of which hosted a single article. The search results page included links to unique articles included within each of the nine branded sites. The order of these links was randomly generated for each participant and the ordering was recorded so that our results could be controlled for an article’s positioning on this results page (i.e., site display rank).

The identity of the nine branded sites was determined based on a Google search for the term “norway elections”; the nine sites included BBC.com, DW, The Economist,

13 For this construct, the same instrument was used for this study as in Study 1. See Appendix A, Table

A.1 for items included within this instrument.

99

The Independent, Inter Press Service (IPS), LATimes.com, NBC News, the Nordic Page, and Reuters. The purpose-built branded sites included the actual sites’ mastheads as well as a branded footer (see Appendix C, Figure C.4). Each page included a link to

“Stories from Other News Sites”, which led to the related articles appearing on the other branded sites included within the experiment. The order of these links matched the display order on the search results page.

Nine different news articles related to the 2003 Norwegian general election were either substantially adapted from existing articles (in six cases) or were entirely original (three cases). The nine articles varied in length from 505 to 767 words; readability ranged from 11.0 to 12.5 on the Fleisch-Kincaid grade level readability scale (see

Appendix C, Table C.5 for article headlines and statistics regarding the content articles).

For each participant, the nine articles were randomly assigned to one of the nine branded sites, thus addressing the potential for sample bias toward a given site based on its hosting of a specific article. For any given site, the same article content was displayed within that site for the entirety of a given user’s task. Apparent links from these pages to pages not included within the experiment (e.g., link to the home page) were inactive.

Those selected into the high structure task were asked to answer ten highly specific questions about the election. Participants were instructed to restrict their search to the search results page and the articles to which that page linked. The answers to five of the ten questions (C1-C5) were common to all nine content articles (i.e., any one of the articles could be used to answer these five questions). In order to ensure that

100

participants would need to access multiple sites to complete the task, answers to the other five, “uncommon” questions (UC1-UC5) were each unique to one of the nine articles (i.e., the answer could only be found by reading the one article in which it was contained). See Appendix C, Table C.6 for the text of the ten questions. The mean of correct answers for all questions for those in the structured task condition was 7.98; the mean of correct answers for the uncommon questions (i.e., those found in only one article each) was 3.55. Note that headlines and article synopses (teasers) provided on the search results page were written such that high structure answers could not be found without accessing the article content. Participants in the high structure task group took a mean of 16 minutes, 24 seconds to complete the task.

Those assigned to the low structure task group were asked use a provided link to the purpose-built search results page and the articles found there to research the 2013 Norwegian general elections for ten minutes and then answer five questions. These five questions (see Appendix C, Table C.7) represent a low-structure task (Browne et al., 2007) in that they have multiple possible correct answers and require the answerer to develop a framework for answering. The mean number of words used in answering the five questions was 132.4 (median = 116). Participants in the low structure task group required a mean of 15 minutes 31 seconds to complete the task.

In both the high and medium borders conditions, participants were shown an interstitial page prior to being provided access to article content. Thus, when the participant clicked on a link on the search results page (or on an article page), that participant would be taken to an intermediate page and instructed to either click to

101

continue to the content page or wait 10 seconds for the content page to appear. This approach reflects actual practice on some content websites (e.g., InformationWeek, Forbes). In the high border condition, the content of the interstitial page reinforced the identity of the site that was loading; in other words, if the participant clicked on the BBC News link, the interstitial page included a large BBC News logo and the

“Continue” link included the name of the site. In the medium border condition, the participant was shown a third-party advertisement¹⁴ and no reference was made to the site being loaded. Please see Appendix C, Figure C.5 for samples of both the high- and low-border interstitial pages.

To capture participant activity during the task, participants were asked to record their activities using the Problem Steps Recorder (PSR) software found within the Windows 7 and Windows 8 operating systems. Once the task was completed, participants were asked to then upload the file generated by the PSR to the survey software environment used in the experiment. In addition, the database used to dynamically display data to participants also kept a record of which articles were

shown on which branded sites and in which order these articles were presented to users on the search results page.