Phase A4 Coding Test with untrained users

3.5. Phase A4. Coding Test with untrained users

In order to understand the agreement on coding among users regarding topics and feelings expressed on social media pages, a user test has been performed before moving forward to the next chapter (Phase B) of this research.

The main contribution of this user test was to assess the presence of a common recognition of the main topic on a page which helped define the stimuli materials foreseen during Phase B of this study.

Success in online promotion and communication is a key factor for successful destination marketing. The way destination marketers project the image of their tourism destination in a digital context is reflected in message cues presented on web pages, such as sentences/words with positive or negative statements, title position on a page, choice of images and their position within pages, etc.

This online communication effort can be perceived by many potential visitors because of the global nature of the internet access. In this context, a special attention is given by prospective travelers in particular to social media pages, where online conversations can be easily placed by other web users, and are perceived as more credible than official sources, and in turn, might affect the decision to visit a destination (Xiang and Gretzel, 2010). However, little research has been done in order to analyze the appearance and recognition of contextual elements on tourism social media websites, in order to understand which kinds of pages characteristics are perceived as prominent by web users (Kim and Fesenmaier 2008; Dickinger, 2011; Gefen et al., 2008; Yoo and Gretzel, 2011).

Thus, this study evaluates users’ agreements on recognizing the dominant topic, and the dominant feeling expressed on social media pages, responding also to a tourism industry need to better understand how to perform effective online communication between tourism players and prospective travelers. Indeed, Destination Management Organizations (DMOs) need to create online communications that can persuade travelers to visit their destination, and to satisfy information needs. But tourism managers should pay attention to other players in the web arena, such as the contents produced by social media users, which are concurring on co-creating imaginaries and tourism narratives about the same destination (Go and Govers, 2005; 2009). Thus, the identification of the message cues that particularly affect the attention among web users might help to have a more comprehensive picture of how a destination is portrayed in the digital context. This might enable destination-marketing organizations to design more effective web strategies to attract prospective travelers and promote the value of a territory.

92 Chapter 3: Classification of public opinions expressed online

Cues Affecting the Recognition of the Dominant Topic and Sentiment Expressed on Social Media Pages

In the online environment, users might form their idea about a future vacation and/or about a destination from the contents presented online, which are based on relatively impersonal textual resources provided by other users (Brown et al., 2007).

Online pages evaluation measures have been proposed in various contexts and fields as they are crucial for the understanding of the performance of online communication (Fogg, 2003). This is particularly relevant in the hospitality and tourism field, where the massive use of internet by prospective travelers who need to search for information, inspiration and purchase for their next holidays, has open to research the communication spread via web and the perceptions by users. Park and Gretzel (2007) analyzed the main critical factors for the development of successful tourism web sites. Nine factors are actually the main research topics in this field: (1) information quality; (2) ease of use; (3) responsiveness; (4) security/privacy; (5) visual appearance; (6) trust; (7) interactivity; (8) personalization; and (9) fulfillment. Scholars argue that website evaluation (and consequent quality) can provide benefits such as customer retention, positive return on investment, and leadership within the competition (Park and Gretzel, 2007). Even if the main focus of these studies is the content of the websites, and the overall evaluation of the browsing experience (Gretzel et al., 2006; Kim and Fesenmaier, 2008), an increasing attention is devoted to the contextual elements present in tourism websites. Studies on destination websites’ persuasiveness (Loda et al., 2009) underline the importance of message credibility, which can impact on the decision to visit a destination. Scholars underline how the first impression that a user has about a webpage is crucial in order to proceed or not with the reading of the contents (Li et al., 2009). As in Kim and Fesenmaier (2008, p. 1), the perception of credibility of contextual elements can influence the decision to continue (or not) to visit a website, and “this decision is based primarily on visitors’ overall impression toward the website, and on their perception of the site inspirational value, and these factors are closely followed in importance by involvement, and by destination knowledge”.

Design of the user test

The test presented here consisted of a content evaluation of thirteen online pages gathered from popular websites about thirteen destinations representing a variety of tourism international sites. Twenty-eight international graduate students (female: 16; male: 12; age range: from 22 to 46) from an European university were selected to participate in a user test, which was conducted in May, 2012.

A room equipped with PCs was used, and the researcher was present in the room during the user test in order to ensure that the instructions were delivered clearly to all participants. Participants received an account to login to a given PC and were asked to

3.5 Phase A4. Coding Test with untrained users 93

browse 13 pre-selected online pages from tourism websites and to provide their opinions using an online questionnaire. The test took around 30 minutes to complete. Data were analyzed at the aggregate level using the software SPSS. Qualtrics (www.qualtrics.com), a professional online survey tool, was used for the development of the online questionnaire used during the user test.

The user test was performed within the following parameters:

i) selection of the social media pages as stimuli materials: four pages from Facebook.com, five pages from TripAdvisor.com, and four blog pages (mainly from TravBuddy.com) were selected, as those websites represented the main social media platforms used in the tourism online domain (Xiang and Gretzel, 2010).

ii) Original comments about destinations were left on each page.

iii) Users were asked to indicate the dominant opinion expressed on each presented page, classifying the dominant topic according to the following categories that emerged from phase A of this study:

- Products and services at the destination

- Society: culture, residents and traditions of the place

- Governance: tourism industry, institutions, and organizations - Environment: weather, safety

- Overall image of the destination

Users were asked to indicate the dominant judgments rendered about the destination on each page. The scale used was a 5-point Likert scale (1 = positive value judgments expressed; 5 = negative value judgments expressed; with the additional point 6 = the contents do not express any value judgment). Users were not given specific guidelines for their content analysis coding, leaving them free to evaluate all the cues presented on the pages.

iv) The Qualtrics online survey platform allowed for a heat map analysis, asking respondents to pick a spot on a page, which helped to underline how pages’ features captured respondents’ attention. Thus, participants were asked to select the area of the pages that communicated the most dominant opinion about the destination (e.g. main topic with positive or negative opinions presented within the page) by moving the cursor on the most interesting area and clicking on it. If users saw many topics within the page, they were asked to indicate the one that captured their attention the most.

94 Chapter 3: Classification of public opinions expressed online

User test findings

Results of the user test show that in an untrained coding context, a majority of users agreed on the recognition of a prominent feeling expressed. In particular, TripAdvisor pages resulted in generation of more agreement (Figure 3.3); then came blogs (Figure 3.4), and, lastly, Facebook pages (Figure 3.5).

This result suggests that Facebook and blog pages are characterized by a wide variety of comment types, as divergent posts and comments are allowed on these pages. Conversely, TripAdvisor tends to have more consistent comments on each page, as it allows users to create specific topic discussions. Regarding the coding results for the topic recognition task: seven out of 13 pages clearly communicated an identical message (> 50%) regarding a specific topic (Figure 3.6). In particular, pages from TripAdvisor seemed to generate more consistent perceptions of a common topic recognition.

Figure 3.3. Agreement on the prominent feeling expressed about the TripAdvisor pages (see the scale details in figure 3.5)

3.5 Phase A4. Coding Test with untrained users 95

Figure 3.4. Agreement on the prominent feeling expressed about the Blog pages(see the scale details in figure 3.5)

Figure 3.5. Agreement on the prominent feeling expressed about the Facebook pages

96 Chapter 3: Classification of public opinions expressed online

Figure 3.6. Agreement on the topic recognition

It was decided to report in this study the findings from a tentative secondary investigation performed during the user test, as the results obtained appear to encourage future research in this direction. This is a first step towards the investigation of the cues/features from social media pages which might affect perceived dominant opinion.

3.5 Phase A4. Coding Test with untrained users 97

The Qualtrics online survey platform allowed for a heat map analysis, asking respondents to pick a spot on an image, which helped to underline how pages’ features capture the respondents’ attention. Users were asked to select the area of the page that communicated the most dominant opinion about the destination (e.g. main topic with positive or negative opinions presented within the page) by moving the cursor on the most interesting area and clicking on it. If users saw many topics within the page, they were asked to indicate the one that captured their attention the most.

Figures 3.7, 3.8, and 3.9 show the graphical representations of the respondents’ selections, which are represented as colors in the matrix (red indicates that the majority of users selected that area). As the figures show, titles, pictures, presence of a rank, and negative expressions seemed to capture users’ attention the most vividly. However, as this secondary experiment did not consider principles from technology persuasion studies (Fogg, 2003), the tendency of the users to look at prominent signals on the page like titles, ranks, or attention on the right corner of the screen, or attention of the first part of the page, the outcome of the study is limited in terms of further interpretation. However, it is possible to glimpse practical implications for designers and managers who want to actively manipulate online messages.

98 Chapter 3: Classification of public opinions expressed online

Figure 3.8. Two examples of heatmaps for TripAdvisor pages