TABLE 3: USER EVALUATIONS AND FOCUS GROUPS – DATA COLLECTION Sample: 46 BA (Hons) Communication and Media, Level 2 students

Session Participant IDs [Code, Initials, Site Evaluated] Date

1 01 (AW), 02 (CB), 03 (ED), 04 (JP), 05 (JW), 06 (LR), 07 (NW),

08 (NR), 09 (SJ) 06/03/2006

2 10 (AB), 11 (BS), 12 (CH), 13 (DB), 14 (EC), 15 (GN), 16 (KK),

17 (SH) 06/03/2006

3

18 (BC), 19 (BW), 20 (CW), 21 (HB), 22 (HM), 23 (JP), 24 (KC), 25 (KG), 26 (KM), 27 (KR), 28 (KS), 29 (LB), 30 (LC), 31 (LW), 32 (NS), 33 (RW), 34 (ST), 35 (TM), 36 (TR)

13/03/2006

4 37 (AM), 38 (BP), 39 (CS), 40 (EW), 41 (JO), 42 (JS), 43 (MK),

44 (NH), 45 (SJ), 46 (TM) 13/03/2006

Key:

18 participants evaluated the website of the Fairtrade Foundation 14 participants evaluated the website of the Soil Association

14 participants evaluated the website of Friends of the Earth (Climate Change) All 46 participants subsequently evaluated the website of The Meatrix

All sessions took place in Weymouth House, Talbot Campus, Bournemouth University, Poole BH12 5BB

but as “equals” and that they engaged in an honest, “real” and sometimes semi-informal conversation with the researcher – whom they knew quite well anyway. Many scholars (e.g. Daniels 1983 in Fontana and Frey 1994) argue that this approach can enhance the depth and validity of the data, rather than infect them with the biases of the researcher.

Session Structure and Variables

Each of the 46 users taking part in this study reviewed two sites, responded to 85 close- or open-ended questions on the four Sheets that were distributed and collected before, during and after the site visits, and contributed to a loosely structured group discussion lasting about 30 minutes.

Each of the four sessions followed a similar order (Appendix G). After being welcomed and briefed on the session’s aims and structure, participants completed two short questionnaires (Appendix H). Sheet 1 included demographic data and focused on the role of the internet in respondents’ everyday lives (v6-10) as well as asking them to assess the extent to which a range of factors motivated them to use the internet. Sheet 2 focused on the respondents’ civic attitudes and covered aspects such as their

self-perception of political engagement (v31) and awareness (v33), levels of online

engagement (v46-54) and attitude towards global issues (v56). In order to understand how broadly young people define political participation (and therefore whether their engagement in activities of a civically alternative or emerging nature could be considered as conscious), the participants were asked to rate the extent to which a list of activities was political (v34-45).

One of the central points in the entire primary research was v55 in which participants were asked to reflect on their own political engagement and to cite motivating or demotivating factors that could make them more or less active as citizens and consumers. Subsequently, they were invited to reflect on those motivating and demotivating factors and link them to their expectations from civic websites by mentioning features, materials, ideas and applications that, if adopted by the civic website, could motivate them to engage (v58). That question was divided into three response boxes (Site Content, Site Design, Site Interactivity) so as to explore the various expectations and motivations in as much depth as possible, as well as produce data that could be easily comparable, both with the data from the web content analysis, and with the participants’ own post-experience evaluations. Finally, participants were offered the chance to rate three current issues or causes (organic food, fairtrade products, climate change) according to their preference (v57).

Following the completion of Sheet 2, each individual was given one of three briefs (Sheet 3) which were identical apart from the address (URL) of the website to be evaluated. The first subgroup of participants evaluated the site of the Fairtrade Foundation; another subgroup assessed the Soil Association website; the final third of each session’s

participants evaluated the Friends of the Earth Climate Change microsite. The allocation of the briefs was based on two criteria, namely the participants’ own preferences (based on their Sheet 2 rating) and the need to distribute the briefs as evenly as possible. The

first user evaluation lasted approximately 15 minutes, after which Sheet 3 was collected.

The evaluation sheet comprised of a section that aimed to collect qualitative data, on which users were asked a series of open-ended questions covering their overall opinions on the site’s strengths and weaknesses (v59-62), and a structured section through which respondents rated the site’s content, design and interactivity (v64-66) and also indicated the likelihood of following up their online experience with a range of political actions (v67-72).

Subsequently, the entire session group was shown the 4-minute animation film that appeared on the homepage of The Meatrix – the fourth site sampled in this study. The short film is the focal point of the website and introduces the visitor to its agenda. After the screening, participants were given Sheet 4 and 10 minutes to review and evaluate the site. The two evaluation questionnaires (Sheets 3 and 4) are identical with the exception of v63, which made specific reference to participants’ expectations of (one of) the three original issue sites and, therefore, was not applicable in the case of The

Meatrix. As all sessions followed the same order of tasks and stimuli, it can be reasonably assumed that variance of responses amongst the groups cannot be attributed to order effects. Furthermore, given that this was an exploratory and largely qualitative study, the size of the sample was not deemed appropriate for a more formal experiment on the order effects of visiting websites.

Following the completion of Sheet 4, all students took part in a 30 minute group discussion, the purpose of which was to complement the individual questionnaires by capturing reactions and attitudes not covered in the Sheets, as well as to further explore issues of motivation and online uses with the added advantage of group dynamics and interaction.

This study combines elements of Group Usability Testing (GUT), Task Based Focus Groups (TBFG) and Multiple-User Simultaneous Testing (MUST) (Downey 2007, Nielsen 2007b): it involved many participants individually but also simultaneously evaluating the chosen sites. The actual structure of the sessions was similar to the protocol followed by Downey (2007), i.e. initial user profile survey, followed by the basic task of individuals reviewing the site, followed by a usability issues (group) discussion. This approach alleviates the danger of co-discovery which is common to usability focus groups (Nielsen 2007b) as users convened to groups after having experienced websites on an individual basis. The actual evaluation was largely unprompted and near natural (Nielsen 2008b) with only a couple of minor tasks being given towards the end of the evaluation of The Meatrix, so as to test certain navigation problems that had been identified during the original content analysis.

Data Entry and Analysis

The structured (close-ended) variables from the four sheets were entered into SPSS.

That data were analysed using frequencies, crosstabulations, composite variables and the visual comparison of small groups or sets of variables. The qualitative data, i.e. the open-ended responses that the participants handwrote on the sheets totalling more than

26,000 words, were typed manually into Word for each participant separately. They were treated both as self-sufficient narratives highlighting common themes, patterns, outliers, questions and issues, as well as supplementing the group discussion transcripts. Finally, these responses were also matched to individual participants’ close-ended answers and group discussion contributions so as to create profiles for each participant. This process of cross-checking and triangulation helped identify common patterns as well as

contradictions both within and across groups.

The four group discussions were manually transcribed into Word. The transcripts then went through a process of distilling that included repeated readings and comments on the margin of the text, engaging thus in a process of reflexive reading of the data (Mason 2002, Croghan etal 2006). That process led to the development of key themes and codes, which were revised throughout the analysis (Creswell 1998: 140-141), thus combining the flexibility of a grounded theory approach with a pre-existing agenda. The transcripts were then coded based on these themes; the full text of the colour-coded transcripts is included in Appendix J and the key of the codes is presented in Table B6. This approach concurs with Silverman’s (1993: 114) argument that we should treat interview accounts as “compelling narratives” that are not merely “accurate or distorted reports of reality” – in other words, they are a reality in their own merit, a

position that is consistent with the ontological paradigm of critical realism (Bryman 2001).

Limitations and Synergies

One obvious limitation of the present study is the strong gender bias. While the purpose of focus groups is not to produce a demographically representative sample of views, this study’s findings should be treated with care and in conjunction with the caveat that this is an in-depth study of a specific student community. As Coleman, Morrison and Svennevig (2006: 3) point out, “focus groups are good for gathering the range of opinions that exist in a population, but not the distribution of those opinions”.

Group discussions have a number of advantages, including being data rich, flexible, stimulating to respondents, recall aiding, cumulative and elaborative, over and above individual responses (Fontana and Frey 1994: 365). Yet, one common problem with group interviews is the potential for the discussion to be dominated by certain individuals while others remain silent. This pitfall was avoided by ensuring that all participants were asked to share their views on the sites that they had evaluated.

Insofar as the observer effect is concerned, Macefield (2007) notes that qualitative techniques (such as the post-evaluation semi-structured group interviewing of

participants) can provide indications of causation mechanisms, i.e. of the mental models users have of the interface and how that affected their response.

Furthermore, few participants reported that they did not have enough time to fully review all aspects of a site, which could be considered as a minor weakness of the study.

However, apart from the practical (time) constraints of the study, 10-15 minutes were deemed an adequate time for users to review and gain an overall impression of the

sampled sites; in reality, few users would spend more than that browsing through all the pages of a single website that they had not encountered before.

As the sessions took place during regularly scheduled seminar slots, and due to factors outside of the researcher’s control, one of the sessions featured 19 participants, which may have limited the input of each participant during the group discussion, but it ensured that the maximum number of users being able and wishing to participate actually took part and, subsequently, that a greater range of views could be collected.

3.6 Concluding Reflections

Punch observed that “fieldwork is definitely not a soft option, but, rather, represents a demanding craft that involves both coping with multiple negotiations and continually dealing with ethical dilemmas” (1994: 84, emphasis in the original). This chapter has outlined the aim, objectives, rationale and underlying philosophy of the thesis’ primary research, while attempting to address both of these issues, i.e. multiple negotiations and ethical dilemmas. The former encompass the entire process of the research design from the articulation of coherent and informed research questions, to the choice of sampling frameworks, to the collection, entry and analysis of data, in addition to the

acknowledgment of the strengths weaknesses, limitations and synergies of the approach adopted. The latter involves reflecting on potential issues of harm, consent, deception and confidentiality that may affect the study’s participants, as well as an ethical and professional rendering of the data that does not pick and choose those parts of the evidence helping the researcher’s case, but taking a reflexive approach that brings out the contradictions, unanswered questions and grey areas that emerge during the research process.

“If what I said or how I felt made more of a difference it would encourage me [to participate] more as I would think I could actually change things. Because [I] feel like I can’t or won’t have any effect on making a difference it makes you feel like ‘why should you bother’. If the government paid more attention to global warming and endangered species I may be more active as these things matter to me”.

(Participant #3.24-KC)

“I do not find it relaxing to be thinking about public affairs and I find it hard to relate to the importance of some issues within my life.”

(Participant #4.38-BP)

Chapter 4