Design

The study design combined a number of direct and indirect mental model elicitation techniques from the literature50_.

Firstly, inspired by Young’s (1983) advice to observe users explaining the system, participants were asked to describe what they understood by the term “web

accessibility”. This would provide an opportunity to explore their knowledge of web accessibility, without providing any prompts or cues that might either constrain or inform their response. The question was completely open-ended, and participants were free to provide as brief or extensive a response as they preferred.

Secondly, inspired by Young’s (1983) advice to ask users to predict the behaviour of the system, participants were asked to predict a) the web development techniques that they think might benefit different user groups (e.g. blind users or physically disabled users), and b) the users groups that they think might benefit from different web development techniques (e.g. providing subtitles and captioning of multimedia content or ensuring webpages can be resized). As well as exploring the extent to which participants can “run” their mental models to predict the consequences of particular actions, this would also establish the strength of their mappings between real world tasks (i.e. supporting different user groups) and procedural actions (i.e. applying web development

techniques). Participants were encouraged to list as many or as few techniques/user groups as they wished without justification.

Thirdly, participants were interviewed about their existing working practices and attitudes towards web accessibility. In addition to further exploring their knowledge of web accessibility within the context of their work, this provided the opportunity to replicate aspects of the contextual inquiry study and to explore the broader applicability of its findings with a larger and different population of web developers.

50 _{The study was conducted alongside data collection for the i2Web Project. I was assisted in the data}

collection by two researchers working on the i2Web Project: Anna Bramwell-Dicks and Lucy Buykx. Anna conducted eight of the twenty-six interviews, Lucy conducted five interviews, and I conducted the remaining thirteen interviews. The analysis is entirely my own.

5.3.2. Participants

Twenty-six professional web developers took part in the study. Three participants were female, the rest were male. The age of participants ranged from 22 to 44 years (mean: 31.0). Three participants were from Italy; the remainder were from various parts of the UK. Five participants worked for i2Web consortium organisations. The others were recruited using opportunistic sampling from web developer mailing lists, online forums, social media, conference flyers and personal contacts in the industry. The participants had between 1 and 14 years of experience of web development, with an average of 8 years. Six participants worked for large enterprises (250+ employees), fifteen worked for SMEs (< 250 employees) and five participants were self-employed freelancers. The study was conducted at the same time as the study presented in Chapter 6. For their participation in both studies, participants were offered £30 worth of Amazon vouchers in compensation for their time and effort.

5.3.3. Materials

A recruitment flyer was distributed at the CHI 2013 conference in Paris, France. At the start of the interview, participants were asked to complete an informed consent form (see Appendix A).

Participants were asked to predict the web development practices and techniques that they think might benefit the following user groups51_:

a) blind users;

b) partially-sighted users;

c) hard of hearing users (This includes people with a mild or moderate hearing loss. This includes people who have age-related deafness. People who are hard of hearing do not use sign language);

51 _{The brief descriptions in parentheses were provided to clarify the distinction between hard of hearing}

users and deaf users and also to constrain the definition of physically disabled users to people with upper limb disabilities that may affect their use of the web.

d) deaf users (This includes people who have a severe or profound hearing loss. They may need hearing aids and may also rely on lipreading. Sign language may be their first or preferred language);

e) physically disabled users (This specifically refers to people with upper limb disabilities that may affect their use of the web);

f) people with dyslexia; and g) older adults.

Participants were asked to predict the user groups that they think might benefit from the following web development practices and techniques:

a) provide subtitles and captioning of multimedia content;

b) avoid referring only to the position or colour or shape of an object; c) provide descriptive headings and titles;

d) ensure web pages can be resized up to 200%;

e) ensure a logical order and grouping of the components on a webpage (form controls, buttons, links, tables etc.);

f) ensure text is of an appropriate size, contrast and justification, with sufficient space between letters, lines and paragraphs;

g) make sure webpages can be tabbed through and operated using a keyboard; h) provide a way of pausing or stopping rapidly updating content (flashing,

scrolling, blinking etc.);

i) include skip links to allow users to jump to the main content on a webpage or to different sections on a webpage; and

j) provide information about a user’s location within a website (e.g. using breadcrumb trails, site maps, navigation menus etc.).

The semi-structured interview schedule explored whether the participants currently attempt to make their websites accessible; whether they conduct any accessibility testing and how they go about it; and whether their clients have any accessibility requirements or expectations. The interview schedule can be found in Appendix C.

5.3.4. Procedure

Face-to-face sessions with the participants were arranged at a location convenient to them. In many cases, this was at the participant’s workplace, but several interviews were

conducted in cafés, restaurants, and at participants’ homes. Each session was audio recorded using a mobile device.

Following a brief introduction to the investigation and its aims, participants were familiarised with the interview method. They were assured that the information they would be providing would be confidential and anonymous and their consent was gained to take audio recordings for later transcription. Participants then read and signed an informed consent form. Once consent had been obtained, participants were interviewed about their mental models of web accessibility.

Once the interview was complete and the participant had been debriefed, they were thanked for their time and asked whether they would be interested in taking part in further research. The interviews lasted approximately 30 minutes each.

5.3.5. Data preparation

The audio recordings from each participant were fully transcribed prior to analysis. Sample transcripts from three participants can be found in Appendix G.

Participants’ responses to the question “what do you understand by the term ‘web accessibility’?” were analysed using a conventional content analysis approach (Hsieh and Shannon, 2005), involving several phases. Firstly, following a close read of each

participant’s response, I extracted all relevant content words; close synonyms (e.g. “blind people” and “people who are blind”) were grouped together. I then recorded the frequency with which each content word was mentioned. Content words that were only mentioned by a single participant were eliminated. From this open coding approach, I derived an initial set of 28 codes. Secondly, I grouped the codes into five broad concept categories (a list of codes and categories can be found in Appendix G). I then combined the concept categories to form an overall hierarchy of codes, representing the

participants’ aggregate mental model of web accessibility.

Participants’ responses to the prediction questions comprised: a) the web development techniques that they felt would benefit each user group, and b) the user groups that they felt would benefit from each web development technique. I scored each participant’s responses according to whether they had correctly identified one or more

techniques/user groups. To gauge the comprehensiveness of their responses, I recorded the number of techniques/user groups mentioned by each participant. I also noted any

comments made by participants during this exercise that I felt reflected their mental model of web accessibility.

The participants’ responses to the questions about their working practices and attitudes to web accessibility did not require content analysis. I collated these responses and, where appropriate, calculated descriptive statistics.

Again, conscious of being a sole researcher, and the threat to validity and objectivity that this can pose, I sought to validate my analysis of the initial question—“what do you understand by the term ‘web accessibility’?”—with an independent researcher. I did not seek to validate my analyses of the remaining questions, as they were largely descriptive. Due to the open-ended nature of the initial question, which allowed participants to provide as brief or extensive a response as they preferred, validation proved difficult. The participants’ responses were conversational and uneven and, because they did not speak in clearly delineated text units, the portions of text to be analysed (e.g. a phrase, a sentence, a paragraph) were inconsistent. Krippendorff (1995) refers to this as the

unitisation problem, due to the possibility of different coders unitising the same text

differently and disagreeing on which portions of text contain a particular meaning. This makes it hard to determine whether their coding is the same, which, ultimately, makes it difficult to calculate inter-coder reliability (Campbell, Quincy, Osserman and Pedersen, 2013).

Campbell et al. (2013) proposed a solution to this problem by having the primary, more experienced, coder first define and code the units of analysis before having the

secondary coder code these specific units. Although this approach achieves greater consistency of unitisation and makes it easier to calculate inter-coder reliability, it

introduces the possibility of bias. For example, the predefined units of analysis may alert the secondary coder to the fact that there is something to be coded in a portion of text that they may not have otherwise coded. This approach also fails to address the problem of multiple codes being applicable to a single portion of text, which further complicates the calculation of inter-coder reliability.

In lieu of clear guidance in the literature for coding semi-structured interview transcripts (for a candid discussion of the problem, see Campbell et al., 2013), I did not attempt to calculate inter-coder reliability. Instead, I invited the independent researcher to code

three of the participants’ responses to the initial exercise (a subset of approximately 10% of the collected data) using the same codes I developed during my analysis. The unit of analysis in this case was each participant’s entire response, to which multiple codes could be applied. A degree of consistency between the researcher’s and my application of the codes would then serve to corroborate, if not confirm, the results. As in the contextual inquiry study (see Section 4.2.4), I resolved any disagreements between the two coding attempts (of which there were very few) through a negotiated agreement process. Although not as rigorous as statistical intercoder reliability testing, this approach brought a degree of verification and validation to my analysis of the unstructured interview data.

5.4. Results