Chapter 3. Method
3.6 Data analysis
3.6.1 First phase of coding of user problems– definition of classification scheme
The aim of the first phase of coding was to build up a classification scheme for user problems, based on a mutually agreed set of categories established by the independent analysis by different coders.
A selection of videos for this first phase included users from the different disability groups and a range of different websites. Each video was initially coded independently by three different coders (the author and his two supervisors), who identified
accessibility problems and assigned them an initial classification and severity rating. After the independent coding of the videos, the three coders met to compare their initial identifications and classifications. During these meetings, a unified list of
problems identified by all the coders was produced. Each problem was classified and a descriptive category was created. Based on the categories that emerged from these discussions, a classification scheme itself was built up.Table 3.3 shows an example of some problems compiled in one of the initial sessions to build the classification scheme
This initial phase of coding involved coding a set of 11 videos with blind, partially sighted and dyslexic participants on the British Museum, Lflegal and Ticketmaster websites.
68 68
Table 3.3. General list of problems identified by independent coders in the initial phase of coding
Problem No/Time
Identified by Problem description from user/coder Preliminary code Type P1 0:27 CP “21 headings”. User grimaced, resigned voice.
Heading Too many headings Coder identified P2 4:38
CP, HP, AF “Heading level 2 highlights” .. I’m not really sure what highlights is.. what it’s trying to indicate Headings Heading content not meaningful User rated P3 5:27
CP, HP, AF What it hasn’t done is tell me where it is Content: Expected content not on page Coder identified
3.6.2 Categories of accessibility problems encountered by print-
disabled users
The main goal of the categorisation scheme was to provide a description of the nature of the main problems encountered by print-disabled users when using websites. The categorisation was divided in problems that were related to six levels: Content,
Delivery media, Web page structure, Website navigation, Information Architecture and Underlying System characteristics. Each category under each of the six levels contains sub-categories that describe the nature of the problem encountered by users. During
the categorisation, each user problem was assigned to a sub-category. During the construction of the Table 3.4 shows a list of the categories defined under each of the levels and a description of each category.
69 69
Table 3.4. Description of top-level categories of user problems Level Category Description
Content Content (Meaning) This category refers to the content in a web page, or the meaning the author wanted to convey
Delivery Media Text Issues related to the delivery of content as text
Images Issues related to the delivery of content as images (including pictures, graphs)
Audio, Video and Multimedia Issues related to the delivery of content as audio, video or
multimedia (including animations) Other media types (music,
mathematical notation, chemistry, etc)
Content delivered in media using other abstract notation
All media types Issues related to the delivery of content in any media type Web page
structure
Headings Issues related to the use of
headings and page structuring with headings
Links Issues related to individual link elements in a web page
Tables Content organised in table
structures with rows and columns Controls, forms and
functionality
Issues related to controls, form elements and functionality implemented in a web page Website
Navigation
Navigation Issues related to the overall navigation as the structure for changing between pages in a website
70 70
Level Category Description
Information Architecture
Information architecture Issues related to the way information is organised and structured in a website Underlying
System
Characteristics
System characteristics System issues related to the web application’s underlying system, such as processing speed or to the assistive technology
The structure in the levels of content and delivery media was based on a conceptual framework for accessibility defined by Power et al. (2009). This conceptual framework makes a distinction between the content or meaning that is conveyed by means of a webpage and the delivery media that are used to encapsulate this content, be it text, images, audio, video or multimedia.
At the content level, subcategories included problems where content was not found where expected by users (missing content), problems where users could not make sense of content, irrelevant content before task content, illogical organisation of content, too much information in pages, difficult language of content and meaning in content lost due to transformations (such as text simplification).
The conceptual framework proposed by Power et al. (2009) also discusses content
adaptation rules that must be applied to make content delivered in different media
available to disabled people. According to this framework, these transformations produce either an alternative that replaces the original resource, or provide an
enhancement to the original resource.
For example, content provided using an image as medium needs an alternative textual description of the content conveyed in the image in order to enable blind users to have access to this content; in this case, the alternative text replaces the image for these users. In the case of content conveyed using a video as medium, deaf users would need an enhancement in the form of subtitles or sign language interpretation to augment the original video.
Each category under the delivery media level had sub-categories describing problems related to alternatives, enhancements or presentation of content. The
following sub-categories were included for the categories text, images, audio, video and
71 71 No alternative: absence of alternative to a given resource, such as an
alternative text to an image for a blind user.
Inadequate alternative: alternative to a given resource is not adequate, such as an alternative text to an image for a blind person that does not describe the content of the image adequately.
No enhancement: absence of an enhancement to a given resource, such as the absence of an audio-description to a video for a blind user or the absence of captions for a deaf user.
Inadequate enhancement: enhancement to a given resource is not adequate, such as inaccurate captions in a video for a deaf user.
Default presentation not adequate: the presentation of a resource is not adequate, such as text being too small for a partially sighted user, text that is not read out properly by a screen-reader or bad colour contrast in a visual medium.
Inability to change presentation: presentation cannot be changed by specific settings in the user’s browser or assistive technology, such as inability to change the size of a video on the screen.
Problems related to difficulties with scanning for content of any media type were in a category named “All media types”, as it could be applicable to any media type.
The categorisation scheme also makes a distinction between problems that are related to an element contained within a single web page and problems that are related to the navigation between pages in a website. At the web page level were included the categories headings, links, tables and controls, forms and functionality.
The problems contained in the links category at web page level are to do with issues that are specific to a single link, such as having one link with unclear destination, link destination not present, poor link grouping, too many links or repeated links.
Problems related to the Headings category included issues such as having no or too many headings, headings that are not meaningful and illogical heading structure.
Problems related to the Tables category include problems where it was not possible to associate table cells to their headings, table structure being too complex, lack of headings and lack of alternatives to data in tables for users who find it difficult to handle tables.
72 72 Problems in the Controls, forms and functionality included issues with interface
elements that could not be reached using a keyboard, lack of indication of how to interact with functionality, unclear description of what controls/form elements do, expected functionality not present, functionality not working as expected, no or insufficient feedback for actions, among others.
The website navigation level contains problems related to the navigation structure of a website and the issues that occur when changing between web pages. Such
problems included navigation elements that do not help users find what they were seeking, no way to return to home page, navigation bar not salient, inconsistent navigation, destination not what anticipated by users, and impossibility to identify destination on arrival.
The Information Architecture level contains one category related to the organisation and structure of information in a website. This category includes problems such as complex organisation of content with too many steps to get to a web page.
The last level in the bottom of the categorisation scheme is Underlying system
characteristics. Categories at this level involve issues that are related to system
characteristics, such as issues with the web server where the application is being executed, broken links or system issues with the assistive technology.
3.6.3 Second phase of coding of user problems
The second phase of coding of the data was performed by one coder, the author of this thesis. In this phase, the remainder of the videos that were not coded in the initial phase were coded using the categories list built up during the initial phase. During this phase, if there was any new emerging problem that did not fit into the existing
categories, a meeting involving three coders (the author and his two supervisors) would be set up to discuss the creation of new categories.
When coding the user problems encountered in the video analyses, both problems explicitly mentioned by the users and problems observed by the coder would be recorded. In cases where the problem was observed by the coder, the coder would also attribute a severity rating to the problem, using the four-point scale (cosmetic- catastrophic).
Each problem raised by a participant or observed by the coder was flagged in the respective Morae recording of the session. For each problem coded from the analysis of the sessions, the following information was recorded:
73 73 - User problem: a description of the problem as experienced by the user.
- Technical problem: a description of the technical causes of the problem. - Problem category: the code assigned to the problem according to its category. - Severity rating: severity rating assigned to the problem
- Identified by user/coder: identification of whether the problem was mentioned by the user or identified by the coder. This is especially important for the interpretation of the severity ratings of the problems.
- Participant code: code of the participant that experienced the problem. - Website: website where the problem happened.
- Task: identification of the task that was being carried out when the problem occurred.
- Web page: identification of the archived web page where the problem was experienced.
- Time when problem occurred (generated by Morae)
The coding of user problems by the analysis of the videos was the most time- consuming activity in the research reported in this thesis. Excerpts of videos had to be watched several times in order to understand the nature of the problem that was being experienced by users and to identify possible causes.
The analysis was considerably harder for videos with blind participants using screen readers. It was crucial to understand what was conveyed by the speech synthesiser of those screen readers in order to know exactly what was happening when a blind user experienced a problem. Although it was asked that participants would use screen reader at a slower speed than that they would normally use, in many cases the speed was still too fast for the coder to follow. In these cases, it was necessary to set the presentation speed of the video to up to 0.7 of the normal speed. The time for coding one hour of video of blind participants could take at least three hours.