Automated accessibility evaluation

Automated evaluation involves the use of tools that automatically examine webpage code to identify (and in some cases, propose solutions to) accessibility issues. Such tools evaluate the conformance of individual webpages or, sometimes, entire websites to a specific set of guidelines or requirements, such as WCAG, Section 508 or other similar principles. Automated tools can check the validity of HTML markup and Cascading Style Sheets (CSS), detect the presence or absence of certain features (such as labels and headings), and compare detected values (such as the colour contrast ratio between foreground and background colours) to values found in guidelines and specifications. The outcome of these tests is then used to generate a report detailing where the issues lie in the webpage or website and what the web developer should do to address them. The format of the report differs from tool to tool. Some tools, such as AChecker8 _by

the Inclusive Design Research Centre, provide text-based reports that describe specific issues or guidelines that have been violated (see Figure 2.1). Other tools, such as WAVE9 _{by WebAIM, generate graphical annotations representing accessibility errors}

and warnings that are dynamically inserted into to the webpage code, allowing potential accessibility issues to be viewed in situ (see Figure 2.2). Still others, such as Testo Accesibilidad Web (TAW)10 _{by the Spanish Fundación CTIC (Centre for the}

Development of Information and Communication Technologies in Asturias) generate reports using Evaluation and Reporting Language (EARL) (Abou-Zahra, 2007), which is a working draft published by the W3C for describing accessibility test results in a machine-readable (XML/RDF) format. Using EARL facilitates the comparison and aggregation of accessibility test results.

8_{https://achecker.ca} 9_{http://wave.webaim.org} 10_{http://www.tawdis.net}

Figure 2.1: Example of report generated by the automated accessibility evaluation tool, AChecker by the Inclusive Design Research Centre. Reprinted under the GNU Lesser Public Licence, v2.1.

One of the earliest and most well-known automated evaluation tools was Bobby, which was originally developed by the Center for Applied Special Technology (CAST). Bobby was a free, public service for automatically determining conformance to a limited subset of WCAG 1.0 and Section 508, with successful websites being endorsed as “Bobby Approved”. The Watchfire Corporation acquired Bobby in July 2002 and later incorporated it into their existing tool, WebXACT, in May 2005. Following IBM’s acquisition of Watchfire in 2007, Bobby/WebXACT ceased to be available as a free service and was taken offline in February 200811_{. Other popular examples of automated}

evaluation tools include, Cynthia Says12 _{by Cryptzone, Tenon}13 _{by Karl Groves, and}

aXe14 _{by Deque Systems. The WAI maintains an extensive list of over 90 web}

accessibility evaluation tools, around two-thirds of which can be used to automatically determine the accessibility of websites (Eggert & Abou-Zahra, 2016).

Automated evaluation tools typically report some metric of web accessibility. For conformance-based evaluation, this is usually a dichotomous pass/fail measurement of whether or not a webpage conforms to a particular set of guidelines or requirements.

11_{https://www.webmasterworld.com/accessibility_usability/3579037.htm} 12_{http://www.cynthiasays.com}

13_{https://tenon.io}

For example, if a webpage satisfies each of the Section 508 requirements (that a tool can determine automatically), it will be deemed to have passed Section 508 (subject to certain manual checks). WCAG 1.0 and 2.0 both provide an additional degree of differentiation by categorising individual guidelines according to three discrete

conformance levels: Level A (lowest), AA, and AAA (highest). These reflect the impact of different guidelines upon users and provide an approximate measure or benchmark of a website’s accessibility.

Figure 2.2: Example of report generated by the automated accessibility evaluation tool, WAVE by WebAIM. Reprinted with permission.

Due to the potential lack of sensitivity and precision afforded by the dichotomous nature of conformance, whereby entire webpages can fail due to a single violation, some authors (e.g. Parmanto & Zang, 2005; Sullivan & Matson, 2000; Vigo & Brajnik, 2011) consider it too blunt a measure. Instead, they have proposed more precise metrics, such as the failure rate metric (Sullivan & Matson, 2000), the WAB (Web Accessibility Barriers) metric (Parmanto & Zang, 2005), and the WAQM (Web Accessibility Quantitative Metric) (Vigo, Arrue, Brajnik, Lomuscio, & Abascal, 2007). These are intended to more precisely indicate the degree of accessibility by taking multiple factors into account, such as the number of potential failure points in a website, the number of guideline violations, and the severity of accessibility barriers.

Automated evaluation tools expedite the otherwise time-consuming and often tedious task of determining the accessibility of websites (Power, Freire, & Petrie, 2010). The tests they conduct are easily repeatable, allowing accessibility conformance to be compared over time. The tools can also be more easily integrated into the development process, increasing the likelihood of accessibility issues being remediated. However, due to limitations in the type of issues that automated tools can programmatically

determine, they can only be used to evaluate websites against a limited subset of guidelines. For example, WCAG 2.0 Success criterion 1.1.1 states:

1.1.1 Non-text Content: All non-text content that is presented to the user has a

text alternative that serves the equivalent purpose, except for the situations listed below. (Level A)

To satisfy this criterion, all non-text content (such as images) must have an appropriate text alternative that that conveys the meaning, or purpose of the image. This is achieved using the alt attribute of the HTML img element (see Figure 2.3).

Figure 2.3: Image of the University of York logo. Reprinted with permission. The image is marked up using the following syntax:

<img src=”uoy_logo.png” alt=”University of York logo” />

While tools can automatically determine the presence or absence of alternative text of images (by identifying the alt attribute) they cannot establish whether the text is

sufficiently descriptive or even accurate. An evaluation of six state-of-the-art automated evaluation tools by Vigo, Brown and Conway (2013) estimates that they cover, at most, only 50% of the WCAG 2.0 success criteria and catch, at best, 38% of violations. Furthermore, due to apparent differences in how automated evaluation tools interpret accessibility guidelines, no two tools will necessarily return the same results (Harper & Chen, 2012). Also, despite their automation, such tools still require evaluators or web developers to interpret and verify what can often be overwhelmingly long and detailed results (Sloan, Gregor, Booth, & Gibson, 2002) in order to remediate any problems.