Approach to web accessibility assessment

The focus of the study is on assessing “meaningful accessibility” from the

perspective of end users with disabilities. Therefore, the method chosen for the study is built on the following pillars:

 To ensure traceability, accuracy and the possibility to repeat the tests, the

defined techniques in WCAG 2.0 were used.

 The tests are based on the concept of Universal Design, so that as many

target groups as possible are covered.

 Robustness, validity and reliability were ensured through a centralised group

of experts doing the web accessibility testing manually in all countries studied. To measure all the success criteria of WCAG 2.0 level AA is not feasible in this type of study. The study team chose indicators that make up a cluster-sampling of success criteria for WCAG 2.0. Tests that cover all four principles of WCAG 2.0 have been chosen, which means different perspectives on accessibility are

covered. The four principles of WCAG 2.0 covered are:

 Perceivable (“users must be able to perceive the information being

presented”)

 Operable (“users must be able to operate the interface”)

 Understandable (“users must be able to understand the content and

interface”)

 Robust (“user must be able to continue to access the content as technologies

and user agents evolve”)

Since the focus of the study is on “meaningful accessibility” rather than just numerical indicator scores, important aspects of accessibility according to 5 themes (navigation, documents, forms, construction and multimedia) have been selected, connecting them to relevant WCAG 2.0 techniques. The tests related to each the themes are described in detail in Annex 2.

 Navigation. The ability to navigate easily is a key factor for all users

 Documents. The accessibility of documents has received little attention in

study.

 Forms. Forms are the core of most e-services. Problems with forms can

result in being unable to complete the task.

 Basic technical construction. This is essential for assistive technology and

cross browser compatibility; problems here can result in the user being unable to use the interface.

 Multimedia. More and more information is being provided with multimedia

and this is in many ways positive, but at the same time it can mean also more accessibility problems for several groups of users.

To ensure a broad range of user groups were included in the analysis, the

research team tried to make sure that the experiences of at least 6 defined users groups were covered. These were people with:

 Motor difficulties

 Reading- and writing difficulties  Cognitive difficulties

 Visual difficulties  Blindness

 Hearing difficulties.

The tests were performed by several of Funka Nu’s experts and quality controlled through comparisons across services and tests respectively. See Annex 2 for details of tests conducted. One assessor was responsible for the test of each service. After that, another assessor compared the services test by test, and a third assessor compared country by country. In this way, each

assessment was conducted by two or three assessors. Since all manual tests are based on human judgement this is an important part of the methodology. The use of manual testing is essential in itself, since accessibility is not only about

technology; it has to do with human interaction and user experience.

A sample of pages to test was drawn from each site. This sampling included (if found):

 The start page (this does not always mean the start page of the whole

website. In some cases it's a subpage that constitutes the start page for a particular department or service).

 A subpage with instructions or similar.  A page with multimedia.

 Three documents in alternative formats (preferable PDF, but in a couple of

cases Word documents were selected)

 The actual service as far as it can be tested without identification of the user.

When this has not been possible, a page with the largest form that we could find relating to the service.

See Annex 3 for a list of all webpages and sites tested. The scoring scheme for all tests is as follows:

 A score of 2 (pass) means that the service complies with this specific

 A score of 1 (marginal fail) indicates the presence of some relatively minor

level of failure with the specific WCAG 2.0 success criteria that were tested. While technically a fail, these are instances where some level of the content or functionality is still accessible to the user.

 A score 0 (fail) means that the service do not comply with the specific

WCAG 2.0 success criteria that were tested.

It is important to note that many WCAG 2.0 success criteria are open to interpretation. This is partly deliberate from the W3C, because exact

implementations can vary over time, techniques can be modified, the support in browsers and assistive technology change, and cultural differences make specific solutions stronger in some countries than others. Standardise globally but

implement locally is the suggested approach. This does however leave

room for interpretations of exactly where the limit is for meeting a specific requirement, just like any measurement of quality.