Summary: Previous sections have described how fully automatic testing can be made part of earlier development phases. For usability tests with real test users, tool support cannot eliminate the need for test subjects to participate, but it can reduce the amount of work required by the developer to prepare, conduct and analyse a test. Once a web application has been implemented, different kinds of tests are necessary to ensure that it works as expected. This includes checking the criteria of the service level agreement, a test of the final hardware and software, determining whether the assigned hardware, software and human resources lead to adequate performance, and planning for future changes, e.g. by set- ting aside a budget [Dumke03WE, chapter 7]. With regard to usability, established procedures include analysis of the finished web application by experts (cognitive walkthrough, heuristic evaluation) and tests with test users or the final users, in a lab setting or in the field (experi- mental evaluation, observation with a think-aloud protocol, interviews) [Dix04HCI, chapter 9], [Constantine99Software, chapter 16]. Especially with web applications, a beta phase is a popular means to achieve wider testing of the application. It allows the end users to access the application, possibly even already in a productive environment.
Some aspects of testing have already been covered in previous sections: By building expert knowledge about usability guidelines into a tool, some tests can be performed continuously dur- ing design and implementation of an application. If any problematic changes are discovered, the developer can be warned about them the moment he makes the change. However, this automated usability evaluation alone is by no means enough to ensure that the application has good usability. It needs to be complemented by the opinions of experts and the results from user tests.
It is difficult to come up with tool support for this phase of development: Humans are in- volved in the testing procedure in most cases, either as experts or end users, and a lot of human
judgment is required to prepare the web application’s user interface, design a test setup, perform the user test, and finally to identify and categorize the problems that are found. However, a closer examination of the procedures and tasks that are actually carried out reveals that a lot of time is spent e.g. organizing rooms or test persons, setting up the testing environment and any equip- ment used to record user actions, and to analyse the data that is recorded during the test. Tools can help with these simpler aspects of the task. One possible approach is described below, it has the potential to make setup easier while at the same time producing detailed logs.
4.4.1 Web-Based Interaction Logging For User Tests
Conducting user tests in the established way is a fairly work-intensive task: Test users must be recruited and invited, a lab must be booked, and the test environment needs to be set up. After the test, the analysis e.g. of recorded video footage can be time-consuming. This section proposes an alternative, “lightweight” usability test which is supported by a special tool. It aims at reducing the cost of user testing (both in terms of time and monetary cost). Compared to the classical user test in a laboratory, less data is typically collected during each user test (e.g. no video recordings), but the data that is collected is more suitable for automatic analysis, and the more automated nature means that the number of test users can be increased considerably without a lot of additional work.
The tool concept is as follows: The test user’s actions in the web browser are recorded in detail. This includes all interaction made via the mouse or keyboard, and it is suitable for all aspects of web-based interaction, including advanced AJAX-based concepts like drag & drop. For each type of interaction, the exact web page element (e.g. a button) is identified automatically and written to a log file. After the test, the logged information can be used to analyse e.g. the time taken to complete a task. The implementation of the approach (presented in chapter7) has a number of interesting properties:
• No installation of software is required on the client side. Because an HTTP proxy is used to
intercept the page requests and (through an AJAX-based technique) to record user actions like mouse movement, the only change required on the client’s side is a reconfiguration of the browser’s proxy setting.
• As a result of the proxy architecture, no changes are necessary to the server either – the
web application that needs to be tested does not have to be changed for the test. Figure4.8 illustrates how the proxy is inserted between the client and server.
• Non-invasive operation: Use of the logging system does not alter the user experience. • The solution works with with normal web pages and non-AJAX web applications, but is
particularly useful for AJAX applications, which are not supported well by comparable alternative logging solutions.
• As the solution relies on established web programming technologies, it is compatible with
log file Tracking data Request modified Response Response Request Standard
Client TrackingProxy StandardServer
Figure 4.8: Architecture of a system for web-based logging of user interaction with web pages, based on a filtering HTTP proxy. [Atterer06UsaProxy] and chapter7
• Detailed logging: Apart from the URLs that are requested, the interaction within a page can
be recorded. For each type of interaction (mouse movement, clicks, key presses, scrolling, drop-down menu selection, selecting text in input fields), not only the directly available information is logged (pixel coordinates of the mouse pointer, key that was pressed etc.), but also the HTML element that the user interacted with, as identified by its ID in the document and its position in the DOM (document object model) tree.
The above ideas have been implemented in the form of “UsaProxy”, the working prototype of an HTTP proxy for interaction logging. Its description in chapter7also includes a discussion of related work, problems and challenges of the implementation, as well as an evaluation.
The tool concept is suitable as an interaction logging facility for a lab-based usability test. When used for this purpose, it can collect data about the user actions which is more accurate than e.g. a screen capture program or a tool which records the mouse coordinates, because the exact links, buttons, input fields etc. on HTML pages are directly identified: Rather than a log entry like“The mouse was clicked at screen coordinates 243/943”, there is a log entry
“The mouse was clicked at coordinates 203/2034 on the page, on the link with
id="moreinfo", labelled ‘More information’, located in the DOM tree at position
abadaacbb”.
(See chapter 7 for the exact format and meaning of log entries.) Automated analysis of the interaction log becomes much easier with this type of entry. Typically, only a few lines of code written in a scripting language like Perl or Python are enough to interpret the timestamps and events in the log file, and to extract information such as the time taken for completion of a task.
However, the concept is also capable of supporting other types of usability tests. Above all, the lightweight, flexible architecture allows user tests to take place remotely – the test persons do not need to be in the same place as the usability expert who conducts the user test, or the system on which UsaProxy is installed. Having been guided through the simple process of reconfiguring their browser’s proxy setting, they can participate in the test immediately. If the server on which
the web application runs is under the control of the usability expert, remote usability testing is even possible without requiring the users to reconfigure their browser. A further possible advantage of remote tests with UsaProxy is that the users can access the web application in their usual surroundings, with their existing computer, operating system and browser.
On one hand, performing a user test remotely has the disadvantage that the usability expert cannot observe the user’s personal reaction to the user interface directly. To some extent, this can be counteracted by asking the test subjects to establish a voice connection to the expert, either by voice over IP or using a regular telephone line, and to have them “think aloud” about their experience.
On the other hand, remote usability tests can be used to the tester’s advantage: Especially with web applications, the prospective audience is often spread geographically, possibly even all over the world. If the test users had to travel to a specific lab for all user tests, these tests would become far too expensive. Even if the audience is not spread geographically, some experts recommend not to recruit test subjects for a company in the same city that is dominated by the company’s industry [NielsenAlertbox,30 April 2007]. Furthermore, if no voice feedback is needed, or if the think-aloud protocol is recorded rather than immediately listened to by the expert, test sessions can be fully automated, and can take place in parallel for many users. At the same time, the results of remote testing are comparable to laboratory tests in terms of the number, type and severity of discovered problems [Brush04Comparison].
Taking the idea of remote usability testing one step further, the tool concept also supports recruiting test subjects over the Internet: An existing, similar application (for example, the pre- decessor of the new web application) can inform users about the testing process on its pages and ask them to participate in a test, the usability expert can post an invitation on support forums related to the topic, a database of existing customers can be used to contact interested users, and even online advertising is imaginable as a means to find test subjects.
Depending on the policies of the company which develops the new web application, it may be a problem that the test users cannot be expected to remain silent about what they saw. Requiring them to sign an NDA (non-disclosure agreement) would quickly make the process much more cumbersome, and it would significantly reduce the number of volunteers. However, especially with rich Internet applications, it is common even for large companies to conduct a public beta phase. If there are concerns about negative reactions to the initial user interface, one possibility is to perform a more closed usability test first and only to start public testing with a later iteration of the UI.
In conclusion, the tool concept facilitates very accurate logging of user actions such as mouse clicks, because the exact clicked link or button is identified. Due to its flexibility, it allows remote user testing, or even fully automated user testing in the sense that no manual action by the usabil- ity expert is necessary for each test subject. Because the log is less accurate than e.g. what can be achieved with recording video in a lab, the tool concept allows a tradeoff between detailed testing with few users, and slightly less detailed testing with many users. It is advisable to combine these two techniques: Usability problems can be identified by looking at a few users’ reactions in detail (e.g. even including gaze tracking), but some problems will only become apparent once a greater
number of people use an application in a wider variety of contexts. Chapter7contains details on the implementation, a comparison with related work, and an evaluation of the concept.