Evaluation of the system should not only come at the end of a system’s creation. Evaluation should be carried out to help guide and correct the interface’s design. This section discusses the wide range of methods for evaluating a user’s experience of a system. The important factors in evaluation have been divided into three sections:
1. For user evaluation to be performed, a sample of users must be chosen. It is important that the users are representative of the intended target group of the system.
2. The users should have provided informed consent and be aware of their rights throughout. A user evaluation session should be a positive experience for all involved. The user has a right to know what information is being collected and how it will be used. Privacy is an important issue and names should not be recorded together with viewpoints expressed. At any point during the evaluation the user can terminate the session without having to give a reason and without fear of recrimination.
3. Another important consideration when evaluating software is which version of the software to present to the user. It is important to evaluate the software early in the development lifecycle. Often if the developer is the only user evaluating the system, certain tasks become too easy to achieve with their background knowledge. An outside user is required to review the system. If the evaluation is left until late in the software lifecycle then it will be more difficult to improve the software. Further, the later in the software lifecycle a development flaw is left,
133
the more expensive in time and money it will cost to fix [282]. Early evaluation by outside users is an important technique for recognising difficult tasks and problem points in an interface. This early evaluation allows the lifecycle to be guided towards the end users’ needs. The evaluation session can be designed in several ways. The least rigid is an open-ended session where a user is not provided with any information or goals to achieve and is allowed to experiment with the system. A more structured approach is to provide the user with a set of tasks to achieve. Selecting appropriate tasks for a user to achieve involves the right amount of required background knowledge and a task that is not specially adapted for the designed system. The tasks should be ordered from simple to hard. This allows the user the opportunity to master the system before attempting the more difficult tasks.
Evaluation can be performed through the observation of a user interacting with the system. The person evaluating the system records the tasks attempted by the user and the methods used to achieve such tasks. In [283] a discussion on which data to collect is provided. The data gathered by observation is differentiated into process and bottom data. The process data is the steps the users are taking to solve one of the tasks set. This process data is a step-by-step report on the activity of the user. The bottom data is the measurement of how many errors were made, which tasks were achieved and the time taken to do so. It is important to collect both types of data, and in particular, to focus on the process data in early testing. If the process data is ignored and only bottom data is collected, there is no way to discern where users went wrong.
The thinking-aloud method [284] of evaluation involves the user of the system verbally detailing each step as they attempt to complete a task. The user should be fully aware that it is the system and not their mental processes that is being evaluated. The role of the observer can be active or passive. The observer can prompt the user with questions such as “tell me what you are thinking” or “keep talking”. These are open-ended question and allow the user to continue giving feedback. The observer should not attempt to guide the user to a feature they may have missed on the interface.
A passive evaluation can be performed through the examination of records maintained by the system that the user has interacted with. Recordings of the actions on the system can be replayed and analysed for achievements and mastery of the system. Frequency and duration of use can be calculated and compared between multiple users.
A more active form of evaluation involves using the System Usability Score (SUS) [257]. SUS gives a number between 1 and 100 that is derived from a short questionnaire. It can be used to provide a score that is comparable to other users and system versions. The questionnaire alternates between positive and negative questions concerning the use of the system and how strongly or negatively the user agrees with the statement. This system was devised to enable users, often tired after a lengthy evaluation session, to provide feedback in a compact way.
134
A range of debriefing interview techniques can be used to evaluate a user’s experience of a system. There is anecdotal evidence that suggest that users will remember the solution to a problem rather than the problem itself. This can lead to a user struggling with a difficult system and not reporting this to the interviewer. This is an example of why interview results alone should not be relied upon. Interviews are a useful technique for getting detailed feedback and fit well as a debriefing technique. The most rigid approach is a structured interview. In a structured interview, there is a fixed order of questions given to the user. The user may respond either from a given set of responses or a more open- ended approach where opinions can be articulated. Due to the ordering of questions being maintained, the possibility of influencing answers through juxtaposition of questions is ruled out.
A semi-structured interview provides a more flexible approach to the interview process [285]. The interviewer may have several aspects of the software they wish to gain feedback on but there are no fixed set of questions. The lack of a fixed set of questions allows a more natural discussion to flow and can be steered towards areas of particular interest of the interviewer. An unstructured interview allows the adaptation of a set of questions based on the user’s capabilities. The user is also not restricted to a fixed set of responses, unlike in a structured interview [286].