After formulating what should be evaluated a discussion is needed about how this could be done. Therefore a suitable evaluation approach needs to be selected and adapted to the needs of this research project.
As has been described in the preceding section an evaluation should be carried out to see if and how the concept simplifies authors’ work in creating adaptive learning content. This research project focuses on users’ perceptions of the developed concept, and as such it has been seen as testing the concept’s usability.
Van Velsen et al. have noted that in contrast to user-centred design practices, questionnaires are extremely popular for the evaluation of adaptive and adaptable systems and even for usability questions. They remark that questionnaires seem to be the “quick and dirty way”. However, “Issues that call for an extensive review of the system, like usability, require other, more exhaustive methods” [122, p 274]. Regarding more exhaustive methods they suggest qualitative research methods that are well known in social sciences, like interviews or focus groups.
Qualitative research interviews have been adopted from social science and are used to get individuals’ points of view on an investigated subject. Because any influence may disturb individuals’ perceptions and maybe even more their ability to realise and talk about them, there needs to be as little as possible that might disturb the interviewees. Social scientists need to be extremely careful about this because often they investigate highly sensitive subjects, welfare cases and fringe groups like HIV infected persons, vagrants, immigrants, delinquents, etc. Hence,
6.2. Method: Expert Interviews these interviews have and need to sustain a very intimate character and therefore need to be in confidence.
According to Lamnek, interviews are usually individual surveys of a mediative character [74, p 60]. Qualitative interviews investigate the interviewees’ interpretive and action patterns and develop them in the course of a conversation. Interviewees are not only seen as a data-source, but as individuals who determine the conversation. They define the point of view and perception and should not be influenced by the interviewer. Thus the researcher needs to adapt to an interviewee’s conversation style by being open and flexible and thus amenable to unexpected information while being able to react to an interviewee’s demands [74, p 64].
Even though usability researchers usually do not work in and with highly sensitive subjects, they still need to consider that reflecting on usability is a fragile process that can be easily biased and therefore may lead to invalid results. As in the social sciences, anything may influence an interviewee such as design, content, keyboard and screen, but also factors like environment, mood, lunch, light, temperature, etc. The questions that should be asked during the interview need special attention because not only how and what is asked may bias the outcome but also the very fact that a question is asked. In more technical terms: Measurement itself influences the item to be measured and therefore the measurement result.
Qualitative research methods are relatively new in information science, including research in usability as well as adaptive and adaptable systems. In 2000, Hartson et al., for example, did not even mention methods like interviews. Moreover, because of “Not being statistically significant”, they criticised the fact that qualitative data cannot “contribute (directly) to the science of usability, but are valuable usability engineering measures within a development project” [59, p 149]. Additionally, they noted the general problem that “researchers are far from agreement on a standard means for evaluating and comparing UEMs” (usability evaluation methods) [59, p 146].
Earlier in 1988 Kaplan and Duchon propagated combining information system researchers’ traditional evaluation methods with social research evaluation methods [69]. They stated that information systems researchers’ exclusive reliance on the methodology of formulating hypotheses that are tested through controlled experiment or statistical analysis have been criticised in the social sciences: “. . . the simplification and abstraction needed for good experimental design can remove enough features from the subject of study that only obvious results are possible” [69, p 572]. Combining methods introduces testability and context into research and increases the robustness of results through triangulation [69, p 575]. Because social research often demands the involvement of many uncontrolled and unidentified variables, “Qualitative strategies emphasize an interpretive approach that uses data to both pose and resolve research questions” [69, p 573]. While interpreting the collected data, social researchers usually attempt to understand an other’s way of construing, conceptualising, and understanding events, concepts, and categories [69, p 572]. Although quantitative methods provide better results in statistical terms, they see the value of qualitative social research methods in giving “richer explanations of how and why processes and outcomes occur” [69, p 573].
The strength of qualitative methods in interpreting data through understanding other individuals’ ways of seeing, feeling, and doing, makes them attractive for usability testing. Both research disciplines investigate human perceptions of items and conditions, something necessary for this research project that concentrates on user experience of the developed adaptive learning concept. Weßel states that qualitative methods target understanding and explaining [127, p 928]. Using these methods is user centred because a computer scientist’s point of view changes from information technology to the user’s view [127, p 934]. Qualitative methods focus on learning about backgrounds and reasons and are able to gather information that cannot be captured in the same form by quantitative methods. Regarding IT-projects, interviews are used to investigate concrete possibilities of software development
6.2. Method: Expert Interviews and implementation [127, p 929] and to understand relationships. In particular, semi-structured interviews facilitate the obtaining of new insights [127, p 927].
As mentioned above, van Velsen et al. confirm this opinion: “Interviews may also be used to assess the usability issues that are typical for personalized systems since it allows detailed feedback. As a result, not only can problem areas be identified, but also perceived causes and solutions.” [122, p 270] They state that interviews can be used to analyse a system’s usability and to get information about the intention behind its usage [122, p 275]. They go one step further and criticise methods like closed item questionnaires as being limited to accessing only known variables. As a result they prefer more exploratory methods such as interviews, and may use the gathered data later as input, for example to create questionnaires: “Because little is known about the interaction between user and personalized systems, it may be best to use exploratory methods to assess important variables. In a later phase, these variables can serve as input for confirming methods (like questionnaires).” [122, p 274]
Van Velsen et al. divide the development process into four phases, see figure 54. For testing the concept of this research with the help of an early stage prototype, regarded as low-fidelity prototype, the second phase seems to be most suitable for the goal of this evaluation: “The second phase in the development process deals with the ideas behind the system and the techniques that are supposed to embody these ideas. A low-fidelity prototype can visualize them, and from this moment on, UCE (User-Centred Evaluation) becomes an option.” [122, p 274] The evaluation variables of this phase, appreciation, future system adoption, and perceived usefulness, address comparable research questions to those formulated in the preceding section of this thesis. Not addressed until now, the variable trust and privacy issues becomes a very important variable through the interviews and will be discussed subsequently.
The low-fidelity prototype Coherence can be additionally used to locate critical issues. It enables evaluators to look at and discuss the concept behind the system
Supporting design
decisions Detecting problems Verifying quality • Questionnaires • Interviews • Think-aloud protocols • Observations • Questionnaires • Interviews • Think-aloud prtcls. • Observations • Interviews • Focus groups • Questionnaires • Interviews • Focus groups No system Low-fidelity prototype High-fidelity
prototype Full system Methods
System
development phase
Variables to address • User characteristics • User needs
• Appreciation • Future system
adoption
• Perceived usefulness • Trust and privacy
issues
• Appreciation • Appropriateness of
adaptation • Comprehensibility • Trust and privacy
issues • Usability • User behaviour • User performance • Appreciation • Appropriateness of adaptation • Trust and privacy
issues • User experience • User satisfaction • Usability • User performance
Figure 54: The iterative design process and the role of user-centred evaluation for personalised systems [122, p 273]
Permission to reproduce this figure has been granted by Cambridge University Press. and the content. But it needs to be considered that discussing a concept is abstract and as such will generate abstract outcomes – discussing a concept’s usability is even more abstract. “Therefore, in low-fidelity prototyping, one needs methods that are flexible and can cover aspects in depth, like interviews and focus groups.” [122, p 266]
The user’s view, and more precisely the expected experience of authors using an environment that needs to be developed and will be based on the concept of this research project, should be investigated in this evaluation. Thus not only direct answers to questions are of interest but also background information about the answer and, additionally, gathered aspects that have not been foreseen.
It was thus decided to use interviews, partly in conjunction with the low-fidelity prototype. The interviews are analysed using quantitative and qualitative methods that are discussed later in section 6.8 Analysis Process on page 200.