Intelligent and Adaptive User Interfaces

Human-computer Interaction encompasses the technologies, practises and understand- ings that allow humans to work with machines, but also machines to work with humans (human-centered interaction [149] or Intelligent Human-Computer Interaction [152]). Jameson [106] describes a user-adaptive system (equivalent to “adaptive interfaces”, “per- sonalisation” or other terms used in literature) as a system that “makes use of some type of information about the current individual user [...], can be defined as an interac- tive system that adapts its behavior to individual users on the basis of processes of user model acquisition and application that involve some form of learning, inference, or deci- sion making”. Thus, the adaptation may result in different outcomes: sometimes clearly visible (such as the content adaptation when the content depends on the user’s activity, choices and habits i.e. commercial website’s item suggestions) and sometimes not visible (such as the system adaptation where the processes are adapted to the user to guarantee an optimum result commonly expected by the system i.e. phone typing auto-correction). Benyon, in [14], gave the guidelines to analyse the usability and to design systems ap- propriately to “build intelligence into the system” and reports the four adaption levels established by Browne et al. [22]: simple (“use a ‘hard-wired’ stimulus-response mecha- nism”, self-regulating (“monitor the effects of the adaptation on the subsequent interaction and evaluate this through trial and error ”), self-mediating (“monitor the effects on a model of the interaction”) and self-modifying (“capable of changing [their] representations”, the models can be adapted). The adaptation of a system may take different aspects: taking over parts of routine tasks, adapting the interface, helping with system use, mediating interaction with the real world and controlling a dialogue according to Jameson [106]. In the following background review, we refer to the major research works focusing on the particular functionality of adaptive interfaces.

Early personalisation interfaces have been experimented by Chesnais et al. [36] and Höök [96]. Chesnais et al. introduced Fishwrap, a personalised electronic newspaper targeting freshmen at MIT. Fishwrap delivered content based on the user’s personal information (place of origin, affiliation with MIT and interests) and interaction (position of the consulted articles within the page), and had been well received by the readers despite concerns dealing with privacy. Höök presented PUSH, an adaptive hypermedia system, which content (information the users wanted to retrieved) was tailored based on the user’s activity (clicking, StretchText8 actions). The evaluation of PUSH revealed

8

2. Background

that users required fewer actions compared with the non-adaptive equivalent system and it was prefered.

Bohnenberger et al. [19] investigated the usability of a location-aware shopping assistant application on a PDA. Their application indicated the shopping areas of interest (adap- tation of the content) for a user based on her location in space, but also on the interest showed in specific products and the purchases during previous shopping sessions. They showed the application performed better (faster shopping and preference) than a paper map.

Content adaptation is often deployed on commercial Internet websites to provide the buyer a fast access to their preferred or likely to be preferred products, ease the purchase process and run marketing strategies. Alpert et al. [6] conducted an evaluation of a user- adaptive online commercial website (prototype) from the users point of view, and found out that users did not always respond in favour of system intrusion and personalisation, and that they preferred “[having] full and explicit control of data and interaction” as well as clearly understand the way personalisation was in place on the system.

Content adaptation is not solely based on the user: Cheverst et al. [38] proposed GUIDE, a context-aware tourist guide. Their system relied on both personal (such as the user’s interests) and environmental (such as opening time of the attraction) contexts. GUIDE ’s users showed a high acceptance of the adaptive system, but the authors realised that it should allow a choice of functionality level as some users found GUIDE somehow confusing. Likewise, Kortuem et al. [117] proposed an adaptive wearable system that adapted itself based on the local environment, by communicating with intelligent objects nearby via infrared beams.

With simple approach of adaptation, McGrenere et al. [129] tested the acceptance and effectiveness of the personalisation of a complex software: users were able to chose the elements from the Full Interface they wanted to keep, and therefore two interfaces were used (Personal and Full). They reported that users appreciated the possibility to person- alise the software interface, but that the system could offer a smarter way of setting the personalisation by assisting the user to create its profile (“mixed-initiative interface”). Gajos et al. [73] evaluated different adaptive graphical interfaces based on existing work (Split Interface, Moving Interface and Visual Popout Interface). Their evaluation showed that the acceptance and the performance of an adaptive interface by the users seems to

2. Background

depend on the accessibility: more accesses brought dissatisfaction and low performance. They also indicated that if the frequency of adaptation is too high during a session, the users may feel unsatisfied.

Adaptation can also be retrieved from sensors that reveal the user’s cognitive state or emotions. Tan and Nijholt [184] investigated the role of Brain-Computer Interaction in Human-Computer Interaction, and among other applications, they encompassed it to be used in adaptive systems, for example to evaluate when to interrupt the user. Iqbal and Bailey [99] used eye tracking to estimate the tasks performed by the users and adapted the system disruption levels accordingly. Still employing eye tracking, Iqbal et al. studied the mental workload of different tasks through pupil response [100] and proposed to use their finding in attention manager applications. Duric et al. [58] suggested other examples of biological indicators of the user’s cognitive state: “facial expressions, upper-body posture, arm movements, and keystroke force” that can be used to build an intelligent adaptive system. Rothrock et al. complemented these examples with “a wide range of possible inputs about the user’s physiological state (e.g. EEG, heart rate variability)”, and also mentioned other user’s traits that can be used for adaptive systems that yet need to be assessed before the interaction (i.e. user knowledge, user personality, cogntive style). In this thesis, we endeavour to find out if the correlation between gaze and hand can be used to understand the user’s cognitive state (hesitation in particular in our case) and if so, propose to use this input to built an adaptive interface system that can respond to the user’s hesitation.

3

Study Setup

This chapter presents the methodology for setting up our studies. The first part deals with eye tracking and relates to both studies covered by Chapters 4, 5 and 6. The second part focuses on the on-screen hand gestures recognition that only concerns the work of Chapter 4 (first data collection).

3.1

Eye Tracker

The procedures found in eye tracking studies for installing and preparing the devices are often the same, positioning the eye tracker, calibrating it and using it consist in the basic steps any eye tracking study must start with. This section describes the steps and methods we followed in our work for the data collection we achieved with two eye trackers: Tobii X2-60 for the first data collection (relating to Chapter 4) and Tobii EyeX for the second data collection (relating to Chapters 5 and 6). Whenever the case happens, we will indicate in this section if we had not followed the standard procedure and why. The type of eye tracker we mention in this section are remote infrared eye trackers, such as the two eye trackers we used in our work.