Implicit Gaze Interaction

A large part of gaze HCI research is concerned with the implicit use of the gaze modality during human-computer interactions, where input from the eyes are in the background of a user's attention. It is motivated by a dierent reason. The previous works considered gaze as a potential replacement to manual input in the pointing task. Instead, the following works investigate methods where the user's manual capabilities remain the main modality, while eye movements are used to improve the manual task.

Initial work was conducted in 1990, where Starker and Bolt proposed their Gaze-responsive Self-disclosing Display [SB90]. A system is presented that operates interpretively, i.e. by aggregating xations of the user to determine an interest level on the displayed ob- jects. Objects of higher interest are shown in more detail, giving the user an impression of subtle gaze highlightings.

MAGIC pointing

The rst work that argued for implicit use of gaze is Zhai et al.'s seminal paper in 1999, where they proposed the Manual And Gaze Input Cascaded (MAGIC) pointing approach [ZMI99]. This approach also investigated gaze and manual input combination as in prior work, but with the main focus on retaining manual input. They argue that it is unnatural to overload a perceptual channel such as vision with a motor control task. The pointing task should remain a manual task to the user, and the role of gaze is rather in the background of the user's attention. The MAGIC technique is an instance of this principle. Users perform the majority of the pointing task with manual control, but large parts of the pointing are eliminated by warping the cursor to the user's gaze area (Figure 2.1). In a pilot study, the technique indicated reduced eort as compared to manual pointing, while having greater accuracy than gaze-only pointing. This showed that the inaccuracy issue found by Ware and Mikaelian can be approached by hybrid gaze+manual interaction techniques, while preserving the speed advantages of gaze as indicated in many other works.

Figure 2.1: MAGIC pointing: the cursor warps to the gaze area (blue circle), followed by ne manual positioning. Image from Zhai et al. [ZMI99].

Note that the idea of using a modality in the background of a user's attention has been generalised by Vertegaal in his work on Attentive User Interfaces (AUI) [V+_03].

As devices will `bombard' users with requests for attention, it is necessary that user interfaces better adapt to the limited user attention. The user's eyes are an ideal indicator of attention to help in this matter. Zhai's MAGIC work is taken as the prime example, as the user's visual attention is combined with manual actions to improve the interaction with the UI.

2. Related Work 2.1. Gaze Interaction research community, as researchers extended and studied the principle idea in many more computing contexts. Drewes and Schmidt extended MAGIC pointing to a touch-sensitive mouse in their MAGIC touch project [DS09]. Their work was motivated by a recurring issue of MAGIC: when should the cursor move to the user's gaze position? Most of the prior work warped the cursor when users moved the mouse, thus requiring small manual eort for each movement. With MAGIC touch, the cursor is warped when the user touches the left mouse button, eliminating the need to initially move the mouse. They compared this technique to mouse input, and found that performance was similar between the techniques, but perceived as faster and more convenient.

Fares et al. explored further variations of the MAGIC technique with a mouse. Instead of warping the mouse cursor, MAGIC-SENSE [FDK12] changes the cursor sensitivity depending on the user's gaze position. If the cursor is far away from the user's gaze, it moves rapidly. If the cursor is close to the user's gaze, sensitivity is reduced for precise pointing. A pilot study showed low error rates and task completion times similar to the mouse. In a follow up paper, they conducted a user study comparing the default MAGIC technique to a mouse, as the original paper used a dierent pointing device [FFK13]. They also provided a slight design improvement by animating the cursor movement to the gaze to better provide visual feedback. The results of a standard pointing user study showed that MAGIC outperformed the mouse by 8%, and that the amount of hand movement is reduced by half, conrming the initial results of the original paper.

For a further discussion of MAGIC to concepts explored in this thesis, we refer to section Gaze-added UI

A notable work that aligns with the direction of implicit gaze interaction is Salvucci and Anderson gaze-added interfaces in 2000 [SA00]. The idea is to keep the conventional manual input as it is, but the user can issue complementary gaze functionality. This gives the benet of giving users more exibility in choosing when and how to employ gaze input. The idea of gaze-added UIs was simple; a conventional WIMP operating system was given that users normally control via the usual mouse and keyboard inputs. However, one button on the keyboard (Control key) was saved for gaze interactions. With this button, users could initiate typical gaze + button interactions as e.g. investigated in Jacob [Jac90] or Bolt's [Bol81] interfaces. Users point with their gaze at a target, push the button to select it, or hold the button to perform drag & drop actions.

Gaze as Contextual Information

Researchers have investigated implicit gaze with manual input beyond the MAGIC point- ing concept. For example, the LookPoint system uses eye gaze to switch input devices between multiple screens [DHVE06]. The system redirects the input of mouse and key- board, with regards to which screen the user is visually attending. This provided practical benets, e.g. the user does not need to reconnect input devices from one to another com- puter. A study showed that this method is faster than traditional methods (mouse, keys, multiple keyboards), and is preferred by users.

The Rake Cursor interaction technique allows users to facilitate multiple cursors, and gaze selects the desired cursor [BO09]. Multiple cursors in one UI can reduce pointing

time [KI08], but need a method to select a cursor. With the Rake Cursor, the user selects the desired cursor by simply looking at it, and then controls it by standard mouse control. Their experiment showed that it is faster than default mouse or MAGIC pointing. As an approach for leveraging implicit gaze input, Santella et al. investigated image cropping with eye gaze [SAD+_{06], as an instance of Vertegaal's implicit AUI [V}+_03].

Their system uses xation data to identify important image regions, and then crops the images accordingly. A study showed that users prefer this technique over uncropped images, or automatic cropping.

During the course of this thesis, two research papers also involved the use of gaze in consideration with direct and indirect manual input. Voelker et al. investigated input redirection on interactive workspaces, systems that provide users a combined horizontal and vertical touchscreen [VMSB15]. Direct touch input is active when users look at the horizontal surface, enabling default input on the reachable screen. But users can also issue indirect touch input from the same horizontal surfaces, when their gaze lies on the vertical screen. In their study, they compared this method to using direct-touch only, and found that the gaze approach leads to higher performance.

Serim and Jacucci use eye-tracking to support varying degrees of visual feedback during manual input [SJ16]. The system distinguishes two cases: input with visual guidance (user looks where they touch), or without visual guidance (user looks elsewhere). In the latter case, the system provides feedback of the non-visual manual input, to the area the user is looking at. Our direct and indirect input exploration goes in a similar direction, but applied to a dierent usage context. Rather than redirecting the visual feedback to the user, our focus is on explicit interaction with either direct or indirect input depending on the user's gaze area.