Summary and Conclusion

The comparison of the cognitive processes in the three different HEI-techniques—feedback- based direct-touch, mid-air full-arm pointing gestures toward real-world proxy objects, and mid- air full-arm pointing gestures toward body-relative proxy zones—revealed several similarities as well as some crucial differences.

Feedback Channels

The combination of selection mechanism and selection proxy in an interaction technique usually determines its main feedback channel. In general, feedback plays an important role in improving the accuracy of an interaction technique. Touch interfaces and Room Pointing are similar in that

84

they both use visible objects as selection proxies: on-screen icons and real-world objects. This means that both techniques rely on vision as the main feedback channel. For both techniques, vision is necessary in order to find the proxy icon and accurately tap on or point toward it. Despite this similarity, I expect that people will show higher accuracy in touch interface due to a difference in selection mechanism: touch interfaces start with comparably accurate proximal pointing and transition to touch, which adds haptic feedback, whereas Room Pointing only uses comparably inaccurate distal pointing. Room Pointing, however, offers people the possibility to blindly point toward the real-world proxy object, thus rely purely on proprioception. This is possible because people have an accurate spatial model of objects in familiar environments and a good understanding about proprioceptive feedback when performing deictic pointing gestures. In

Virtual Shelves, in contrast, selection proxies are virtual and thus do not generate visual feedback

directly. Instead, people have to rely on feedback generated from the selection mechanism—the mid-air full-arm pointing gesture—, which is mostly proprioceptive and, to a lesser degree, visual. As discussed above, I expect low selection accuracy in Virtual Shelves due to the initial inaccuracy of proprioceptive feedback. Despite using the same selection mechanism, I expect people’s selection accuracy in Virtual Shelves to be lower than in Room Pointing because the different selection proxies in both techniques result in different main feedback channels. Memory Systems

The combination of selection mechanism and selection proxy also determines which memory systems are most relevant for an interaction technique. In general, the involved memory systems and the pre-existing knowledge plays an important role in the initial learnability of an interaction technique. As with feedback, touch interfaces and Room Pointing are similar in that they both mostly rely on semantic memory, i.e., spatial, relational, and visual memory. With touch interfaces, people require a single cue–response pair for translating their intention to the visual and spatial information about the proxy icon. With Room Pointing, people need two pairs: one for translating the intention to the proxy object and another one for retrieving the object’s visual and spatial information from memory. The first step in Room Pointing is therefore additional compared to touch interfaces. Weather this additional indirection in Room Pointing will lead to decreased selection accuracy and increased selection time will most likely depend on the amount of meaning between the intention and the real-world proxy-object. In Virtual Shelves, the

85

learning stage (cognitive, associative, and autonomous). During the cognitive stage, both semantic and procedural memory are involved in creating a pointing gesture toward a target zone. This translation between memory systems adds an element of inaccuracy to the execution of the pointing gesture, which I believe will lead to a decreased pointing accuracy.

Limiting Factors, Predicted Performance, and Conclusion

I expect that people will show high selection accuracy with direct touch but might display low selection speed. Whether selection speed will be low will most likely depend on the structure of the input space: on a flat input space, which does not require menu navigation, selection speed will be high, on a hierarchical input space, it will be low. I expect people to show high selection speed with Room Pointing. Selection accuracy, however, might be reduced due less accurate feedback compared to direct touch. For Virtual Shelves, I expect people to show the same level of selection speed as in Room Pointing due to the similarities in selection mechanism. However, I expect significantly lower (initial) selection accuracy because of the differences in feedback.

Table 2: Comparison of Direct Touch, Room Pointing, and Virtual Shelves

Direct touch on on- screen icons: “Touch interface” (3.2.1) Mid-air full-arm pointing gestures toward real-world proxy-objects “Room Pointing” (3.2.2) Mid-air full-arm pointing gestures toward body- relative proxy-zones “Virtual Shelves” (3.2.3)

Selection mechanism Direct touch Mid-air full-arm pointing gestures

Mid-air full-arm pointing gestures Selection proxy On-screen objects _(icons) Real-world objects Virtual zones

Main feedback Visual Visual Proprioceptive

86

Main memory system Semantic Semantic Procedural Reference frame Device-relative Absolute Body-relative Main limiting factors

(selection accuracy) — Pointing accuracy

Gesture recall Pointing accuracy Main limiting factors

(selection speed) Menu navigation — —

Predicted accuracy High Medium Low

Predicted speed Medium High High