Virtual Keyboard Text Input

Virtual keyboard based text input systems are examples for GUI interaction that combine cursor control and item selection. These text input systems have already been largely investigated with different motion based inter- faces, which are a super set of freehand interfaces. I first describe several systems using a hand-held device such as the Nintendo Wii Remote. After that, I present several attempts to implement such text input in freehand interaction as with the Kinect.

In the text entry system presented by Shoemaker et al. [161], characters were written by moving the Nintendo Wii Remote in 2D or 3D space and then pressing a button. The motion of the Nintendo Wii Remote was tracked via its infrared camera and LEDs. A comparison of a QWERTY layout, a circular layout and a 3D cube layout revealed that the QWERTY layout outperformed the other layouts both in terms of speed and error rate. Depending on the distance between the participants and the screen, the authors measured a performance of 14.5 WPM–18.9 WPM, and error rates of 2.4%–8.5% whereby the performance decreased with the distance from the screen. A similar approach was WiiNote developed by Mugellini et al. [126] that, in addition to keyboard based text input, also offered a gesture alphabet based mode. However, no data regarding the performance of this system were given in their paper. The main difference between these two approaches and our work is that the users actually have a device in their hands and that the selection of keys happens via pressing a button on this device. As I do not want to additionally employ a hand-held device, I have to explore other possibilities for the key selection.

A potential solution was given by Jones et al. [81] who solely used the accelerometer of a Nintendo Wii Remote for key selection. They avoided the need to use a button by including a selection area in their keyboard to which

users had to move back to between the characters. This way of continuous writing was in line with earlier approaches on PDAs like Quikwriting [143] and Cirrin [116] that both had the characters arranged circularly around a centered selection area. Jones et al. compared two different keyboard layouts with their approach, a “matrix layout” that arranged groups of characters in squares around a center a square, and a “tri-center layout” that arranged the characters in separate squares around three centers. In a first study, the matrix layout that was more close to Quikwriting gained a better performance with 3.7 WPM and 9% error rate in comparison to 3.3 WPM and 19% error rate for the tri-center layout. At the end of a longitudinal study with four sessions, the participants managed to improve their average speed in both layouts to 5.4 WPM. However, the users still had to hold a device in their hands.

In the approach by Markussen et al. [117], this disadvantage was slightly reduced, as users only had to wear a glove prepared with markers for track- ing. The markers were tracked with an OptiTrack motion capture system. The hand motion was used to control a cursor and tapping with the index finger was used for selection. They further investigated three different key- boards: a multi tap system as used in mobile phones with nine button, a QWERTY layout, and H4 which was an own layout that had been adopted from text entry with a game controller. They measured the highest writing speeds for the QWERTY layout with 11.63 WPM, and the lowest writing speed with 4.19 WPM for H4.

Figure 2.19: Microsoft Xbox Kinect text input

The need for a hand-hold device was completely avoided in the Microsoft Xbox Kinect interface, to which Microsoft added text input with the Bing search in 2012 (Xbox Firmware 2.0.14699.0). It used a layout which ar- ranged characters alphabetically in a single horizontal line as shown in

character, and while approaching it, the line of characters was zoomed to display only 10 characters for easier selection. By performing a pushing gesture in upward direction, the intended character was selected. To speed up text writing, Microsoft made use of auto-completion. The advantage of the keyboard layout was that it only requires a small area on the screen. Disadvantages were the high arm position during interaction, the perma- nent interruption of the cursor movement by the pushing gesture and the frequent change of the interface caused by the zoom-effect. Indeed, Hoste et al. [71] measured a rather low writing speed for the Microsoft text entry system of just 1.83 words per minute (WPM, with one word = five char- acters, and no auto-completion active). Together with the high error rate of partially more than five errors per sentence, there was a lot of room for improvement.

Ren et al. [148] compared three different character selection techniques and two keyboard layouts for virtual keyboard text input using a depth sensor. They compared a dwell based selection method (Timeout ) with two methods employing hand movements in specific directions for charac- ter selection (Reach and Expand&Reach). Furthermore, they compared a standard QWERTY layout with a dual-circle layout. For the dwell-based method, they chose a dwell time of 1.2 seconds. They measured the fastest writing speed for the Reach method on a dual-circle keyboard layout with 8.57 WPM on the fifth day of a longitudinal study. The least error-prone method was the dwell-based one on the dual-circle keyboard with 0.00% error rate on the fifth day.