WheelSense V2 (Embedded and Embodied)

7.2.3.1 Explored TGIF Aspects

As for other WheelSense systems, also this system allowed to explore the use of tangible gestures for the control application domain, specifically, for controlling the infotainment system of the car. The interaction was designed using the expert-driven method, with a reification of new gestures performed on a given object, the steering wheel. Following field requirements, all gestures were designed including the hold component. Gestures were designed exploiting embodied metaphors and the form affordances provided by this object and chosen according to the recognition performances allowed by the selected embodied and embedded technology for recognizing gestures (pressure sensors). The technological approach chosen for the implementation of this system is the main difference with the previous system. Software-side, machine learning has been used to recognize gestures. The design space of TGIF explored through this system is represented in Figure A3 in Appendix A.

20 This work has been published in [7] in 2013, in collaboration with Francesco Carrino, Stefano Carrino and Maurizio Caon.

7.2.3.2 Design Rationale and Methods

As for WheelSense V1, the aim of WheelSense V2 was to provide a richer user experience and a safer interface for the in-vehicle infotainment system. Following literature analysis and according to field requirement, the system should comply with the “eyes on the road, hands on the steering wheel”

paradigm introduced by González et al. [79]. To this purpose, gestures should oblige the user to keep both hands on the steering wheel in the position suggested by the Swiss manual for driving school teachers [103]. This system has been designed following an expert-driven method, with a particular attention to the requirement of firmly holding the steering wheel with two hands. The system design was oriented towards an embedded and embodied approach, with sensors integrated in the steering wheel, to recognize gestures performed on its surface, exploiting the form affordances of the steering wheel to guide tangible gesture execution. The tangible gesture design was partially influenced by the approach chosen to develop the system. Since pressure sensors have been chosen to recognize gestures performed on the steering wheel, following the embedded and embodied approach, the tangible gesture choice was limited to only those gestures that imply the application of a pressure on the surface of the steering wheel.

The implementation of the system required several iterations for the optimization of the physical design and the sensor integration. In particular, the sensor position was crucial for optimizing the system performances according to the chosen gestures. Therefore, different sensor positions have been tested and compared in order to obtain a good quality of sensor data. At the same time, the ergonomics of the steering wheel should not be affected by the integration of the sensors. These aspects will be further discussed in the next section, as part of the interaction design.

7.2.3.3 Interaction Design: Tangible Gesture Syntax and Semantics, Feedback

Four tangible gestures have been designed for this system (Figure 29). In all the four gesture-object pairs, the object is obviously the steering wheel. A hold component is always present in the gesture syntax to comply with the requirement of having constantly both hands on the steering wheel while interacting with the system. Three gestures that have been designed for the wearable system, i.e., the fist squeeze for stopping music and the dragging up and down for browsing songs, have been adopted also for this system. For the fourth gesture, an index tap while holding the steering wheel (which introduces an additional touch component) replaced the index abduction gesture (move component) for starting music. Indeed, not only the index tap has a better referential association to the starting music command, but it also exploits better the affordances provided by the steering wheel. The sound generated tapping the wheel provides inherent feedback to the user and foster the association between the steering wheel and a musical instrument, which in this case becomes the interface with the infotainment system of the car. To this purpose, it is worth recalling that the purpose of this system

was limited to the interaction with the music player, with the aforementioned commands for starting and stopping music and for browsing songs.

Figure 29. Representation of the four gestures: a) tap, b) dragging upward, c) dragging downward, d) squeeze.

Since tangible gesture execution relies on proprioception and on the inherent haptic feedback provided by the steering wheel, the visual demand for the driver to interact with the IVIS through this system is reduced, especially if coupled with proper feedback. Haptic feedback was excluded because, as stressed from Bach et al. [17], there is a risk of increased distraction if the secondary task competes on the perceptual resources required by the primary task. Indeed, haptic feedback coming from the road and perceived through the steering wheel has an important role in the driving task. As suggested by Wickens and Hollands [225], the perceptual resources needed for the secondary task can be distributed over other senses. As a result, to comply with the “eyes on the road” rationale, the only auditory feedback coming inherently from the IVIS was considered for this system, which is largely sufficient for the simple control tasks for which the system has been designed. Obviously, the purpose of the application, i.e. listening to music, ensures that the auditory channel is not disturbed, thus the feedback is effective.

As part of the interaction design, the object design was also important to ensure the best user experience in gesture execution. Indeed, the integration of the system in the steering wheel (a Logitech G27) required a particular attention for the object crafting. As discussed in Section 7.2.3.2, the sensor position was important to optimize recognition performance. At the same time, the surface of the steering wheel should remain smooth and soft, to allow proper operation of the steering wheel while

driving, as well as the slick execution of tangible gestures on its surface. For the same reason, I avoided adding tactile cues on the steering wheel to help the user spot the right position for performing gestures, relying only on the original physical form and ergonomics of the steering wheel chosen for the integration. Transparent adhesive tape has been used to stick sensors on the steering wheel, which ensured smoothness of the surface and an easy rearrangement of sensors for further testing.

Unfortunately, other design rationales, such as the visual appealing of the prototype, were sacrificed, resulting in a raw solution with wires hanging from the bottom of the steering wheel (see Figure 30).

Such poor aesthetics can affect the perceived usability of the system, as shown by Sonderegger and Sauer [191].

7.2.3.4 Implementation

The sensing system implemented in this prototype is based on five Tekscan FlexiForce sensors with a range of 0-1 lb²¹. They are connected to an Arduino Duemilanove board that converts signals to the digital domain and sends measured data to a PC for further elaboration through a wired serial connection. Data are acquired with a rate of 50 Hz. Four sensors have been placed on the right side of the Logitech G27 Racing Wheel, for the right hand, and one sensor has been placed on the left side for the left hand. The sensor placement on the steering wheel is depicted in Figure 30.

Figure 30. Placement of the 5 FlexiForce sensors

Sensor 1 is placed on the back of the steering wheel to recognize the tap gesture with the index finger. In a relaxed position, the hand generally covers the three other sensors. The wrist flexion and the wrist extension performed for the dragging upward and downward gestures uncover respectively Sensor 3 and Sensor 4. In order to segment gestures and avoid false positives that could occur while manipulating the steering wheel, an explicit segmentation strategy was adopted: the driver squeezes the left hand while gestures are performed with the right hand. This additional gesture for explicit segmentation is recognized through Sensor 5.

21 Tekscan FlexiForce: http://www.tekscan.com/flexible-force-sensors

The raw data of the five sensors are elaborated in the PC using the ARAMIS Framework [44].

First, gestures of the right hand are segmented setting a threshold on the left hand sensor. Afterwards, the segmented data are used as input for a Hidden Markov Model (HMM) classifier. The HMM classifier was configured with 4 hidden states with forward topology and implemented the Baum-Welch algorithm to find the unknown parameters. The data supplied to the HMM classifier are modeled as time series (as depicted in Figure 31). The whole architecture of the WheelSense system is reported in Figure 32.

Figure 31. Representation of the temporal signals associated to the four gestures: a) is index tap, b) is dragging upward, c) is dragging downward and d) is squeeze.

Figure 32. Block diagram of the WheelSense system architecture

A video of the system can be found at the following address: https://youtu.be/OfpAkJ3cUrE.

7.2.3.5 Evaluation

The aim of the evaluation was to assess the system accuracy as well as the usability of the system. The evaluation was composed of three phases: the evaluation of the gesture recognition accuracy in controlled conditions, the evaluation of the gesture recognition accuracy while the user was driving using a simulator and the evaluation of the system usability through a standard questionnaire. Eight users (six males and two females, aged 25 - 31) participated to these evaluations. The setup, depicted in Figure 33, is composed by a laptop that executes the recognition task and the City Car Driving

Simulator version 1.2²². The monitor on the right shows the results of the classification and it was used by the experiment supervisor.

Figure 33. A user participating at the evaluation.

During the first part of the evaluation process, the users were asked to perform each gesture 40 times, for a total of 160 gestures. The order of the gesture to be performed was chosen randomly and the user was guided by a graphical interface. The user was requested to rest at half of the recording phase. A 10-fold cross-validation test on the recorded data was performed. The resulting average accuracy was 87% and the standard deviation was 17%. The HMM classifier has been trained using the data recorded during the first phase. The 8 participants were asked to drive using the City Car Driving simulator and to interact with the IVIS through the gesture interface. In this case, the gestures were used to control a music player with the gesture semantics explained in the previous subsection.

The experimenter requested the command that the participants had to perform; they had to remain focused on the driving task and to perform the gesture only when they were feeling confident, i.e., when the participants considered interacting with the IVIS as not dangerous. The total number of gestures that each user had to perform during the driving simulation was 40 (10 per type of gesture).

The average accuracy was 82% and the standard deviation among users was 16%. In fact, during the experience, high variability between the different participants was noticed.

After the second phase, participants were asked to fill a System Usability Scale questionnaire (SUS) [31]. From the questionnaire the overall usability, perceived usability and the learnability had been calculated. The overall usability (calculated following the standard procedure) scored 84 points

22 City Car Driving - Car Driving Simulator http://citycardriving.com/

out of 100 (standard deviation: 13); the perceived usability scored 82 points out of 100 (standard deviation: 12); the learnability scored 91 points out of 100 (standard deviation: 17). The last two factors were calculated as suggested by Lewis and Sauro in [127].

The two performance evaluations showed a high variability among users, which affected also the results of the usability evaluation. This high variability could be explained with the different hands position of the users during the interaction. In some cases, the right hand was not always positioned over the pressure sensors, which decreased consistently the quality and the strength of the acquired signals. Variations could also occur over time: for example, in one case, the system confused several times a squeeze with a dragging up gesture, because the participant was not pressing anymore on one of the sensors. This suggests that a robust system should be difficult to achieve without taking into account the changes in the users’ behavior over time. An adaptive learning approach could be implemented in order to avoid this issue.