Software for Gesture Recognition: Techniques and Algorithms

Four main different tasks have been individuated in Section 6.1 for the recognition of tangible gestures.

In this Section, I will not deal with the first task, i.e., the identification of objects, which is trivial when objects are identified through RFID (wearable approach), or when objects are already equipped with technology (embedded and embodied approach), while it becomes a complex task when the recognition is performed through computer vision (environmental approach). As stated before, in my thesis I did not investigate computer vision techniques for the recognition of tangible gestures. The other three tasks deal with the recognition of gestures, either implying movements, i.e., hands or objects movements (move, hold+move) and finger movements on surfaces (touch+move), or static finger touches on objects or surfaces (hold+touch, touch). As discussed in Section 5.6, two main system behaviors can be designed: either the system provides continuous feedback to the users while they are performing the gesture (e.g., for direct manipulation), or the system replies to the users only when the tangible gesture ends. The first behavior implies a particular recognition system that is able to couple continuously the user input to the system output. This requires continuous evaluation of users’ tangible gestures, and, in particular, an estimation of the parameters associated to the output with a transfer function. Different techniques can be implemented to obtain such behavior, but will not be discussed in depth in this thesis. Instead, I will deepen the discussion on systems that do not require continuous evaluation, thus, on systems that perform the recognition at the end of the users’ gesture. In particular, I will deal with three different topics: (i) gesture segmentation, (ii) gesture classification, and (iii) segmentation and classification fusion (for hybrid approaches). Feature extraction (as discussed by

Pavlovic et al. [155]) will not be discussed in this thesis since it could vary largely according to the application domain and the sensors used to recognize gestures.

6.3.1 Gesture segmentation

As evidenced in Section 2.3.3.3 for gesture interaction and as discussed in Section 4.3.2, also in tangible gesture interaction the delimitation of gestures in time is a key issue that should be addressed already during the design phase, in order to facilitate the spotting of gestures and the accuracy of the recognition. As discussed in Section 4.3.2, a gesture is delimited in time by a change in one component or attribute. In some cases, tangible gesture interaction offers an implicit support for gesture segmentation. For example, a hold+move gesture could be easily delimited through two events: the user take the object in the hand (gesture start) and the user release the object from the hand (gesture end). Similarly, a touch+move gesture could be easily isolated detecting when the user is touching the surface of the object. These considerations allow reducing the system attention to a shorter interval.

However, in some other cases, a more accurate segmentation could be required. For example, the designer could be interested in isolating the hold+move gesture from the preparation and closure phase of the gesture, which include picking up the object and reaching the desired start position as well as leaving the object when the gesture end. Similarly, the designer may need to separate subsequent gestures that the user could perform without releasing the object from the hand. Two strategies are possible: an explicit segmentation, operated by the user through an additional device or interaction modality, or an implicit segmentation, operated by the system through an automatic analysis of the signal. While the first strategy could be more accurate than the second one (since the users knows when they want to communicate something to the machine), it requires additional cognitive and motor resources from the users.

6.3.2 Gesture Classification

As for gesture interaction [155], the purpose of gesture classification in TGI is associating to an observed tangible gesture one of the gesture classes that have been conceived for the system. The advantage offered by tangible gesture interaction is the possibility to enlarge the gesture set without increasing the difficulty to recognize gestures: adding more objects to the system will allow increasing the gesture-object pairs without defining new gestures. Still, the recognition of the gesture is a key point that should be carefully addressed to build a tangible gesture interactive system. As for gesture interaction, the observed gesture can be represented through a vector of data, which can be either the

raw data collected by the sensors or features extracted from the raw data. In particular, two types of gestures should be considered: static postures, such as hold+touch gestures, and dynamic gestures, such as hold+move and touch+move gestures. While in static postures, the variation of the data over time, between the start and the end of the gesture is not important, in dynamic gestures, the temporal variation is particularly important and should be taken into account by the gesture classifier. Often, data related to dynamic gestures are considered as time series.

Two main types of gesture classifiers can be identified: those based on heuristics and those based on machine learning. Gesture classifiers based on heuristics exploit algorithms developed on purpose by the system designer to recognize gestures based on predefined criteria. Most simple heuristics are based on thresholds, which are determined by the designer after inspection of sensor data. Gesture classifiers based on machine learning (or supervised learning) rely on functions that are automatically generated by the system and that are able to determine to which class an observed gesture belongs. In order to generate these functions, the designer must provide to the classifier sufficient labeled examples for each gesture class. To ensure the genericity of the system, these gesture examples should be performed by different users, providing several repetitions for each gesture class. In this manner, the generated functions should be able to cope with typical variations that can occur across gesture executions of the different users. An alternative approach granted by machine learning classifiers is training the system only on one person, i.e., the one that will use the system. This is particularly interesting for all systems designed following the wearable approach, as well as for the systems embedded in personal objects.

The possibility to train the system only on the user not only allows increasing the accuracy of the classifier, but allows also the user to define his or her own set of gestures, without the need to learn gestures chosen by the designer. Several classification algorithms based on machine learning exist.

Among them, Hidden Markov Models, Support Vector Machines and Linear Discriminant Analysis are often used for gesture classification. The choice of the different algorithms to be used depends on the specific application and in particular on the data collected by the sensors. Moreover, most algorithms allow tuning several parameters to optimize the recognition accuracy. This operation requires additional data (independent from the training set) to test the performances of the classifier with the different parameters.

It is worth noting that two type of outputs are possible for machine learning classifiers: hard classifiers provide crisp results, assigning an observed gesture to a class or to another one; fuzzy classifiers are able to provide soft results, generating as output the likelihood (a probability that ranges from 0 to 1) for each class that the observed gesture belongs to that class (the chosen class will be that with the highest likelihood). This additional information is particularly interesting when comparing the

results of different classifiers and, in particular, for the problem of classifier fusion, which is discussed in the following Section.

6.3.3 Classification and segmentation fusion¹⁷

As discussed in Section 6.1.4, in some cases it is interesting to benefit of tangible gesture interactive systems developed with different approaches and to merge them into a single synergistic system that exploits the best capabilities of each sub-system. In particular, each sub-system is equipped with different technologies for spotting tangible gestures and for classifying them. Therefore, an intelligent strategy for merging the classification and segmentation results of each sub-system should be found.

In order to merge the result of different classifiers, an approach based on late fusion has been investigated [10]. Since the fusion is applied only at the last stage of the classifiers, the synergistic system would not be affected by changes in the sub-system classifiers. In the context of my research, I investigated and compared 10 different methodologies to weight and merge the results of each classifier are presented in this section. While some methodologies were taken from literature others have been conceived and assessed in the context of this research [10]. These methods can be applied only to classifiers with soft outputs, while some methods cannot be applied to more than two classifiers.

Most of these methods rely on a weighting process that is based on the confusion matrices, which gives an a priori knowledge of the ability of each classifier to recognize correctly a given gesture (and the probability to miss-classify it with another gesture class). Confusion matrices of the classifiers can be estimated in a cross-validation phase (for example, performing a k-fold cross-validation on the training set). For brevity purposes, the proposed methods will not be reported in this thesis. The reader could refer the published article [10] for further details on the proposed fusion methods.

For the realization of a synergistic paradigm (cf. Section 6.1.4), a critical step is also the processing of the segmentation signals. Mainly for two reasons: first, from an interaction point of view the segmentation can have an impact on the cognitive load on the user; second, having sub-systems working asynchronously, the fusion system can receive gestures that are delayed and it can deal with missing signals. The fusion classifier should deal with the possible delay between the outputs of the different classifiers by fixing a maximum delay d that should be accepted to consider two asynchronous sub-system outputs as belonging to the same gesture. When the delay between the outputs of the two classifiers is greater than d, gestures are treated separately and results are not merged. This approach

17 This section of the framework has been pushed in 2013 in [10] in collaboration with Stefano Carrino, Francesco Carrino and Maurizio Caon.

implies that the user should wait at least the time d between a gesture and the following one. In a synergistic system, three different strategies can be implemented to merge the segmentation decisions coming from two or more sub-systems. In the case of two classifiers, the OR strategy considers as valid results the contribution of both single and coupled classifiers (as the logic OR). The AND strategy takes into account only gestures simultaneously detected by the two classifiers. The ADAPTIVE strategy opportunistically switches between the OR and the AND strategies, taking into account the a-priori probability that the classifiers could have erroneously spotted (false positive) or missed (false negative) a gesture. This a priori probability can be calculated from the confusion matrices of the single classifiers.

6.4 Conclusion

This Section provided guidance on how to build tangible gesture interactive systems, both from a hardware and software perspective. In particular, it presented three different approaches (embedded and embodied, wearable, and environmental) for integrating technology respectively in the objects to interact with, on the user, and in the environment, as well as a hybrid approach for mixing systems of different nature. Design criteria are provided to evaluate the advantages of the different approaches and what can be obtained by mixing them together. Moreover, this section presented criteria for choosing the hardware platforms for embedding tangible gesture interactive systems in the objects or on the user body, as well as a generic overview on software techniques for segmenting and recognizing tangible gestures. Examples of implementations of the embedded and embodied approach, of the wearable approach as well as of the hybrid approach are presented in the following chapter. For practical reasons, the environmental approach and computer vision techniques for recognizing tangible gestures have not been explored in this thesis.

Part III – Framework