Image descriptors for handshape appearance are used along with hand location and move-
ment trajectory based features in a sign spotting framework by [Dreuw and Ney., 2008,Alon
et al., 2009,Yang et al., 2010]. Farhadi et al. [Farhadi et al., 2007] propose a transfer learn- ing approach, where sign models learnt in a training domain are transferred to a test domain utilizing a subset of labelled signs in the test domain that overlap with those of the training domain (for instance, sign models learnt from one viewpoint can be transferred to a different viewpoint). These approaches do not explicitly distinguish between different handshapes and as a result do not leverage linguistic constraints on handshape transitions.
Buehler et al. [Buehler et al., 2009] describe an approach to automatically extract a
video template corresponding to a specified sign gloss (e.g., ‘GOLF’) from TV broadcast continuous signing video with weakly aligned English subtitles. A similarity score for a pair of windowed video sequences is defined based on image features for shape, orientation and location of the hands. This framework, however, treats the sign recognition problem as an instance of a general temporal sequence matching problem and does not exploit phonological constraints on signing parameters. Inter-signer variations are not addressed and the image alignment between hand image pairs is restricted to 2D rotations.
3.5 Summary
In summary, while there has been work that has looked at handshape articulation in sign language, none has modeled the linguistic constraints that govern the start and end hand- shape articulations in lexical signs. We formulate a data-driven probabilistic model, the HSBN, for start/end handshape inference in monomorphemic lexical signs. The HSBN utilizes a layer of hidden variables in order to represent the properties of sign- and signer- independent handshape variation. The properties of hidden variables and the probability distributions that relate the values adopted by these variables are estimated using a vari- ational Bayes learning approach utilizing a training set of monomorphemic lexical signs annotated with start/end handshape labels. A grouping of signs into distinct lexical items is also assumed. Since only a single HSBN model needs to be trained for the entire class of monomorphemic lexical signs, the learning algorithm is able to utilize examples of all signs in this class which in-turn substantially improves the robustness of the estimated HSBN parameters. As a result, the HSBN shows promising improvement in handshape inference performance on a person-independent handshape recognition task. The training set used in our experiment contains 2, 636 distinct signs, the corresponding number of productions from 5 signers is 6, 958. The test set used in our experiments contains 577 signs produced by a single signer. In query signs from the two-handed : same handshapes articulatory class, the HSBN yields further improvement in the handshape recognition accuracy by leveraging bilateral symmetry properties in the handshapes articulated on the two hands.
46
Task Hand-
shape References Approach Generality of approach
Hand pose tracking for general hand gestures 3D model based [Stenger et al., 2001, Lu et al., 2003, Tomasi et al., 2003, Sudderth et al., 2004, Chang et al., 2005, Bray et al., 2007, de La Gorce et al., 2008, Oikonomidis et al., 2012] Track finger articulations using a 3D kinematic model for the hand skeleton
Models self-occlusions; assumes constrained hand orientation, high resolution
hand images, good initialization and person
specific hand model 2D model based [Wang et al., 2008, Wu et al., 2005] Finger articulation via graphical and
piecewise planar models
Initialization not needed; assumes palm orientation parallel to the camera
Sign language handshape recognition from static images Active Appear- ance Models [Fillbrandt et al., 2003]
Uses PCA to capture shape and appearance
variations
Fast computation; learnt modes for shape and appearance variation are tuned to the training set Nearest neigh- bor [Athitsos, 2006, Athitsos et al., 2008a] Handshapes rendered using 3D model, BoostMap for fast
NN
Computationally fast and person-independent; difficult to match synthetic images to real hand images Handshape
classifica- tion for sign
recognition
Cluster- ing based
[Ong and Bowden, 2004, Bowden et al.,
2004]
Markov chain used to model hand shape, movt. and
loc. aspects
Good performance on 40 signs; requires colored
gloves, inter-person variations are not addressed Handshape verification 2D appear- ance features
[Alon et al., 2009, Yang et al., 2010]
Appearance used along with hand trajectory in DTW
framework
Used for sign recognition, sign spotting and retrieval;
inter-person variations are not handled Learn video templates for signs from continuous signing video with subtitles 2D appear- ance [Buehler et al., 2009, Buehler et al., 2008] Pictorial structures model for upper body tracking, hand shape, movt., loc. for
spatio-temporal similarity
Extracts video templates using multiple instance learning from broadcast
video with little supervision; between signer
variations not addressed image alignment restricted
to 2D-rigid Phonetic alphabet to describe hand loc. and hand movt. Finger angles from data- glove
[Vogler and Metaxas, 2001, Vogler and
Metaxas, 2004]
Phonetic transcriptions for loc.
and movt. aspects, parallel HMM (networks for each
channel)
Good recognition results for continuous signing; uses
multiple cameras and physical sensors mounted on the body for tracking in
3D movt.: hand movement trajectory, loc.: hand location with respect to torso.
Table 3.1: A review of approaches for handshape recognition (in sign lan- guage) and handpose tracking (general hand gestures).
ASL Lexicon Video Dataset (ASLLVD)
4.1 Objectives and requirements for the lexicon dataset
The data collection for this research was carried out with a view towards developing a com- puter vision system for lexical lookup in sign language video datasets. A ‘lexicon dataset’ containing isolated signs was collected and annotated with linguistic attributes to enable the implementation of a query-by-sign system for sign lookup in an ASL dictionary.
In this chapter we describe the aspects of the lexicon dataset as they pertain to the HSBN formulation for handshape inference in monomorphemic lexical signs. We envision that the lexicon dataset (in concert with other datasets under development for sign language) would facilitate further research into person-independent large vocabulary SLR. Further details
about the lexicon dataset are presented in [Neidle et al., 2012b,Athitsos et al., 2008b] and
in this webpage [Neidle et al., 2012a].
A large number of distinct signs is needed to provide a representative sample set of lexical items contained in the vocabulary. Productions from multiple native signers are also needed to train recognition methods that are able to accommodate variations in articulation. In this research, the lexicon dataset serves as the primary resource for articulatory patterns in ASL. It is hence essential that videos in the dataset be collected in a controlled environment to facilitate linguistic annotations and the development of reliable and accurate computer models for sign language recognition and retrieval.
Linguistic distinctions between different signs are often attributable to subtle differences in their articulation. To enable computer models to be trained to make such critical dis- tinctions, annotations of several linguistic properties are essential. Some of these required
annotations are listed in Section 4.2.4. To the best of our knowledge, previous datasets do
48
not include all these attributes.