Direction based recognition using Levenshtein and context

The direction based algorithm is based on the direction the points in a gesture stroke progress. Possible directions a stroke can progress is divided into 8 different zones like an 8-connected path which can be seen in figure 4.4. For each point in the gesture stroke the algorithm starts by calculating the angle between itself and the following point. The angle is used to determine which one of the directions that are closest to the vector formed by these two points. The output from this is a chain of values describing all the shifts in direction during the gesture stroke. An example sequence is shown in figure 4.3, with the matching path illustrated in figure 4.5. The pseudo code for building the direction list is listed in 4.1.

1 def calculateDirectionList(gesture):

2 #build list of directions of each step in the gesture 3 lastPoint = None

4 for point in gesture: 5 directionList = [] 6 if my_lastPoint == None: 7 lastPoint = point

8 else:

9 directionList.append(calculatePointDirection(lastPoint, ←-

→ point))

10 return directionList

11

12 def calculatePointDirection(fromPoint, toPoint): 13 x = toPoint.x - fromPoint.x

14 y = toPoint.y - fromPoint.y 15

16 angle = math.degrees(math.atan2(x,y)) 17

18 if angle < 0:

19 angle = 360-abs(angle) 20

21 direction = int(round(angle/45.0)) 22 if direction == 8: direction = 0 23

24 return direction

Listing 4.1: Pseudo code for building the direction list

When you have obtained the direction sequence for the gestures you want to compare, there are different possibilities for how to proceed. During this project two direction based recognition algorithms has been implemented, one using a well known method of comparing strings, and one implemented using a self invented context sensitive method.

The well known method for comparing strings is ”edit distance”, and in this case Levenshtein distance [16] is used. Levenshtein distance is defined as the minimum

4.2. RECOGNITION METHODS 47

0

1

2

3

4

5

6

7

Figure 4.4: Illustration of the different directions a gesture stroke can progress. number of edits needed to transform one string into another, with the allowable edit operations being insertion, deletion, or substitution of a single character. Lev- enshtein distance is chosen because of its simplicity, and the ability of comparing strings of unequal length. The pseudo code for Levenshtein method is shown in listing 4.2.

1 def levenshteinDistance(directionList1, directionList2): 2 listLength1 = len(directionList1)

3 listLength2 = len(directionList2) 4

5 #Matrix of the same size as the two direction lists to compare 6 matrix = [0...listLength1][0...listLength2]

7

8 #Fill inn the initial direction index in the matrix 9 for i in range(0,listLength1 + 1):

10 matrix[i][0] = i 11

12 for j in range(0,listLength2 + 1): 13 matrix[0][j] = j

14

15 #Start in the top left corner , and start filling out the matrix 16 for j in range(1,listLength1 + 1):

17 for i in range(1,listLength2 + 1 ):

18 if directionList1[i-1] == directionList2[j-1]: 19 matrix[i][j] = matrix[i-1][j-1]

20 else:

21 matrix[i][j] = min( (matrix[i-1][j]+1) , #Deletion 22 (matrix[i][j-1]+1) , #Insertion 23 (matrix[i-1][j-1]+1)) #Substitution 24

25 #The number in the down right corner is now the minimum edit distance

26 return matrix[listLength1][listLength2]

Listing 4.2: Pseudo code for Levenshtein distance algorithm

The self invented solution takes in to consideration the actual context, and how these direction numbers relate to each other. The difference between direction 3

48 CHAPTER 4. GESTURE RECOGNITION

and 4 is much less than the difference between 3 and 7. Instead of calculating how many insertions, deletions and substitutions, this method calculates the distance between two gestures in a much more fine grained manner. The pseudo code for the context sensitive direction method is shown in listing 4.3.

1 def contexSensitiveDirectionScore(directionList1, directionList2): 2 listLength1 = len(directionList1)

3 listLength2 = len(directionList2) 4

5 #Check if lists are of equal length 6 if listLength1 != listLength2:

7 return "Error"

8

9 #Initiate distance counter 10 totalDistance = 0

11

12 #For each index in the two lists add the difference in direction 13 for i in range(listLength1):

14 totalDistance += abs(directionList1[i]-directionList2[i]) 15

16 #Return the total distance between the direction lists

17 return totalDistance

Listing 4.3: Pseudo code for the context sensitive distance algorithm By showing two simple examples, it is easy to see that there is huge difference in the score these two methods will give for matching gestures. The two examples show a more or less straight line drawn from left to right. By comparing the ges- tures in example 4.3 and 4.4 we get almost the same score using both Levenshtein distance and the self invented method. For clarity example 4.3 and 4.4 are illus- trated in figure 4.5 and 4.6 respectively. Using Levenshtein will give a match with the score 87.5%, while the self invented method will give a score of 95.8%. There is nothing wrong with this result, as minor differences in score will always be the case.

[2, 2, 1, 1, 2, 2, 1, 2, 1, 2, 2, 2] (4.3)

[1, 2, 1, 2, 2, 2, 1, 2, 2, 1, 2, 2] (4.4) It is first when we try to compare two similar gesture stroke shown in example 4.5 (figure 4.7) and example 4.6 (figure 4.8) we will see a huge difference between the algorithms. In this case my algorithm gives a score of 89.3% while the method using Levenshtein gives a score of only 25%.

[1, 2, 2, 2, 3, 2, 1, 2, 2, 2, 3, 2] (4.5)

4.2. RECOGNITION METHODS 49 2 2 2 2 2 2 2 2 1 1 1 1

Figure 4.5: Illustration of the different directions a gesture stroke can progress.

2 2 2 2 2 2 2 2 1 1 1 1

Figure 4.6: Illustration of the different directions a gesture stroke can progress.

2 2 2 2 2 1 1 2 2 2 3 3

Figure 4.7: Illustration of the different directions a gesture stroke can progress.

2 2 2 1 2 2 2 1 2 3 3 ₃

Figure 4.8: Illustration of the different directions a gesture stroke can progress.

Although the two examples presented here are simple, this will be the case for many different gestures strokes. Even though you try to be accurate when you draw a gesture, a bit wobbly noise is really hard to get rid of, and some of the directions gets interpreted as an another direction than that you intended.

To make the direction based method more robust, a rotation invariant mode has also been implemented. If the rotation invariant parameter is set to True, the gesture stroke is checked for a match in eight different angles by rotating the gesture around. Since we already have the direction sequence, the only thing we have to

50 CHAPTER 4. GESTURE RECOGNITION

do to rotate the stroke is increasing each direction number by one, and apply 8 modulo to wrap the number around. The gesture is also checked backwards to try all possible ways a user can draw the symbol. In this mode the algorithm will have to run 16 times, but since the main computation time lays in calculating the direction list, this does not have a huge impact on the computation time. The algorithm is still able to recognize gestures in real time, without the user noticing any delay.

To further enhance the direction based approach other algorithms for edit dis- tance can be implemented. It is also possible to use other types of algorithms to compare two strings like for example some kind of artificial neural network ap- proach explained in section 4.2.6.