Lower arm

With the skin color model built during the preprocessing step (Section 3.2), we filter each frame using this model to produce the skin map Mapskin (a binary image in which

the value of skin pixels is 1, and the rest is 0), see Fig. 3.5(D). Combining this with the body part height ratio provided by anthropometry data [50], we can eliminate easily the head and neck areas from the skin map, thus leaving alone the skin of the left arm and probably a small portion of the right arm if it is not occluded by the torso. To deal with the noise and this unwanted right arm, we only preserve the skin pixels within the bounding box having the closest distance to that from previous frames. With the short-sleeve T-shirt constraint, as mentioned in Section 3.2, the lower arm will be fully exposed while only a small part (or none) of the upper arm is visible. This condition gives us enough information to correctly estimate these two body parts.

In order to correctly estimate the orientation of the upper arm, due to its different orientation with the lower arm, we need to separate the lower arm from the upper arm by removing the pixels belonging to upper arm in the skin map when performing line fitting. Although the number of upper arm points compared to lower arm points is small (or zero in case of no visible upper arm), the risk of still having some effects on the lower arm estimation is still high for our gait analysis system. To simplify the skin map of the arm without losing directional information, we apply a scanline (line of pixels) technique [43] which has already proved its efficiency in preserving body parts orientation over a 2D/3D cloud of points.

Fig. 3.8(A,B) presents the overview of this process. In summary, we firstly estimate a rough direction of the lower arm using linear least squares on all arm skin pixels to obtain a line LineroughLA, this line is for comparison purpose only. Based on the size

of the bounding box of skin points, we generate a list of vertical (from left to right) or horizontal (from up to down) scanlines in a way that has a maximum number of scanlines. We then cluster each scanline using k-means with k = 2 (i.e., two center points for each scanline). The Euclidian distance between these two points is used to distinguish the lower arm from the upper arm within a scanline: if the distance is large enough, that means each cluster centre belongs to one part of the arm (we keep two points); otherwise, they all belong to one part (we only keep the mid-point of these). By sampling this way, we always have two extreme points: one represents the hand and the other one represents the top point of the upper arm. Generally, the extreme point which is farther to the shoulder implies the hand position. In some special poses of gait, in which the hand is very close to the shoulder (first case in Fig. 3.8(A), not usual for a normal walk), kinematic constraints are needed to choose the correct one. The number of scanlines generated is a trade-off between accuracy and computational time. The more scanlines we have, the more exact the linear fitting is, but also the more computational work for the CPU. In practice, we set empirically a distance of 4 pixels (at resolution 540 × 960) between two consecutive scanlines for best performance.

A

B C

Figure 3.8: (A) Scanline type decision (horizontal or vertical) depends on LineroughLA

orientation and hand location. (B) Lower arm estimation using linear least squares on scanline clustering. The white line represents the rough line Line_roughLA, red dots are scanline centres that belong to the lower arm, while the orange dots do not. The final green line is estimated by performing linear least squares on the red dot set. (C) Sam- pling technique for estimating the upper arm: blue points follow a Gaussian distribution whose centre is located at the known elbow. Many yellow rectangles, such as C1,C2,C3

representing three candidates of upper arm, were generated.

Next, the task is to obtain the points that belong to lower arm. Due to the rise of foreshortening4problem when swinging the arm, we can not rely solely on the lower arm joint length LLAto remove the upper arm from the set. As shown in Fig. 3.8(B), taking the

hand joint P_has the center, we generate a list of rectangles List_rectla, in which each rect_lai has size (LLA, wlar) where wlarcontrols how close the points fall within the rectangle, by

4. Foreshortening is the visual effect or optical illusion that causes an object or distance to appear shorter than it actually is because it is angled toward the viewer (source: Wikipedia).

following a circle distribution with the angle from 0◦to 360◦ . The best rectangle, that is the one containing most of cluster central points, is calculated producing red points in Fig. 3.8(B). The drawback of this approach is that it could not work correctly if upper arm and lower arm have a similar orientation under foreshortening issue (see Section B.3 for more discussion). By using linear least squares on these red points, we can estimate the line LineLArepresenting the lower arm direction (green line in Fig. 3.8(B)). The index

of the extreme scanline belonging to the hand is then located. Finally, we locate the hand joint Phwhich is the projection of the hand scanline on LineLA. The detail algorithm can

be found in Algorithm 3.

Algorithm 3: Lower arm orientation estimation. Input: Skin map of the arm Mapskin

Output: Hand position Phand and the orientation of the lower arm LineLA

1 Line_roughLA← compute line fitting on Map_skin 2 Rect_bb← compute the bounding box on Map_skin

3 (width_b, height_b) ← calculate size of the bounding box Rect_bb 4 if width_b> height_bthen

5 Listscan← sample vertical scanlines along x axis

6 else

7 List_scan← sample horizontal scanlines along y axis

8 List_cps← apply k-means (k = 2) clustering on each scanline of List_scan 9 P_hand← locate two extreme points of List_cps, then choose the hand joint

10 List_LA← ∅

11 List_rectla← generate rectangles of size (L_LA, w_lar) with center at P_hand

12 rect_best ← ∅

13 for rect_lai in List_rectlado

14 if Number of center points in rect_lai> Number of center points in rect_best then 15 rect_best ← rect_lai

16 List_LA← points within rect_best 17 if List_LA6= ∅ then

18 Line_LA← compute line fitting on List_LA 19 return <P_hand, Line_LA>