Upper arm

After estimating the lower arm orientation LineLAand locating the hand, this section

aims at locating the last missing joint of the arm: the elbow. Due to foreshortening issues, the accumulated information until now is not sufficient to allow us to directly calculate the exact location of the elbow, but is enough for evaluating where an elbow joint should be with the following contraints

1. Reasonable distance to the hand joint Phand

2. Must lie on the lower arm line LineLA

3. Reasonable distance to the shoulder joint Pshoulder

4. Must be a skin point of upper arm (orange points in Fig. 3.8(B))

In such a case, a filtering technique, such as particle filter, is the best fit technique to solve our problem. The first two constraints can be used for limiting the range of newly generated samples, while the last two give us clues to calculate the cost function.

Starting from Phand, we limit the search range by locating the farthest possible posi-

tion of elbow Pce along the line with the known length of lower arm LLA. However, the

real location of the elbow can be closer due to foreshortening. Therefore, from Pce to

P_hand, we sample candidate points Selbow (blue points in Fig. 3.8(C)) on LLA following

a 1D Gaussian distribution whose centre is located at Pce and σ is the forearm segment

length. For each candidate Si_elbow, using the walking direction knowledge, we generate a new set of candidates Sshoulder for shoulder following a semi-circle-distribution (yellow

rectangles in Fig. 3.8(C)). Each shoulder candidate S_shoulderj has a distance

dist(S_shoulderj , Si_elbow) = L_UA∗ ρua (3.10)

where LUAis the upper arm length and ρua is a random scale parameter (which is em-

pirically set from 0.7 to 1 in our experimentation) to deal with foreshortening of the upper arm. We will discuss ρua more in the next chapter. In summary, we combine

S_elbow and Sshoulder into a final list of candidates Scandidates, each candidate is a pair

define a cost function on which each candidate will be evaluated.

In order to verify the skin coverage of each candidate, as many other Pictorial Struc- ture approaches, we use a rectangle-form to represent the upper arm. We therefore create a rectangle from S_elbowi to S_shoulderj and calculate the skin pixels that fall inside this rect- angle. The cost function E_UAi for evaluating candidates i of Scandidates consists of two

terms which are defined below:

E_UAi = αuarm∗Hua1i + (1 − αuarm) ∗Hua2i (3.11)

in which αuarm represents the importance of one term to the other, the first term H1i

controls how close S_shoulderj is to Pshoulder estimated in Section 3.3.1. Note that each term

is normalized to the range [0, 1]. The first term is defined as follows

H i

ua1= min(

dist(Si_shoulder, P_shoulder)

M , 1) (3.12)

where dist(Si_shoulder, P_shoulder) is the distance between a candidate and P_shoulder. In our implementation, we set Mua equal to the head height because this value is small enough

to discard bad candidates while it is still large enough to cover the maximum movement range of upper arm during walk.

The second termH_ua2i controls how much skin pixels are represented by the rectan- gle. Taking our short-sleeve T-shirt constraint into account, the skin pixels of the upper arm only cover a portion at the bottom of the rectangle (or not at all), the ratio between this portion and the whole rectangle is indicated by a threshold rshirt, which is a value

between [0, 1]. This parameter is calculated approximately based on the ratios of how much the skin map covers the upper arm area. In our experimentation, we also tested our system in cases such that no upper skin arm were exposed with success. Although the upper arm orientation error is a little bit larger than that in normal cases, the shoulder constraint always keeps the process from estimating bad results. H_ua2i is described as follows:

H i

ua2= rshirt−

∑ LH − ∑U H

∑ total pixels (3.13)

where LH are skin pixels in lower half, U H are skin pixels in upper half. After many experimentations on different subjects, we empirically found that setting αuarm= 0.35

gives the best results. The candidate giving the lowest score of the cost function will contain the positions of the elbow Pelbowand shoulder Pshoulder that we are looking for.

Fig. 3.8 demonstrates that our method gives relative accurate value of the angle between upper arm and lower arm and is capable of dealing with the foreshortening problem. In Algorithm 4, we present the main steps of our process.

Algorithm 4: Main steps of the elbow estimation process by Monte-Carlo sam- pling technique.

Input: Hand position Phand, shoulder Pshoulder and lower arm orientation LineLA

Output: Elbow position Pelbow

1 S_candidates← ∅

2 S_elbow← initialize set of elbow candidates 3 for Si_elbowin S_elbowdo

4 S_shoulder ← initialize set of shoulder candidates 5 for Sj

shoulder in Sshoulder do

6 Add < S_elbowi , S_shoulderj > to S_candidates 7 for candidate l in S_candidates do

8 E_UAl ← calculate cost function using Equation 3.11 9 P_elbow← elbow position having minimum cost function E_UA∗ 10 return P_elbow

In term of performance, thanks to the known lower arm orientation (see Section 3.3.2), we can reduce the search space for elbow (using only a 1D Gaussian distribution), thus greatly reducing the final number of candidates in Scandidates. We ran our system

on a machine with configuration: CPU Intel core i5 (2.6GHz), 4GB RAM, and using Matlab environment. The upper arm processing was the longest computational time step in the whole process, it took 1.8-2.4s per frame (640×480). Since Matlab is slow in

running heavy loop computing, we believe that we can get faster results if using other programming languages (such as C++).