Automated Elbow Point Extraction and Upper Arm Length Estimation from 3D Human Body using Neural Network

2020 4th International Conference on Modelling, Simulation and Applied Mathematics (MSAM 2020) ISBN: 978-1-60595-674-9

Automated Elbow Point Extraction and Upper Arm Length Estimation

from 3D Human Body using Neural Network

Hao-yang XIE

, Xi CHEN

, Zhi-cai YU

and Yue-qi ZHONG

1_{College of Textiles, Donghua University, Shanghai 201620, China}

2_{Zhengzhou SIAS University, Zhengzhou, 451150, China}

3_{Key Laboratory of Textile Science and Technology, Ministry of Education, Donghua University,}

Shanghai, 201620, China

*Corresponding author

Keywords: 3D human measurement, Elbow point extraction, Upper arm length estimation, Neural network.

Abstract. Elbow point is a very important landmark for body measurement and certain garments. However, little attention has been invested. It is rather difficult to extract the elbow point based on the commonly used geometric methods since 1) the arm and elbow have high degrees of freedom, and 2) there is no prominent feature when the arm is unbent. In this paper, we use human stature, one of the most accessible measurement, to estimate the elbow point for arbitrary arm poses. Specifically, we first train two end-to-end neural networks for males and females to estimate the Euclidean length of the upper arm, and then propose a framework to approximate the elbow point and to calculate the tape measurement of the upper arm. Experimental results have verified that our method is efficient for clothing applications. The mean squared errors for male network and female network on the test dataset are 3.23 and 2.69, respectively. Both the absolute errors for the tape measurement of the upper arm for males and females are less than 1 cm.

Introduction

With the popularity of low-cost 3D scanner, like Kinect, more people can bring their virtual bodies into the virtual world. The dramatic increase in the number of virtual characters has spawned many 3D human-based applications, such as virtual try-on, customized sizing, customized clothing, etc. For most of these applications, the first step is to obtain various anthropometric parameters like stature, leg length, upper arm length, forearm length, waist girth, chest girth, etc. For 3D human measurement and landmarks extraction, the stature is one of the most accessible anthropometric parameters; however, the elbow point is one of the most challenging landmarks to detect, due to 1) the degree of freedom of arm and elbow is rather high, and 2) the prominent curvature feature will be lost if the arm is unbent. In this paper, we propose a new method to estimate the elbow landmark and the upper arm length (UAL) based on the minimum input, stature, using the neural network. In the rest of this paper, the UAL refers to the tape measurement, unless stated otherwise.

mesh itself. However, no matter what method, the first step is usually to extract landmarks or feature points from images, point cloud, mesh, and other representations. Whether the final measurement is accurate depends entirely on these features extracted.

All of the references mentioned above do not deal with the elbow point, but the elbow is crucial for some garment and particular purpose [9]. In this paper, we focus on the elbow point extraction. Our method is inspired by recent medical research [10], which points out that there is a linear relationship between human stature and the upper arm length. However, 1) they use the upper arm length to estimate the human stature, and 2) the linear relationship is not evident for females.

For a standing 3D human, stature is one of the most accessible measurements. In this paper, we establish a fully connected neural network to predict the Euclidean length of the upper arm from human stature. Then we propose a framework to approximate the elbow point and the UAL (tape measurement) automatically according to the upper arm Euclidean length. To our best knowledge, this is the first paper to estimate the elbow point using neural network for clothing and tailoring.

Training Data Collection

The 3D human models used in our research come from SPRING [11], and it contains over 3000 3D human models include 1517 males and 1531 females. All of the models are under standard A-pose, as shown in Fig. 1.

Figure 1. A-pose body shape. No serve bending on arm, leg, and torso.

Figure 2. Interactive calculating the Euclidean distance from shoulder point A to elbow point B. The blue and black colors denote the corresponding parts on the line segment AB are inside and outside the human surface, respectively.

Network Structure

We build a six-layered fully connected end-to-end neural network for our tasks. The network structure is illustrated in Fig. 3. There are four hidden layers, one input layer, and one output layer. Since we only use human stature as the network input, the input layer contains one unit. Similarly, the output layer also only includes one unit to indicate the Euclidean length of the upper arm. The opacity proportional to the connection weights.

Figure 3. Network structure used in this paper. The network includes four hidden layers, one input layer, and one output layer. The red connections denote the positive weights, and the blue connections denote the negative weights.

The opacity proportional to the connection weights.

We use the Rectified Linear Unit (ReLU) as the activation function in the hidden layers. The ReLU function is defined as:

𝜎(𝑥) = 𝑚𝑎𝑥(0, 𝑥) (1)

where 𝑥 denotes the output of the previous layer. The activation function is very important for our network since 1) the final output will be a linear combination of each layer without the activation, which is not the result we expect, and 2) the ReLU also alleviates the problem of gradient disappearance during training. Note that we do not add the ReLU to the output layer in our network. We use the Mean Squared Error (MSE) as the loss function, and it is defined as:

𝐽 = 1

𝑁∑ ∥ 𝑦𝑖 − 𝑦𝑖′ ∥2

𝑁

𝑖=1

(2)

Elbow Point Extraction and Upper Arm Length Estimation

The output of the networks are not the real tape measurements of the upper arm. It is usually less than the tape measurement. In this section, we introduce the framework we developed to estimate the elbow point and the real tape measurement. The method is described as follow:

(1) Predicting the upper arm Euclidean length 𝑙 according to our neural network model. (2) Locating the shoulder point 𝑝_{𝑠ℎ𝑜𝑢𝑙𝑑𝑒𝑟} based on the method described in [12].

(3) Extracting the human skeleton based on [13], and the upper arm direction 𝑑 can be determined by the skeleton part corresponding to the upper arm.

(4) Defining a plane 𝑃 that passes through 𝑝_{𝑠ℎ𝑜𝑢𝑙𝑑𝑒𝑟} and is perpendicular to 𝑑. (5) Moving 𝑃 along with the direction 𝑑 with distance 𝑙, and getting a new plane 𝑃′.

(6) Cutting the arm with 𝑃 and the point with the max x-value is consider as the elbow point

𝑝_{𝑒𝑙𝑏𝑜𝑤}, as shown in Fig. 4.

Figure 4. Algorithm for elbow point extraction.

Once we get the shoulder point 𝑝_{𝑠ℎ𝑜𝑢𝑙𝑑𝑒𝑟} and the elbow point 𝑝_{𝑒𝑙𝑏𝑜𝑤}, the tape measurement of the upper arm can be determined according to the following algorithm.

(1) Discretize the line segment 𝐿 with uniform sampling. The two endpoints of 𝐿 is 𝑝_{𝑠ℎ𝑜𝑢𝑙𝑑𝑒𝑟} and

𝑝_{𝑒𝑙𝑏𝑜𝑤}. The line segment 𝐿 can be denoted as 𝐿 = {𝑣₁, 𝑣₂, … , 𝑣_𝑛}, where 𝑛 is the number of sampling points.

(2) Moving the discrete line segment 𝐿 toward the direction 𝑑_𝑢𝑝 = [0, 1, 0]𝑇, we can then get a new discrete line segment 𝐿̅.

(3) Projecting 𝐿̅ on the human surface, the projected vertices consistent a discrete curve which can be used to approximate the real tamp measurement, as shown in Fig. 5.

Figure 5. Algorithm for tape measurement of the upper arm length. The blue curve approximates the tape measurement.

𝑝𝑠ℎ𝑜𝑢𝑙𝑑𝑒𝑟

𝑝𝑒𝑙𝑏𝑜𝑤

𝑑

𝑙 𝑛

𝑃′

𝑃

𝑙

𝑝𝑠ℎ𝑜𝑢𝑙𝑑𝑒𝑟

𝑝𝑒𝑙𝑏𝑜𝑤 𝐿 𝐿̅

Results

According to the method described above, we first manually measure 1000 3D human models, 500 per gender. The measurement results are illustrated in Fig. 6. Fig. 6 (a) is the result for females and (b) is for males. Intuitively, there is indeed a linear relationship between the human stature and the upper arm Euclidean length, and the linear correlation for males is stronger than females.

We train two neural networks to predict the upper arm Euclidean length from stature for males and females using the same network structure. For male model, we first split our 1500 male data into three parts, 900 for training, 300 for validation, and 300 for testing. We use the same style to divide the female dataset into the training set, validation set, and test set. The test set is never used during the training process. Thus, the test set is the actual unseen data for network models.

Fig. 7 demonstrates the MSE vs. training epochs for males and females. In our experiments, we do not normalize the input and output data. Thus, the output of our model is the upper arm Euclidean length, and no additional post-processing is necessary. Both the networks for males and females are trained 40 epochs, and the MSE on test set for males and females have achieved 3.23 and 2.69, respectively, which means that the absolute error is 1.79 cm and 1.64 cm, respectively.

Figure 6. Stature and upper arm Euclidean length distribution for females (a) and males (b).

Figure 7. Training loss and validation loss for females (a) and males (b).

Table 1. Mean measurement results for our real scans.

(cm) Stature UAL

GT Ours AE GT Ours AE

Female 176 176 0.0 32 31.34 0.66

Male 163 163 0.0 29 29.18 0.18

All the experiments are implemented on a laptop with 4 cores processor at 2.8 GHz and 8GB RAM. The entire process to estimate the elbow and upper arm is less than 0.3 s.

Conclusion

In this paper, we propose a new framework to estimate the upper arm length and the elbow point. The upper arm Euclidean length is calculated firstly based on our neural network models. The elbow point and real upper arm length are estimated based on the Euclidean length. To our best knowledge, this is the first paper to estimate the elbow point using neural network for clothing and tailoring. The experimental results also prove that our method is efficient for elbow and upper arm estimation. However, because all training data is from adults, the results cannot be used for particular groups such as children. We will collect more various 3D data to build network models for various groups.

Acknowledgement

This work is supported by the National Natural Science Foundation of China (Grant No. 61572124), and the Fundamental Research Funds for the Central Universities (Grant No. CUSF-DH-D-2017006).

References

[1] C.-Y. Tseng, I.-J. Wang, and C.-H. Chu, “Parametric modeling of 3D human faces using anthropometric data,” in 2014 IEEE International Conference on Industrial Engineering and Engineering Management, 2014, pp. 491–495.

[2] V. Sladek, J. Machacek, C. Ruff, E. Schuplerova, R. Prichystalova, and M. Hora, “Stature estimation from long bones in the Early Medieval population at Pohansko (Czech Republic): Applicability of regression equations,” in American Journal of Physical Anthropology, 2014, vol. 153, pp. 242–242.

[3] P. Rastogi, R. Murali, and S. Rastogi, “Hand Biometrics-A tool for gender and Stature estimation,” Journal of Forensic Medicine and Toxicology, vol. 31, no. 1 & 2, pp. 87–90, 2014.

[4] T. Kohlschütter and P. Herout, “Automatic Human Body Parts Detection in a 2D Anthropometric System,” in Advances in Visual Computing, 2012, pp. 536–544.

[5] Z. Li et al., “Anthropometric body measurements based on multi-view stereo image reconstruction,” in 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2013, pp. 366–369.

[6] J. W. Jo et al., “Automatic human body segmentation based on feature extraction,” International Journal of Clothing Science and Technology, vol. 26, no. 1, pp. 4–24, 2014.

[7] C. Dorin and B. Hurwitz, “Automatic Body Part Measurement of Dressed Humans Using Single RGB-D Camera,” in Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, New York, NY, USA, 2016, pp. 3042–3048, doi: 10.1145/2851581.2892337.

[9] W. Fang-yuan, “Research on the Optimum-Sized Sleeve Elbows Modeling of Women’s Garments,” Journal of Zhejiang Textile & Fashion Vocational College, no. 2, p. 8, 2012.

[10] S. Navid, T. Mokhtari, T. Alizamir, A. Arabkheradmand, and G. Hassanzadeh, “Determination of Stature from Upper Arm Length in Medical Students,” Anatomical Sciences Journal, vol. 11, no. 3, p. 6, 2014.

[11] Y. Yang, Y. Yu, Y. Zhou, S. Du, J. Davis, and R. Yang, “Semantic parametric reshaping of human body models,” in 3D Vision (3DV), 2014 2nd International Conference on, 2014, vol. 2, pp. 41–48.

[12] Y. Zhong and B. Xu, “Automatic segmenting and measurement on scanned human body,” International Journal of Clothing Science & Technology, vol. 18, no. 1, pp. 19–30, 2006.