Detection of partially occluded human using separate body parts classifiers

(1)

DETECTION OF PARTIALLY OCCLUDED HUMAN

USING SEPARATE BODY PARTS

CLASSIFIERS

BY

NURUL FATIHA BINTI JOHAN

INTERNATIONAL ISLAMIC UNIVERSITY MALAYSIA

2015

(2)

DETECTION OF PARTIALLY OCCLUDED HUMAN

USING SEPARATE BODY PARTS

CLASSIFIERS

BY

NURUL F ATIHA BINTI JOHAN

A dissertation submitted in fulfilment of the requirement

for the degree of Master of Science in Mechatronics

Engineering

Kulliyyah of Engineering

International Islamic University Malaysia

MAY 2015

(3)

ii

ABSTRACT

The application of computer vision in the surveillance system has provided huge advantages in the field of security and safety system. In recent years, human detection and classification subjects have shown an increasing focus in finding specific individual such as in the case of detecting person in crowded places at a time. Detection and classification of human can be a challenging task due to the wide variability of human appearance in terms of clothing, lighting conditions and the occlusion. These constraints directly influence the effectiveness of the overall system. To cope with these problems, human detection and classification system is presented in this thesis which requires fast computations in addition of accurate results. The propose system will first detect the human in an image by using YCbCr color thresholding for skin color detection algorithm and then classify the body parts using artificial intelligent neural network classifier into specific class and finally extend the classification system with the majority voting technique in order to improve the classification performance.The first hypothesis of the research is that YCbCr skin color detection method can be used to detect and identify the exposed human body parts even with the existence of various illumination conditions and complex background. In this work, the body parts then only cover face and hands. The body features are then extracted using feature extraction technique with the dimension of region detected fixed to a standard size.These body features are then used as an input to neural network system in order to classify the body parts into specific class. Meanwhile each class consists of three classifier which is taken from the extracted body regions and separated into face classifier, right hand classifier and left hand classifier. Finally, the results of each body parts classification will be processed using majority voting technique for overall conclusion of the classification system which is robust to partial occlusion. Experimental results indicate that the human detection using YCbCr color space is capable to detect the human body with the percentage of face detection is 92%, right hand detection is 86% and left hand detection is 85%. Meanwhile the performance of ANN classification system is successful in identifying face, right hand and left hand which are 90%, 73% and 74% respectively. Whereas, the accuracy of all 9 classes (Class A until Class I) is found to be 43% and highest to be 95%. Based on the extended classification system using majority voting technique, the results have shown a bit improvement on the classification performance for all 9 classes which is the lowest is increase to 45% and the highest is increase to 100%.

(4)

iii

ثحبلا ةصلاخ

نإ

بع تاقيبطتلا

تاونسلا في .ةملاسلاو نملأا ةمظنأ لامج في ةلئاه ياازم ترفو ةبقارلما ماظن في رتويبمكلا

،ةيرخلأا

ترهظأ

لأا تاعوضولماو يرشبلا فشكلا

في زيكترلا فينصت يلع ةردقلا في ةدياز ترهظأ يرخ

ةحمدزلما نكاملأا في ناك ول تيح ينعم صخش ىلع روثعلا

هسفن تقولا في

.

نإ

صتلاو فشكلا

يرشبلا فين

ارظن ةبعص ةمهم نوكي نأ نكيم

و ةءاضلإا فورظ و سبلالما ثيح نم ناسنلإا رهظم في ةعساولا تابلقتلل

ياؤرلا دادسنا

ايرثتأ رثؤت دويقلا هذه .

ارشابم

.ماعلا ماظنلا ةيلاعف ىلع

و

فينصتو فشكلا مادختسا نم دبلا

تبااسح بلطتت تيلا ةحورطلأا هذه في يرشبلا ماظنلا

رس

ةفاضلإبا ةعي

إلى

ا

ةقيقدلا جئاتنل

ماظنلا موقيس .

ممادختسبا ةروص لكش في ناسنلإا نع لاوأ فشكلبا

يسحسا اردلإاو نوللا ةجةاع

مث نمو فشكلا ةيمزراوخ عم

منصت

مهفنصتو يعانطصلاا ءاكذلا مادختسبا مسجةا ءازجأ ف

ل

ايرخأو ،ةنيعم ةئف

ةينقت عم فينصتلا ماظن عيسوت

لا

تيوصت

جأ نم

فنصلما لعجو فينصتلا في ءادلأا ينستح ل

أ

حترقلما ماظنلا.يئزجةا دادسنلاا ةيحنا نم يوق

قفو مدختسي

ةرشبلا نول نع فشكلا ةيمزراوخ

و

ماظنو ةيبصعلا ةكبشلا

يعانطصلاا ءاكذلا

ثيح

ىلع تينب

مةعست

عضاو

لامسجةا في

.يرشب

و

نم لىولأا ةيضرفلا

في

ثحبلا

لياحسا

نأ يه

يسحسا اردلإاو نوللا ةجةاعم

ةقيرط

لل

دختسا نكيم ةرشبلا نول نع فشك

ا

دوجو عم تىح ناسنلإا مسج نم ءازجأ ديدتح و فشكلل اهم

فلتخلما فورظلا

ة

.ةحضاو يرغلا ةيفللخاو ةءاضلإاك

،ثحبلا اذه في

مهلما

مسج ءازجأ نم

ناسنلإا

هجولا

نيديلاو

طقف

يضوت متي ث نمو

مادختسبا تازيلما خ

زيمتلا ةينقت

و

مجح لىإ ةتبثا فاشتكا ةقطنم نم دعبلا

ممتي ث .يسايق

مادختسا

مم

تائف لىإ مسجةا ءازججفينصت لجأ نم ةيبصعلا ةكبشلا ماظنل لخدمك مسجةا تازي

ةنيعم

فاضإ .

ة

كلذ ليا

،

ةثلاث نم فنص لك نوكتي

أ

فانص

لآا قفو تفنصو مسجةا في قطانم نم تذخا

،تي

و نىميلا ديلا

.فينصتلاةيحنا نم ىرسيلا

ايرخأ

ممتتس ،

مادختسبا مسجةا ءازجلأ فيناصتلا لك جئاتن ةجةاعم

يرشت ةيبيرجتلا جئاتنلا.يئزجةا دادسنلاا ةيوقتو فينصتلا ماظنل ماعلا جاتنتسلاا عم ةيبلغلأا قفو تيوصتلا ةينقت

ق يرشبلا نولل يسحسا اردلإا و نوللا ةجةاعم مادختسبا فشكلا نأ لىإ

ناسنلإا مسج نع فشكلا ىلع ردا

لدعبم هجولا نع فشكلل ةيوئلما ةبسنلا عم

92 لدعبم نىميلا ديلا نع فشكلاو ، ٪

86 ديلا فشك و ٪

لدعبم اضيا ىرسيلا

85 ٪

.

,هسفن تقولا فيو

أ

ةيبصعلا ةكبشلا ماظن ءادأ رهظ

نبا

ديدتح في حجانفينصتلا

اجو ىرسيلا ديلا و نىميلا ديلا ،هجولا

ء

ك ت

لآا

تي

90 ، ٪

73 و ٪

74 مت ،ينح في .لياوتلا ىلع ٪

روثعلا

ِِ ِ( ةئفلا نم ةعستلا تاقبطلا عيمجة ةقدلا ىلع

أ

( تىح )

ذ

نوكتل )

43 تناك ىلعلأاو ٪

95 ىلع ءانب .٪

ةينقت مادختسبا فينصتلا ماظن

لا

تاقبطلا عيمجة فينصتلا ءادأ ىلع لايلق انستح جئاتنلا ترهظأ ،تيوصت

دأ تغلب ثيح ةعستلا

ةدياز نى

45 تلصوامنيب ٪

لأا

لع

لىإ

100 ٪

.

(5)

iv

APPROVAL PAGE

I certify that I have supervised and read this study and that in my opinion; it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Master of Science in Mechatronics Engineering.

….…... Yasir Mohd Mustafah

Supervisor

….…... Nahrul Khair Alang Md Rashid Co-supervisor

I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a thesis for the degree of Master of Science in Mechatronics Engineering.

……… Amir Akramin Shafie

Internal Examiner

……… Hadzli Hashim

External Examiner

This thesis was submitted to the Department of Mechatronics Engineering and it is accepted as a fulfilment of the requirement for the degree of Master of Science in Mechatronics Engineering.

……… Md Raisuddin Khan

Head, Department of Mechatronics Engineering

This thesis was submitted to the Kulliyah of Engineering and is accepted as a fulfilment of the requirements for the degree of Master of Science in Mechatronics Engineering.

……… Md. Noor Salleh

(6)

v

DECLARATION

I hereby declare that this dissertation is the result of my own investigations, except where otherwise stated. I also declare that it has not been previously or concurrently submitted as a whole for any other degrees of IIUM or other institutions.

Nurul Fatiha binti Johan

(7)

vi

INTERNATIONAL ISLAMIC UNIVERSITY MALAYSIA

DECLARATION OF COPYRIGHT AND AFFIRMATION

OF FAIR USE OF UNPUBLISHED RESEARCH

DETECTION OF PARTIALLY OCCLUDED HUMAN USING SEPARATE BODY PARTS CLASSIFIER

I hereby affirm that The International Islamic University Malaysia (IIUM) holds all rights in the copyright of this work and henceforth any reproduction or use in any form or by means whatsoever is prohibited without the written consent of IIUM. No part of this unpublished research may be reproduced, store in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without prior written permission of the copyright holder.

Affirmed by Nurul Fatiha binti Johan

…... ….……… Signature Date

(8)

vii

ACKNOWLEDGEMENTS

First of all, Alhamdulillah, a sincere praise to Allah the Almighty since with His Power and Authorization, I have completed my master dissertation successfully. My highest appreciation goes to Universiti Teknikal Malaysia Melaka (UTeM) for their financial and moral support and gives me an extra time to complete this thesis.

A million of thank you to my supervisor, Dr Yasir Mohd Mustafah for his suggested topic along with the encouragement and guidance during the research. Thanks for his willingness to spend his time in giving ideas, instructions, support and motivations throughout my research.

I am also thankful to my co-supervisor, Dr Nahrul Khair Alang Rashid for his supervisions and keen support in assisting this work. I am deeply grateful to my special friend, Nursabillilah bte Mohd Ali for many helpful suggestions and being such a good tutor to me. Thanks for the encouragement and willingness to share the ideas and information to complete this research. Next, the everest of thank you to my family especially my husband for his support, patience, bless and understanding during my study period.

Finally yet importantly, thanks to all my friends and lecturers whose direct and indirect support helped me during this study. I really appreciate it and thank you for what they have done for me.

(9)

viii

LIST OF TABLES

Table No. Page No. 2.1 Human detection methods using background subtraction 17 2.2 Human detection methods based on direct detection 18 2.3 Summary of Color Models 23 2.4 Types of classification algorithm and their performance rate 30 3.1 Skin color threshold value in YCbCr color space 42 3.2 Number of detection based on types of human detection result 53 3.3 Performance of human detection result 55 4.1 Successful classification of human body parts 84 5.1 Results from Class F 92 5.2 Sample of false Classification 93 5.3 Example of partial occlusion 94 5.4 Performance of class F using majority voting technique 97 5.5 Performance of the overall system 98

(13)

xii

LIST OF FIGURES

Figure No. Page No

1.1 Flowchart of the research methodology 7

2.1 Digital image 12

2.2 RGB model in 3D 13

2.3 HSI color space 15

2.4 RGB color cube in the YCbCr space 16

2.5 Foreman image 22

2.6 The histogram distribution of Cb and Cr components 22 2.7 Skin color distribution 22 2.8 Erosion and dilation image 25 2.9 Opening and closing operation 25 3.1 Proposed system for human body parts detection 36 3.2 Skin color among various ethnics groups 37 3.3 Representation of pixel element 38 3.4 The separation of Y, Cb and Cr components 40

3.5 Conversion Image 41

3.6 Binary color formation 43 3.7 Structure of opening and closing operation system 45 3.8 Example of morphological basic operation 46

3.9 Opening operation 47

3.10 Closing Operation 47

(14)

xiii

3.12 Bounding box illustration 49

3.13 Human body parts detection image bounded in a rectangular box 50

3.14 True positive in human body parts detection 52 3.15 False positive in human body parts detection 52 3.16 False negative in detection of human body parts 53 3.17 The performance of detection result 56

4.1 Block diagram of the proposed system 58 4.2 The detected body parts before cropping 59 4.3 The cropped image 60

4.4 Distribution of human body dataset 60

4.5 Example of the face feature extraction technique 61

4.6 Feed-forward multilayer perceptron (MLP) network architecture 63

4.7 Artificial neurons structure 64

4.8 Generic representation of output and target data 66

4.9 Block diagram of classification system using ANN classifier 67 4.10 Face Images 68

4.11 Neural network architecture for face classifier 70

4.12 Training system 70

4.13 Testing system of neural network 72

4.14 Load testing file of 001 image 72 4.15 Image after load testing file 73

4.16 Load training data from 9 classes 73 4.17 The recognition testing process for human ID 001 74

4.18 The recognition testing process for human ID 037 75 4.19 Classification process 76

(15)

xiv

4.20 Hands image 78

4.21 Neural network architecture for hands classifier 79

4.22 Training system 79

4.23 Neural network architecture for left hand classifier and 80 its training system

4.24 The recognition testing result 81 4.25 Classification result 83 4.26 Processing speed for ANN recognition and classification 85

rate / frame

5.1 Classification system 88 5.2 Schematic diagram of proposed majority voting technique 89

for multiple classifiers system

5.3 Majority voting rule (MVR) 89 5.4 Human detection with right classification 91 5.5 Human detection with false classification 91 5.6 Image with partial occlusion 95

(16)

xv

LIST OF ABBREVIATIONS

RGB Red, green and blue GVF Gradient vector flow SVM Support vector machine 2D Two dimensional 3D Three dimensional HSV Hue, saturation and value HSI Hue, saturation and intensity RBF Radial basis function

CP Color predicate LUT Look-up table

ANN Artificial neural network MLP Multilayer Perceptron

MMLP Multiple multilayer perceptron Tr True positive rate

FPr False positive rate FNr False negative rate SCG Scale-conjugate gradient MVR Majority voting rule

(17)

1

CHAPTER ONE

INTRODUCTION

1.1 INTRODUCTION

Currently, along with community development, public safety is a very important issue that opens up various research topics including intelligent video surveillance to increase public safety. Most of the buildings in metropolitan cities are using video surveillance system in taking precautions especially in sensitive area. Moreover, these days, even individuals are seeking for a security system not only for their own safety but also for others such as detecting child abuse in kindergarten. Related research for human detection and classification has thus given the priority in security, video surveillance and privacy protection (Jadhav and Mane, 2009).

A robust method is needed in order to analyse the object of interest, to ensure that the system can detect, recognize and classify the object of interest. Detection means to detect or estimate the object of interest in the image while recognition is to determine the similarity of object of interest to the reference object. Classification basically is to classify an object of interest to specific category or class.

Detection and recognition of human in the images or video feeds are getting more important nowadays with the aims to identify that the object in the image is belong to human or not. There are several researches on automatics human detection and classification algorithms have been done targeting numerous applications such for video surveillance system, privacy protection, medical images analysis, information security, behaviour analysis, tracking and search and rescue.

(18)

2

Detection of human is a complicated task due to the human appearance such as clothing, articulation, shape of body and their pose and gesture. Although the position of human like standing and walking has reduced the constraints of human gesture but the possible variations are still large. In addition, the presence of multiple humans with a moving camera and the human subjects may be switched to each other makes the detection of human reliably becomes challenging task (Ru and Nevatia, 2006).

Since human detection from video is implemented on uncontrolled condition, objects appear in the video are often affected by occlusion. Non-occlusion environment still can be considered easy to implement in any system but how to cope with partial occlusion is not yet reliable enough to solve. However, human body with partial occlusion can be handled easily because if one of body part is occluded, the human still can be detected using another body parts.

Although a lot of works have been done to improve existing human detection system, there are still many issues arising among researchers to obtain a more convenient system at a lower cost with less computation time and can be further explore for the next research.

(19)

3

1.2 PROBLEM STATEMENT AND ITS SIGNIFICANCE

Human detection and classification is very important for various applications especially in security and safety area. Hence, this topic has been extensively researched. Currently, one of the main challenges on human detection is be able to detect and label the human body parts in order to track the pose or gesture of an individual. In addition, the human detection is always prone to occlusion by objects in the scene or due to lighting. Human detection with partial occlusion can provide solution for application such as search and rescue of a person in the crowded and complex environment. The important aspect when dealing with an occluded human body parts is human occlusion verification. The verification states whether the human body parts is occluded or not and at the same time properly identify which part of body is occluded. Generally, human body consists of various pose and shape. Detection of a human can be done according to the structure of human body such as head, face, neck, torso, limbs and etc. From literature reviewed, previous human detection systems normally detect the full human body but sometimes lead to results in failure when occlusion happens. However, none of the works focus on occlusion detection but only on body parts tracking. Hence, the work is proposed in detecting human by separate body parts classifiers which can detect human even when some parts of the body is under occlusions. In addition, this detection system can also be used to track human body parts for future research such as in pose and gesture tracking.

(20)

4

1.3 RESEARCH OBJECTIVES

The objectives of this research are:

1. To design an algorithm to segment human body from complex background scene using the fusion of skin colour detection algorithms.

2. To develop an intelligent human classification system using ANN classifier based on feature extraction method.

3. To optimize the classification algorithm using majority voting of the body parts classifiers suitable for classification under partial occlusion.

4. To evaluate the performance of the developed algorithm.

1.4 RESEARCH SCOPE

The research focuses on the development of detection and classification of human body parts. The algorithm development is done using Matlab software. Digital camera must be in a static position when capturing images in order for the system to work properly. The developed system is able to detect and classify the human body by extracting the features of human body parts individually. The human subject can be in a various position for detection stage, however, for classification stage, the image taken for the system database must be in frontal view with upright position. This research considers the body parts that can be detected using human skin color feature which cover only the face and hands. Since the skin color feature is used, it is important that proper lighting is available. For partial occlusion, the system uses the imaginary occlusion instead of using real occlusion.

(21)

5

1.5 RESEARCH METHODOLOGY

This research will be accomplished by developing and implementing the algorithm for detection and classification of human body parts which includes the following research stages:

1. Literature Review:

In order to make the research successful, advantage and disadvantage must be taken as a guideline to create another work which is not exactly the same, but could be much better than the original one. Information about the previous works related to this research are collected and examined thoroughly. Most of the source of the information is mainly obtained from conference paper, journals, printed material and internet and then can be proposed an approach that can be best to implement the required system. 2. Development of the Human Body Parts Detection Algorithm:

The main equipment for this research is computer and digital camera. The purpose of using a computer is for the implementation of the algorithm and a digital camera for evaluation and dataset construction. Skin color technique is used to extract skin color features of the human. Then, background rejection technique is implemented to eliminate noise or clutter object in order to smooth the image for further processing. Modifications of the existing technique are necessary to improve the efficiency of the proposed system. In spite of this, it is worth to note that the main part of this research is algorithm programming. Programming language or software mode that is going to be employed throughout the processing algorithm for detection and classification is based on Matlab 2012 software.

(22)

6

3. Development of the Human Body Parts Classification Algorithm:

System for classifying human is designed based on artificial intelligent system of neural network with the aim to identify that either the extracted body parts belongs to a particular class or not.

4. Development of Majority Voting of Human Body Parts Classifiers:

The voting technique is applied after the classification stage. From the voting of multiple classifiers, the majority will determine the final result. Apart from that, for partial occlusion, the proposed system is tested for recognizing human subject under partial occlusion.

5. Evaluation of the System Performance:

The system performance is tested and analysed in terms of accuracy and speed using the testing dataset.

(23)

7

Figure 1.1: Flowchart of research methodology Yes

No Start

Literature Review

Image Acquisition

Test and Evaluate the System Development of Human Body

Parts Detection Algorithm

Development of Human Body Parts Classification Algorithm

Satisfied?

End

Development of Human Body Parts Voting Classifier

(24)

8

1.6 DISSERTATION ORGANIZATION

This dissertation consists of six chapters and organized as follows:

Chapter one firstly describes the introduction and background of study. Apart from that, the problem statement, research objectives, research scopes and research methodology also included in this chapter.

Chapter two covers the literature review on computer vision and images processing, image acquisition, color image processing and the important parts on this chapter are related work on human detection, skin color human body detection, morphological operation, artificial neural network and voting classification technique.

Chapter three presents the concept of human detection in terms of human skin color, image acquisition, skin color segmentation, background rejection method and lastly detection of the human body parts.

Chapter four discusses on the classification starting from pre-processing of acquisition, feature extraction and artificial neural network for classification.

Chapter five explains the voting technique and evaluation of the system of partial occlusion problem.