Designing a Gesture Based Device to
Recognize Sign Language Using Leap Motion
Controller
Shivangi S Nayak1, Vimal H Nayak2
PG Student, Electronics & Communication, SOCET, Ahmedabad, Gujarat, India1
Asst. Professor, Electronics & Communication, SOCET, Ahmedabad, Gujarat, India2
ABSTRACT: This paper presents the leap motion controller which is capable of detecting gestures made by hand, fingers through which the human with natural disability such as deafness and dumbness can communicate with the world. It is difficult to understand the sign language of such people for the normal people that is why the gesture based technology is a boon for them to forget their disabilities. Aim of this technology is to make the sign language through gesture of hand and finger which is detected by the leap motion controller. The leap motion controller is connected with an embedded device on which the further process of voice recognition will be performed. For getting the audio results, a microphone or speaker will be attached to the embedded device. The whole proposed system will be as small as possible so that it would become a small wearable device.
KEYWORDS: Leap motion controller, Gesture Technology
I. INTRODUCTION
When some people have the disability with hearing or speaking they cannot communicate through the world. As a result of this, they are differentiated among the society. However technology can only make the difference at most zero. Nowadays, gesture technology has made all the impossible things into possible as its algorithms are easy to identify body movements. The systems determine which device command a particular gesture represents and take the appropriate action.
Sign language is important for facilitating communication between hearing impaired and the rest of society. However, most vocal people do not understand sign language, hence, the need to develop system capable of translating sign language. The purpose of the system is to generate the voice data through the motion of hand so that whatever the hand movement is detected it has its own particular voice. The technology is exiting for detecting the gesture into images but the future work will be on developing a gesture to speak.[2][3].
II. RELATED WORK
classifier. In [5] it presents the dynamic early gesture recognition with support vector machine. Firstly, 3D motion trajectories are captured by the leap motion sensor, then the feature vectors of all gestures are extracted. Secondly, the experimental database is extended to achieve early recognition and the SVM model is utilized, while the test set is used to detect the performance.
III. LEAP MOTION CONTROLLER
A. Leap Motion Controller:
The Leap Motion controller is a small USB sensor device which is designed to be placed on a physical desktop, facing upward and capable of real time tracking of hands, fingers and finger-like tools in three-dimensional space. The device operates in an intimate proximity with high precision and tracking frame rate and reports discrete positions, gestures, and motion.[6][5]
B. Description :
The hardware consists two monochromatic IR cameras and three separate IR LED emitters which observe a roughly hemispherical area, to a distance of about 1 meter and have a field of view of about 150 degrees. The effective area of the Leap Motion Controller extends from approximately 25 to 600 millimeters above the device (1 inch to 2 feet). The LEDs generate pattern-less IR light and the cameras generate almost 300 frames per second of reflected data, which is then sent through a USB cable to the host computer, where the data is combined by the Leap Motion software with an internal model of the human hand to help cope with challenging tracking conditions. Detection and tracking work best when the controller has a clear, high-contrast view of an object’s outline.[6][5]
The Leap Motion system employs a right-handed Cartesian coordinate system. The origin is centered at the top of the Leap Motion Controller. The x- and z-axes lie in the horizontal plane, with the x-axis running parallel to the long edge of the device. The y-axis is vertical, with positive values increasing upwards (in contrast to the downward orientation of most computer graphics coordinate systems). The z-axis has positive values increasing toward the user.[6]
Figure-1 Cartesian Co-ordinate system of LMC
IV.PROCEDUREFORLEAPMOTIONCONTROLLER
tracks hands, fingers, and tools in its field of view, it provides updates as a set – or frame – of data. Each frame object representing a frame contains lists of tracked entities, such as hands, fingers, and tools, as well as recognized gestures and factors describing the overall motion in the scene. The Frame object is essentially the root of the Leap Motion data model. There are various leap motion APIs available such as fingers, fingers list ,hand, handlist,gesture,gesturelist,arm,device,controller,config,tools,toollist,pointtable,bone,listener,matrix,circle gesture, swip gesture, screen tap gesture, key tap gesture etc. To get tracking data from the API frames are used.[6]
Flow chart of work done:
Figure-2 Flow chart of workdone
Gestures
the frame gesture list. For continuous gestures, which occur over many frames, the Leap Motion software updates the gesture by adding a Gesture object having the same ID and updated properties in each subsequent frame.[6]
CircleGesture
Attributes:-center,normal,progress,radius,pointable,class_type Class- class Leap.CircleGesture
controller.enable_gesture(Leap.Gesture.TYPE_CIRCLE)
Figure-3 Circle gesture
Swipe Gesture
Attributes:- direction,pointable,position,speedstart_position,class_type Class- class Leap.SwipeGesture
controller.enable_gesture(Leap.Gesture.TYPE_SWIPE)Extends Gesture. The SwipeGesture class represents a swiping motion of a finger or tool
Figure -4 Swipe gesture
Screen tap gesture
Attributes:- Direction,position,pointable,class_type Class- class Leap.ScreentapGesture
controller.enable_gesture(Leap.Gesture.TYPE_SCREENTAP)
A screen tap gesture is recognized when the tip of a finger pokes forward and then springs back to approximately the original position, as if tapping a vertical screen. The tapping finger must pause briefly before beginning the tap
Figure -5 Screen tap gesture
Key Tap Gesture
controller.enable_gesture(Leap.Gesture.TYPE_KEYTAP)
The KeyTapGesture class represents a tapping gesture by a finger or tool.A key tap gesture is recognized when the tip of a finger rotates down toward the palm and then springs back to approximately the original position, as if tapping. The tapping finger must pause briefly before beginning the tap.
Figure -6 Key tap gesture
V. SIMULATION RESULTS
The simulation studies involve the different gestures recognition using leap motion controller.
Figure -7 Leap motion device connected with computer
Figure- 8 Initialization of Leap motion device with command
Hand Gesture is detected using leap motion controller through hand class and its subclass .
The following results shown in figure-10 are obtained by attaching leap motion controller and ardunio board to the host computer. The ardunio board is used to show the blinking LEDs according to the gesture patterns as explained in the Flow chart.
Figure -10 Results of performing different gestures
VI.CONCLUSION AND FUTURE WORK
The simulation results showed that the proposed work recognize the different types of gestures. It provides various features such as hand type, fingers, bones and their positions with respect to x, y, z co-ordinate system. As the performance of the proposed system is analyzed through leap motion controller and the attached ardunio board, the patterns according to the gestures are defined We have used leap motion controller which is a very small device The application-sign language which is implemented by using leap motion controller is so helpful to those who cannot speak or listen. The gesture based sign language helps them for a better communication with the world. The future work will be related to make the gestures speak.
REFERENCES
1. Zeyu Ding , Zexiong Zhang , Yanmei Chen ,Yen-Lun Chen, Xinyu W ,“A Real-time Dynamic Gesture Recognition Based on 3D Trajectories in Distinguishing Similar Gestures”, Information and Automation, 2015 IEEE International Conference on, 8-10 Aug. 2015,pages 250-255 2. M. Mohandes, S. Aliyu and M. Deriche, “Arabic Sign Language Recognition using the Leap Motion Controller” , Industrial Electronics (ISIE),
2014 IEEE 23rd International Symposium on, 1-4 June 2014,pages 960-965
3. A.S.Elons, Menna Ahmed, Hwaidaa Shedid and M.F.Tolba , “Arabic Sign Language Recognition Using Leap Motion Sensor”, Computer Engineering & Systems (ICCES), 2014 9th International Conference on, 22-23 Dec. 2014,ISBN 978-1-4799-6593-9;9781479965939,pages 368-373
4. Lee Garber, “Gestural Technology : Moving Interfaces in a New Direction”, Computer,volume 46,Issue 10,ISSN 0018-9162;00189162,Oct-13,pages 1-5
5. M. Mohandes, S. Aliyu, M. Deriche, gfhjg“Prototype Arabic Sign Language Recognition using Multi-Sensor Data Fusion of Two Leap Motion Controllers ”, Systems, Signals & Devices (SSD), 2015 12th International Multi-Conference on, 16-19 March 2015,pages 1-6
