FINGER POINTER BASED GESTURE SENSING SYSTEM

GE-International Journal of Engineering Research

Vol. 4, Issue 7, July 2016 IF- 4.721 ISSN: (2321-1717)

© Associated Asia Research Foundation (AARF) Publication

Website: www.aarf.asia Email : [email protected] , [email protected]

FINGER POINTER BASED GESTURE SENSING SYSTEM

SnehaShri .G, Rashmi .R, Priyanka .S

Department of Electronics and communication (BE student)

B.N.M Institute of technology

Bangalore, India.

ABSTRACT

The proposed project is to design and develop the input devices based on air gestures. Using gesture

sensing technology people can control number of applications which leads a new path way to human

machine interaction. The proposed method deals with the enhancement of human interaction with

computer using finger pointer as input. Finger pointer uses fingertip positions as parameters. There is

limitation on old style methods of some like using keyboard and mouse. It can be overcome with the

help of gesture sensing technology. The main purpose is to optimize the technology and make use of it

to make a path way to the human machine interface which would make the use of various digital

systems still easier. Image processing technique is very important to implement the gesture sensing

technology based applications. In this project, gesture sensing technology has been adapted to

develop 3 applications gesture based paint application, gesture based music player application, and

gesture based keyboard application. The entire design has been coded in MATLAB software to

develop the applications. The developed designs have been tested for different cases and are found to

1.1INTRODUCTION

With the wide use of computers in the society, human

computer interaction has become an increasingly

important in our daily life. It is widely believed that as

computing, processing, display techniques etc.

progress even further and the existing human

computer interaction using keyboards and mice may

become ineffective for the information flow.

Keyboards and mice have grown to be familiar but

they inherently limit the speed of interaction. Thus

gesture sensing technology is an attractive alternate

solution to increase the speed and ease of interaction

between humans and computers.

1.2 PRESENT TECHNIQUES

In computing, an input device is a peripheral used to

provide data and control signals to an information

processing system. Examples of input devices include

keyboards, mouse, scanners, digital cameras and

joysticks. A 'keyboard' is a human interface device

which is as a layout of buttons and helps to enter

characters and also perform some control functions.

Each button, or key, can be used to either input to a

computer, or to call upon a particular function of the

computer. They act as the main text entry interface for

most users. It is typewriter like device composed of a

matrix of switches. Pointing devices or mice are the

most commonly used input devices today. A pointing

device is any human interface device that allows a

user to input data to a computer. In the case of mouse

and touchpads, this is usually achieved by detecting

movement across a physical surface deflection. The

mouse also enables well-coordinated actions between

movement of the cursor and the selection of an object

by button clicks.

1.3 OBJECTIVE

To eliminate hardware input devices like mouse and

keyboard, achieve portability. To design and develop

the input devices based on finger pointer gestures. To

optimize the technology and make use of it to create a

new path way to the human machine interface. To

adapt gesture sensing technology for the simulation of

gesture based Keyboard, gesture based Paint, and

gesture based Music player control.

1.4 SCOPE AND RELEVANCE OF THE PROJECT

Nowadays, the society requires an advanced

technology to fulfil their requirements. Currently no

media or technology is used to connect directly from

physical to digital information. The gesture sensing

technology is the one which connects directly from

people to computer without involving any input

hardware devices. Gesture recognition is achieving

rapid market adoption, enabling humans and machines

to interface more easily in the home and at work. The

proposed technique is easy to learn and as well as to

implement gesture sensing technology. Using this

technology, we can control any number of

applications. This technology will ease our interaction

with the digital world.

1.5 BRIEF INTRODUCTION OF THE PROJECT

For the implementation of the three applications

gesture based keyboard application, gesture based

painting application and the gesture based music

player application, finger pointer technique is used. It

uses colored objects as the input devices. The entire

design has been coded in MATLAB software to

2. BLOCK DIAGRAM

To implement gesture sensing technology we make

use of image processing software such as MATLAB.

The codes are written in MATLAB as M- files. The

files accessed by the applications are added into the

same directory as the application codes.

The block diagram of the system is as shown.

Fig.1 Block diagram of the system design

a. Input device: Input device is a color substance that

is colored object such as markers used to detect the

gesture. It is developed for human machine interaction

using hand air gestures.

b. Laptop: A laptop is used to analyze the user inputs

and to perform appropriate actions to realize various

applications.

c. Camera: A laptop contains a camera which is used

to capture the input image and also for the

morphological processing of the image.

d. Gesture based Keyboard: It is a software

component that allows a user to enter characters with

the help of gestures instead of any manual devices,

such as typical keyboard. This also decreases the cost

of a typical keyboard.

e. Gesture based Paint: This application is similar to

typical paint application with user inputs being

gestures.

f. Gesture based Music player: Music player can be

controlled without using input hardware devices with

the help of gestures. Control is achieved through Play,

pause, resume, and stop, next, previous, shuffle and

repeat.

3. IMPLEMENTATION OF PROJECT 3.1 COLOR OBJECT DETECTION

The centroid also known as geometric center of

a plane figure is the arithmetic mean position of all the

points in the shape. The centroid of 3 different colors

is used to actuate the gesture sensing technology. The

colors can be chosen based on the environment.

Initially the color that is to be used has to be input by

the user. This is in order to avoid detection of other

colors present in the background. Each color is

assigned a number 1 for red, 2 for green and 3 for

blue. Any number of colors can be included by

altering the algorithm.

If an invalid choice is entered, a prompt will appear

saying “Enter a valid choice”. When the valid choice

is entered, the software enters the main loop where the

image is captured. The captured image is now

processed using image acquisition toolbox and

computer vision toolbox. The difference image is

obtained after subtraction of color component image

and gray image of the original RGB image. Noise is

removed using median filter. Thresholding is done to

convert the difference image to binary image.

Threshold values are different for different colors.

Pixel size (blob size) is defined in the program in

order to avoid detection of objects whose size is not in

the range of interest. The remaining white pixels in the

binary images are examined and the centroid is

obtained. The obtained centroid is used to implement

Fig.2 Block diagram of RGB color image

processing

3.2.1 IMPLEMENTATION OF GESTURE

BASED PAINT APPLICATION

The gesture based paint application is analogous to the

native paint application with options to choose

different colors, erase and save option. All of this can

be implemented without having to use a mouse or any

other similar hardware. The application recognizes the

object and based on position of object and paint is

performed. Another option to increase or decrease a

marker size is also given.

3.2.2 BLOCK DIAGRAM OF GESTURE BASED

PAINT APPLICATION:

Fig.3 block diagram for gesture based paint application

As shown the gestures are recognized with the help of

camera and software. The application software

determines these gestures and finally paint is

performed.

3.2.3 WORKING OF GESTURE BASED PAINT APPLICATION

The application recognizes the input object with a help

of a RGB camera. The RGB image captured is then

processed as discussed earlier to determine the

position of the centroid. These centroid values are

plotted using the plot function, which performs the paint function on a MATLAB „figure‟. The initial

color used to paint is blue. Later other colors such as

red, green, yellow, cyan, magenta, and black can be

selected. To choose colors, boundaries are marked and

when the centroid falls within these boundaries

particular color is selected. The boundary in this case

is of 160x80 pixels for each color and can be varied

according to programmers wish. The decrease or

increase in marker size in done for every input frame,

with a variation in size by +/- 3. The boundary size for

this function is 60x30 pixels each. The erase option is

also provided in which white paint is used to give the

erase effect. The save option lets you to save the

„figure‟ as a JPG file in the specified directory. When

the save option is triggered a pop up appears to enter

the file name and later the file is saved with the name

entered. The program continues to execute as long as

the save option is triggered. The partition for the paint

application is as shown. The color switching option

is provided to right, the save to the top left, the marker

size on top, and finally eraser on the bottom left. The

layout for the gesture based application is as shown.

3.2.4 MERITS OF GESTURE BASED PAINT APPLICATION

This reduces the need of mouse to perform paint. This

is more user friendly compared to using mouse. Since

options for color switching, erase option, save option

and changing the marker size is provided, it is very

similar to native paint application.

3.3.1 IMPLEMENTATION OF GESTURE

BASED MUSIC PLAYER APPLICATION

In this application we can control music like in the

natural music player or software. And this can be done

without the help of the mouse. The application

distinguishes the gestures based on its position of the

object and hence performs tasks similar to the music

applications we use currently. Choices are given to

play, pause, resume, stop, shuffle, repeat, next and

previous in the application.

3.3.2 BLOCK DIAGRAM OF GESTURE BASED MUSIC PLAYER APPLICATION

Fig.5 block diagram of gesture based music player application

As shown the gesture recognition is established with

the help of camera and software. The application

software resolves these gestures and controls the

music player.

3.3.3 WORKING OF GESTURE BASED MUSIC PLAYER APPLICATION

The application distinguishes the gestures with a help

of RGB camera , which is used in real time to capture

the images which are further processed to recognize

gestures and further action in case of valid gesture.

The position of object in terms of its x and y

coordinate is obtained. Depending on these values the

music can be played, paused, resumed, or stopped.

The functions are activated for 30 input frames to

provide a more time space avoid errors due to the

quick timing.

The application is developed for 10 music files which

are added into directory of the MATLAB program

being executed. The files are of mp3 format. A next

and previous alternative is given to change between

the songs. A shuffle option plays the songs in random

order every time and display appears whenever

switched on or off. When the object is placed within

the boundaries of shuffle button the shuffle is turned

on, placing the object again switches off the shuffle

and the original order is restored. Next and previous

options are also provided when shuffle is turned on.

Repeat option is also provided which continuously

plays the current song repeatedly, next and previous

options are disabled if repeat is on. It is turned on by

placing the object in the boundary specified, once,

when placed again it switches off. The GUI also

displays the album art of the currently playing song.

The distribution for the application is as shown. For

each of these options the boundary size is defined as

160x160 pixels. An exit option is also provided at the

top so as to exit the application. The size of this is

specified as 40x40 pixels. The layout of the gesture

based music player application is as shown in the

Fig.6 working of gesture based music player application

3.3.4 MERITS OF GESTURE BASED MUSIC

PLAYER APPLICATION

This reduces the need of mouse to select different

functions. This is more user friendly compared to

using mouse. Since options for play, pause, resume,

stop, previous, next, shuffle and repeat is provided it‟s

similar to default music player application.

3.4.1 IMPLEMENTATION OF GESTURE

BASED KEYBOARD APPLICATION

This application is used to enter alphanumeric

characters without using keyboard hardware. The

alphabets (lower case) and the numbers (0 - 9) can be

entered without the help of human interface hardware

such as keyboard or using virtual keyboard along with

the mouse. This is done by recognizing the input

object and its position and thereby deciding on the

characters and displaying it on the control panel.

3.4.2 BLOCK DIAGRAM OF GESTURE BASED KEYBOARD APPLICATION

Fig.7 block diagram of gesture based keyboard application

The gestures are recognized with the help of the

computer hardware and MATLAB software. Gestures

are decided and characters are entered into the control

panel.

3.4.2 WORKING OF GESTURE BASED

KEYBOARD APPLICATION

With the help of the camera the gesture are

recognized. The colored object‟s position is used to

enter the intended character. The characters can be

alphanumeric characters (lower case alphabets and

numbers from 0 – 9) and to remove last character

(backspace), enter a new line (enter), or give a tab

space (space).

The characters that need to be entered are displayed

only for each 20 frames. This is done as to provide a

sufficient time lapse to enter characters without

unintended error. The pixel boundary for each of them

is specified is 64x120. The backspace is implemented

using „/b‟ function, enter using „/n‟ and space using „/t‟. The characters are entered in the „command window‟ of MATLAB. An exit button is provided as

to exit after the use. The partition is as shown.

Fig.8 working of gesture based keyboard

4. ALGORITHMS AND FLOWCHARTS

4.1 ALGORITHM AND FLOWCHART FOR COLOUR OBJECT DETECTION

Algorithm and flowchart has always been an easier

method to understand functionality of any design. This

section illustrates algorithm and flowchart for colour

object detection using finger pointer based gesture

sensing and it is as follows.

4.1.1 ALGORITHM FOR COLOUR OBJECT DETECTION

Step 1: Initially enter the choice of the marker that is

used as the input for the gesture sensing applications.

If an invalid choice is entered, a prompt displaying

“enter a valid choice” will appear.

Step 2: The RGB image is captured and processed

using image acquisition toolbox and computer vision

toolbox.

Step 3: The captured RGB image is converted to grey

image.

Step4: Then captured RGB image is converted to

component color image, i.e., if the input marker used

is blue color, then RGB image is converted to blue

frame image.

Step5: The grey image and the blue frame are

subtracted. The result is a difference image with noise.

Step 6: In order to remove the noise, filter like median

filter is used.

Step7: After the removal of noise, we perform

thresholding operation. The thresholding value for

blue is 0.18. This value depends on the environment

and color chosen.

Step 8: Thresholding operation converts the image to

binary image. Blob size is limited between 800 and

3000 pixels.

Step 9: The white pixels in the image is scanned and

the centroid is found.

4.1.2 Flowchart for colour object detection

Fig.9 flow chart for colour object detection

4.2 ALGORITHM AND FLOWCHART FOR GESTURE BASED PAINT APPLICATION

The gesture based paint application is analogous to the

native paint application with options to choose

different colors and erase, save option. The algorithm

and flow chart is as shown.

4.2.1 ALGORITHM FOR GESTURE BASED PAINT APPLICATION

Step 1: camera is initialized; the image is captured and

processed.

Step 2: depending on the values centroid obtained the

Step 3: the functions performed can be normal

paint/colour selection/erase/marker size variation.

Step 4: when save option is triggered the program

saves the file and application exits.

4.2.2 FLOWCHART FOR GESTURE BASED PAINT APPLICATION

The flowchart for gesture based music player

application is as shown in Fig.10.

Fig.10 flow chart for gesture based paint application

4.3 ALGORITHM AND FLOWCHART FOR

GESTURE BASED MUSIC PLAYER

APPLICATION

In this application we can control music like in the

natural music player application or software. And this

can be done without the help of the mouse. The

application distinguishes the gestures based on its

position of the object and hence performs tasks similar

to the music applications we use currently. Choices

are given to play, pause, resume, stop, shuffle, repeat,

next and previous in the application.

4.3.1 ALGORITHM FOR GESTURE BASED MUSIC PLAYER APPLICATION

Step 1: camera is initialized; the image is captured and

processed.

Step 2: depending on the values centroid obtained the

functions are determined and executed.

Step 3: the functions performed can play, pause,

resume, stop, shuffle, repeat, next and previous.

Step 4: when exit option is triggered the application

exits.

4.3.2FLOWCHART FOR GESTURE BASED MUSIC PLAYER APPLICATION

The flowchart for gesture based music player

application is as shown in Fig.11.

Fig.11 flowchart for gesture based music player application

4.4 ALGORITHM AND FLOWCHART

GESTURE BASED KEYBOARD APPLICATION

This application is used to enter alphanumeric

characters without using keyboard hardware. The

alphabets (lower case) and the numbers (0 - 9) can be

entered without the help of human interface hardware

the mouse. This is done by recognizing the input

object and its position and thereby deciding on the

characters and displaying it on the control panel.

4.4.1 ALGORITHM FOR GESTURE BASED KEYBOARD APPLICATION

Step 1: camera is initialized; the image is captured and

processed.

Step 2: depending on the values centroid obtained the

functions are determined and executed.

Step 3: the functions performed can entering the

alphabets, numbers, back space, enter and space.

Step 4: when exit option is triggered the application

exits.

4.4.2 FLOWCHART FOR GESTURE BASED KEYBOARD APPLICATION

The flowchart for gesture based music player

application is as shown in Fig.12.

Fig.12 flowchart of gesture based keyboard application

5. RESULTS AND DISCUSSION

This discusses the results of proposed method and

different simulated results that are obtained in

MATLAB software.

5.1 SIMULATION RESULTS FOR GESTURE BASED PAINTING APPLICATION

The entire design has been coded in MATLAB and

simulated results are obtained for painting application.

Once the “centroid‟ of input device is found out then a

dot will be created to draw something in painting

window with respect to the hand movement is as

shown in Fig.13.

Fig.13 overall view of gesture based paint application

5.2 SIMULATION RESULTS FOR GESTURE BASED KEYBOARD APPLICATION

The entire design has been coded in MATLAB and

simulated results are obtained for painting application.

The centroid is found out using the image processing

techniques and the characters are entered. The

simulation result is shown in Fig.14

Fig.14 choosing characters in gesture based keyboard application

5.3 SIMULATION RESULTS FOR GESTURE BASED MUSIC PLAYER APPLICATION

The entire design has been coded in MATLAB and

simulated results are obtained for music player

application. Figure 6.1 and 6.2 shows the painting

output. Once the “centroid‟ is found then a function is

performed appropriately with respect to the hand

movement is as shown.

Fig.15 choosing options in gesture based music player application

6. CONCLUSION AND FUTURE SCOPE

6.1 CONCLUSION

The applications were coded in MATLAB and

executed and simulated for various test cases. The

application was found to be working satisfactorily.

The applications gesture based keyboard, gesture

based music player, and gesture based paint

application were implemented results were obtained

and eased the interaction. The application was found

to remove the hardware such as mouse, keyboard, and

touch pad thus implementing gesture sensing

technology‟s merits.

6.2 ADVANTAGES

Since we are making use of gestures, it controls will

ease our interaction with devices. Pressing a key or

ticking a knob can be avoided. Gesture sensing can

also be combined with other advanced user interface

technologies such as voice commands and face

recognition. This creates a richer user experience. The

hardware like keyboard, mouse, joystick, etc. can be

avoided. Thus a simple webcam and a color object is

sufficient for the human computer interaction. Sensors

or any other hardware devices are not needed for the

image acquisition.

6.3 FUTURE SCOPE

In music player application we can implement volume

up and down and automatic recognition of music files

and a scroll bar. We can also extract the embedded

information such title, genre, album info, album art

and other tags and display it. Options to show all the

songs, add playlists, equalizer settings and many other

options can be provided. In paint applications options

for many other colors and also to add custom colors

and other object such as pencil, other brushes, fill

color etc. can be provided. In paint we can implement

in a way the paint is not inverted with respect to the

input gestures. In keyboard options to enter all the

Overall improvement can be made to detect gestures

without the help of colored markers using normal

finger color as input. Finally gesture can be

implemented in various other applications and can be

combined with voice recognition too.

7. ACNOWLEDGEMENT

We would like to thank Dr. P. A. Vijaya, Professor

and Head of Department of Electronics and

Communication Engineering, BNMIT, Smt.

Chellamma K, Assistant Professor, Department of

Electronics and Communication Engineering, BNMIT

who have shared thier opinions and thoughts which

helped us in completing our project successfully.

8. REFERENCES

[1] Mutthuraju K.S, P.A. Vijaya, “Design and

Implementation of Rover controller and Music player

control using sixth sense technology”, International

Journal of Engineering Research and Technology

(IJERT), Volume 4, Issue 5, May-2015.

[2] Mr. Ashish Parmeshwar Tiwari, Miss. Tanvi

Praful Khandhedia,Prof. Umesh W. Kaware, “Sixth

Sense Technology”, IJREAT International Journal of Research in Engineering & Advanced Technology,

Volume 3, Issue 2, April-May, 2015

[3] Gaurav Subhash Nikam, Rushikesh Prataprao

Bhoite, Nilesh Anil Jagtap, “The Sixth Sense Technology”, International Journal of New

Innovations in Engineering and Technology, Volume

3 Issue 1 – March 2015

[4] G. Mallikarjuna Rao, Dr.Ch.Satyanarayana,

“Object Tracking System Using Approximate Median

Filter, Kalman Filter and Dynamic Template

Matching”, I.J. Intelligent Systems and Applications,

2014, 05, 83-89

[5] Keerthi Lingam, Sindhuja Raghupatruni, “Sixth

Sense Enabled Campus – Possibilities and

Challenges,” IJCA, Vol. 75,-No 8, August 2013.

[6] Vladimir I. Pavlovic, Rajeev Sharma, and Thomas S. Huang, “Visual Interpretation of Hand Gestures for

Human-Computer Interaction: A Review” IEEE

transactions on pattern analysis and machine