Development of Low Cost “GloveMAP” Based on Fingertip Bending Tracking Techniques for Virtual Interaction

(1)

Development of Low Cost “GloveMAP” Based on

Fingertip Bending Tracking Techniques for Virtual

Interaction

Nazrul H. ADNAN, Khairunizam WAN, Shahriman A.B., M. Hazwan ALI, M. Nasir Ayob and Azri A. AZIZ

Abstract—Thedevelopment of a new low cost GloveMAP based on fingertip bending tracking techniques for measuring the sign language is presented. Nowadays, the communication between two parties has always been a huge obstacle in the context to convey their conversation information.The project specification significant requirement is to facilitate people with verbal disabilities to communicate and interact with the normal people. This research study used the flexible bend sensor in order to obtain the output resistance values of each bending fingers of patient. The sensors are located at the upper fingers and physical of the GloveMAP made by the universal stretchable and covered with lycra fabric known for wearingcomfort. The evaluation of the prototype GloveMAP was designed to distinguish in deferent size of human hands to

Nazrul H. ADNAN is serving in

Advanced Intelligent Computing and Sustainability UNIMAP Research Group

School Of Mechatronic Engineering, Universiti Malaysia Perlis, 02600 Arau, Malaysia

c_{InstitutKemahiran MARA Beseri Perlis, Malaysia}

Khairunizam WAN is serving in

Shahriman A.B. is serving in

M. Hazwan ALI is serving in

c_{InstitutKemahiran MARA Beseri Perlis, Malaysia}

M. NasirAyob is serving in

Azri A. AZIZ is serving in

This work is upprted by fundamenta research grant scheme (FRGS) awarded by the Ministry of Higher Education to Universiti Malaysia Perlis

(FRGS 9003-00313).

(DOF) of fingers orientation. The main propose of developing the GloveMAP is to provide the natural input control of interaction in virtual circumstances, multimodal and tele-presence environments as an input devices provide as they can monitor the dexterity and flexibility characteristics of the human hand motion.

Index Term-- degree of freedom (DOF), flex sensor, fingers orientation, DataGlove, GloveMAP

I. INTRODUCTION

Research and development of the DataGlove for people who are disable such as deaf and mutein order to ensure that the facilities are able to be as a platform of communication to a normal person.Disabled people need special services to enable them to live independently. For example, the visually impaired, it is necessary to read and write Braille skills before they can use those skills to learn or get an education to live independently. Similarly, deaf people need the skills to understand and use of sign language before they can use it to communicate with others. At the present time, a lot of equipment has been designed to facilitate disabled people to communicate. For example, sign language translation process [1,2].Since the late seventies, many variations of instrumented glove design has been developed for facilities in various fields such as computer games, virtual reality, sign language recognition, medical, rehabilitation and robotics.

This work is upprted by fundamenta research grant scheme (FRGS) awarded by the Ministry of Higher Education to Universiti Malaysia Perlis (FRGS 9003-00313).

(2)

In this research paper, the development of a low cost GloveMAP is aimed to find the best method and system in order to produce the maximum usage in helping people with disabilities to communicate especially for Malaysian Federation of Deaf (HFD) Organization. The research is structured as following section: Section II will be addresses as the literature reviews of the suitability researches, research approaches, research applications and research problems of identifying the new development of the low cost GloveMAP. Section III more on the study of the methodologies of the system that used in the developing the GloveMAP. Section IV will be describing more on the experimental design method including the experimental setup and result. Finally, Section V will present the conclusions and proposing some possible future work.

II. LITERATURE REVIEWS

Flexible bend sensor is a unique device that changes the resistance when the sensor is bent or flexed. Many researchers used this flex sensor in order to realize their main application to become reality. The flex sensor is known as the potentiometer that consisting the coated substrate, such as plastic, that capable to change the entire amount into an electrical resistivity when the physical of the Flex sensor is in bending orientation. Saggio G. et al. [11] used 15 sensors in order to develop a biomedical glove that capable to measure the surgical gestures activities by classify and then evaluate the potential skill of surgeons. Oz et al. [12] used artificial neural networks (ANNs) to translate ASL words into English. The system uses a sensory glove called the Cyberglove™ and a Flock of Birds® 3-D motion tracker to extract the gesture features. The designed glove has 18 sensors, which measure the angle of fingers bending at various positions. Locations of sensors are placed with three sensors for the thumb, two sensors for each of four fingers, and four sensors between the fingers. Frequency of data collection could be up to 150 Hz. Fingers translation for virtual interaction is designed to expose the user in a virtual environment system in combination with various displays and interface device for the effects of immersion in a 3D interactive overview. Some of the researcher using an alternative extension device such as Wireless Bluetooth, Wii-Remote and Zigbee because of low power consumption and low rate [13,14,15].

According to the Y. Han [16] the development of the low cost DataGlove using the single channel video rather than multi-channel video in order to minimized the cost of the visual analyzer of the Visual Motion DataGlove System. The solution for the problem occurs is using the thin-bar-type optical indicators and a closed-form reconstruction algorithm. L.K. Simone et al. [17] and Nazrul et al. [18] also stated that the developed DataGlove is able to be applied or applicable to the disabled, through the rehabilitation process, especially to those with disabilities. Privileges of this invention is to enable the fingers may be used suffix Monitor developed to measure hand function in individuals with hand dysfunction were evaluated for the possibility, measurement repeatability and reliability of wireless transmission, and consumer acceptance.

III. METHODOLOGIES

The kinematic structure of the human hand is important in order to clarify some significant part of the structure to measure the movement of the human fingers. Distal, intermediate, and proximal phalanges are the Osteology of the phalanges of the hand as shown in Fig. 1. Each of these fingers is important in determining the actual position of the flex sensors that will be placed later. In this research, only the index and middle fingers are involved in determining the effectiveness of this study. Hand skeleton model is used in these works that consisting of 19 links imitate the corresponding human bones, and able to measure approximately 24 degrees of freedom (DOF), which is represented by the joint of two kinematic configurations considered in this model, one for the index and middle fingers. Therefore, the same kinematic configuration is used for the rest of the fingers (thumb, ring and small fingers).

In this situation, the direct kinematic equation is very much needed during the simulated virtual grip. This information will also be used to calculate the contact points of objects, grasping the route, and etc. The sensor on the surface of the finger is capable to detect the movement of a Metacarpophalangeal (MCP) and be able to measuring more than 10 DOF of the hand. With a so much variety of finger DOF, then the movement of the finger become highly constrained, so that it cannot make gestures indiscriminately.

Fig. 1. Anatomy of the hand [19]

For the examples, the movement of DOF for fingers cannot bend in the backwards direction too much. The motion constraints capabilities to simplifying the research and the development of the DataGlove can be typically classified [20];

A. Intrafinger Constraints

This is a common constraint on the movement of the same finger joints. Using this constraint, all types of movement particularly all fingers will be much easier to using the following equation:

(3)

Where is the stated as the fingers bending angle of the DIP joints and is called as the structure for the PIP joint bending angle.

and,

(2)

The proposed DataGlove is planned to be able to measure several DOF of human fingers. The study will be focusing on two fingers only. For the mathematical formulation as shown in Eq. 1 and Eq. 2, it‟s could be used as a basic reference to designing the DataGlove.

B. Voltage Divider

For low cost DataGlove development, basic resistive divider is chosen as test experiment for circuit connection between microcontroller and strain gauge sensor. Basic resistive divider consist only one resistor which are ideal for low cost development although, more experiment have to be prepared in order to entirely understand and exploit basic resistive circuit for strain gauge.

(3)

C. Gaussian Filter

Gaussian filter is the process of the filtering an impulse response from the Gaussian functions which is in methametically a Gaussian filter can change the input signal range by modifying using the Gaussian function. This transformation is normally known as the Weierstrass transform [23].

Fig. 2.Shape of a typical Gaussian filter [23]

Some of the equations that useable in the Gaussian filtering are Gaussian filtering using one dimensional sensor:

√ ₍₄₎

And, with the standard deviation as parameter

√

(5)

Meanwhile for two dimensional sensor, the Gaussian filtering

₍₆₎

Where:

x is the distance from the origin in the horizontal axis, y is the distance from the origin in the vertical axis, and σ is the standard deviation of the Gaussian distribution.

IV. DESIGN AND DEVELOPMENT OF DATAGLOVE: “GLOVEMAP”

Design and development process of implementinga GloveMAP would be more interesting when the designing priority of the glove is to the wearer comfort and effectiveness. The usage of microprocessors is also to be emphasized since it plays a key role inimplementing the successful operation of the system.

Fig. 3.Arduino Microprocessor [21]

Fig. 4.Flexible Bend Sensor 4.5” [22]

(4)

A. Interaction with Virtual Alphabet

The intended primary tool for numerical computing in this research and development is called Matlab. Effectiveness and accuracy in carrying out this research has become a priority. In fact, this software has become popular

to all researchers around the world. Matlab has become a stepping stone in the translation between the GloveMAP and

the system. Various methods can be used to ensure the success of this research including the Fuzzy Logic software. The application of Fuzzy Logic in this research is capable to analyze the entire signal received from strain gauge / flex sensor and then the signal will be sending via a serial communication throughout the Matlab software.After fuzzy decide which of those data is belong, Matlab will produce

database on V-Realms Builder setting.

(a)

(b)

Fig. 5. Commercial finger motion glove (a) GloveMAP (b) Fuzzy Logic of Eight Symbols of Sign Language

Matlab V-Realms Builder is important for virtual world capability. For the experimental setup, the ability to interpret the DataGlove for sign language into the virtual word language shows the potentials adaptability of the disabled usage. Figure 5a shows the GloveMap that consisting two flexible bending sensorsthat attached on the upper position of the DataGlove. Meanwhile, Fig.5b shows the sample design and development of low cost GloveMAP that capable to measure or translate 7 potentials sign language “Word A, B, C, D, F, K and number 8”. Each letter or word that is translated by the translator will

(5)

B. Experiments

GloveMAP experiment progress was carried out by using two strain gauge sensors, with intent to shows the reliability of its

with Matlab applications. Signal from strain gauge resistivity change will be evaluate so that signal that are compatible to be analyze and well-matched with Matlab application.

Fig. 6. Simulink Matlab basic setup for data reading

(a) (b)

(c)

Fig.7. GloveMap Index and Middle fingers bending activities waveform (Unfiltered Data) (a) 00_{(b) 45}0_{(c) 90}0

Signal produce by strain gauge are translate by microcontroller Arduino Uno into digital form and through serial communication transferring the data to Matlab software.This Simulink diagram is basic setting to establish connection between microcontroller and computer with Matlab software install. Arduino IO library must be pre-install in Matlab, while programming for serial communication is integrate into Arduino UNO. COM9 is location of Arduino connect to USB ports while gain block is calculation for conversion value from Arduino pin via serial port into

corresponding voltage references. Gain block is block that multiplies input data by a constant value (gain) as shown in Fig. 6.Figure 7 shows the unfiltered data from GloveMAPbending activities for Index and Middle finger at a range of 0º, 45º and 90º. Figure 7(a) obtain when 0º finger activities for index finger and middle finger is taken without using filter. Range of data for index finger is between (2.65V~2.7V) while middle finger is (2.8V~2.85V). Meanwhile for Fig.7(b) is obtaining when the index finger (sensor#1) and middle finger (sensor#2) are bend for 45º. The

Sensor #2

Noise

Noise Noise

Noise

Sensor #2

Sensor #1

Sensor #2

Sensor #1

(6)

result data for index finger range around (2.75V~2.8V) while middle finger (2.9V~2.95V). Finally, Fig. 7(c) shows that both index finger and middle finger are bending for 90º. Data

shows that index finger yield value (2.88V~2.9V) and middle finger yield data in range (3V~3.02V). Figure 9 shows the sample activities of index and middle fingers bending.

Fig. 8. Finger bending degree

Fig. 9. Bending activities (a) straighten fingers (b) bending of index finger (c) bending of middle finger (d) bending of both fingers

C. Gaussian Filter Experiment

Gaussian filter provide more stability in signal curve and are designed to give no overshoot to a step function input while minimizing the rise and fall time. Gaussian filter can significantly reduce noise from circuit resulting more smoothing curve in signal. Figure 10 shown the importancesample Input per symbol (N) and BT product on ensuring more stability curve signal. Figure 10(a) apply number Input per symbol (N) 10 and BT product of 0.5 show graphs with several noise unfiltered. Figure 10(b) apply number per symbol (N) 10 but BT product of 0.1 resulting much smoothen graph.Input sample per symbol (N) refer to a

positive integer representing the number of samples per symbol in the input signal. Input sample per symbol (N) used constant „N‟ as symbol for input sample assign to constant „N‟. „N‟ is constant which array of data, integer or bits are assign to it. BT product parameter representing bandwidth multiplied by symbol period (time) used to reduce bandwidth at the expense of increased intersymbol interference. Both BT product and Input sample per symbol (N) play great role in determining magnitude response (dB). Figure 12 shows the Gaussian filtering activities wheareas the filtering activities are much importance in developing the GloveMAP. Figure 13 shows the other direct feed from scope with each individual Gaussian setting to find most stable curve signal.Figure 13c

Sensor #1(Index)

(7)

shows that with increasing the value of input sample of symbol (N) to 50, the frequency of the magnitude response (dB) become much higher compared to the Fig. 13a.

At this stage, the waveform shown that the constant of finger bending output signal can be produced while there is no disturbance signal appeared.

(a)

(b)

Fig. 10. Gaussian filtering signal (a) Unfiltered Signal (b) Filtered Signal

Fig. 11. Gaussian Block set

Fig. 12. The relationship between BT Product with Input sample per symbol (N) and magnitude Responses.

D. Finger Signal with Gaussian Filtering

To ensure the smooth transmission of signals during the process of finger bending, the used of Gaussian Filtering is very important to add in the system. Figure 14 shows the

(8)

(a)

(b)

(c)

Fig. 13. Individual Gaussian setting (a) BT=0.5, N=10 (b) BT=0.1, N=10 (c) BT=0.1, N=50

E. Experimental Results and Discussion

For the experimental results, several activities of the fingers motion have been taken. Starting with a straight finger gestures until the fingers to bend by 45 degrees, finger bending process is intended to provide an appropriate signal to the microprocessor then the signal from the Arduino transmitting directly to the virtual program that can make it

(9)

Fig. 14. 900_{finger bending of GloveMAP using Gaussian Filtering}

(10)

Fig. 16. Interaction with Virtual Alphabet

V. CONCLUSIONS

The development of a low cost GloveMAP that capable to translate the sign word based on the gesture recognition of sign language is achieved. The functionality and development of the GloveMAP is capable to be market in price of 44-55 percentage lower compared to the other commercial DataGlove. The main application of this GloveMAP is to design the systems that are capable to translate the finger language or sign language in various DOF of fingers. The construction and production application of this system is using Matlab software in associate with the Fuzzy Logic translation system which is driven to be the perfect development of low cost “GloveMAP”. The translation result is this research is capable to do a translation accordingly to 6 words “A, B, C, D, F, and K” and also the number of “8”. However this research is only focused on two main fingers “index finger and middle finger”, in order to shows the potentiality and continuity for both fingers with the developed system. For the future research, the study will be emphasized into the other finger such as thumb, ring and little finger. Therefore, the planning for improving the design and development for these gloves will be more specialized on the data or to be more focused on the other 26 alphabet and 10 numbers in the basic sign language communication.

ACKNOWLEDGEMENT

Special thanks to all members of UNIMAP Advanced Intelligent Computing and Sustainability Research Group and School Of Mechatronics Engineering, Universiti Malaysia Perlis (UNIMAP), 02600 Arau, Perlis, Malaysia for providing the research equipment and internal foundation. This work is also supported by the fundamental research grant scheme

(FRGS) awarded by the Ministry of Higher Education to Universiti Malaysia Perlis (FRGS 9003-00313).

REFERENCES

[1] Stokoe, William C. "Dictionary of American Sign Language on Linguistic Principles", Linstok Press. ISBN 0-932130-01-1,1976 [2] Stokoe, William C. "Sign language structure: An outline of the visual

communication systems of the American deaf", Studies in linguistics: Occasional papers (No. 8). Buffalo: Dept. of Anthropology and Linguistics, University at Buffalo, 1960

[3] Ninja P Oess, Johann Wanek and Armin Curt, "Design and evaluation of a low-cost instrumented glove for hand function assessment", Journal of NeuroEngineering and Rehabilitation 2012, 9:2

[4] Oz, C., Leu, M.C. "Recognition of Finger Spelling of American Sign Language with Artificial Neural Network Using Position/Orientation Sensors and DataGlove", ISNN, 2005.

[5] Hernandez-Rebollar, J.L., Lindeman, R.W., Kyriakopoulos, N. A, "Multi-Class Pattern Recognition System for Practical Finger Spelling Translation", In: Fourth IEEE International Conference on Multimodal Interfaces (ICMI‟02), pp. 185–190. 2002 IEEE Computer Society Press, Los Alamitos

[6] Liang, R.H., Ouhyoung, M. "A Real-time Continuous Gesture Recognition System for Sign Language. Automatic Face and Gesture Recognition", IEEE, p. 558–567, 1998

[7] Lee, C.-S., Park, G.-T., Kim, J.-S. "Real-time Recognition System of Korean Sign Language based on Elementary Components", In: Proceedings of the Sixth IEEE International Conference on Fuzzy Systems, Spain, IEEE, Los Alamitos 1997.

[8] Takiguchi, T., Nakamura, S., and Shikano, K. "HMM-separation-based speech recognition for a distant moving speaker", IEEE Transactions on Speech and Audio Processing 9 (2) 2001 127–140.

[9] Kramer, J., Leifer, L.J. "A „Talking Glove‟ for Nonverbal Deaf Individual", Technical Report CDR TR 1990 0312, Center for Design Research, Stanford University.

[10] Lee, L.K., Kim, S., Choi, Y.K., Lee, M.H. "Recognition of hand gesture to human–computer interaction", IEEE, 2000, 2122–2177. [11] Saggio G., Bocchetti S., Pinto C.A., Orengo G. "Wireless DataGlove

(11)

[12] Oz, C., Leu, M.C. "American Sign Language word recognition with a sensory glove using artificial neural networks", Engineering Applications of Artificial Intelligence 24, 2011.1204–1213.

[13] J. C. Lee, "Hacking the Nintendo Wii Remote", IEEE Pervasive Computing, vol. 7, no. 3, pp. 39–45, 2008.

[14] 0. Portillo-Rodriguez, C.A. Avizzano, E. Sotgiu , S. Pabon, A. Frisoli, J. Ortiz and M. Bergamasco. "A wireless Bluetooth DataGlove based on a novel goniometric sensors", 16th IEEE International Conference on Robot & Human Interactive Communication, August 26 - 29, 2007. 1185-1190.

[15] Giovanni Saggio, Bharat Gupta, MatteoQuagliani, Mauro De Sanctis, Ernestina Cianca and Tommaso Rossi. "Power Efficient Wireless Connectivity of a wearable Data Glove", 978-1-4244-8132-3/10 2010 IEEE.

[16] Youngmo Han. "A Low-Cost Visual Motion DataGlove as an Input Device to Interpret Human Hand Gestures",. IEEE Transactions on Consumer Electronics, Vol. 56, No. 2, 2010.

[17] Shuai Jin, Yi Li and Weidong Chen, “A Novel DataGlove Calibration Method”, The 5th International Conference on Computer Science & Education Hefei, China. August 24–27, 2010.

[18] Nazrul H Adnan, Khairunizam WAN and Shahriman AB, "Accurate Measurement of the Force Sensor for Intermediate and Proximal Phalanges of Index Finger", International Journal of Computer Applications 45(15):59-65, 2012.

[19] Informationonhttp://www.yalemedicalgroup.org/stw/ Page.asp?PageID=STW023547

[20] Fahn C.-S., Sun H. "Development of a Fingertip Glove Equipped with Magnetic Tracking Sensors", Sensors. 10(2):1119-1140, 2010. [21] Information on http://www.arduino.cc/

[22] Information on http://www.cytron.com.my/usr_attachment/SN-FLX-01.pdf

[23] Wikipedia,“Weierstrasstransform”,