A real-time vision-based touchless interaction of virtual heart using integration of motion gesture interaction methods

Full text

(1)A REAL-TIME VISION-BASED TOUCH LESS INTERACTION OF VIRTUAL HEART USING INTEGRATION OF MOTION GESTURE INTERACTION MEll-iODS. NG ING KYE. PERPUSTAKMIt •. _ 1~~'Wfr.Sm t.'M AYSIA SAM". A DISSERTATION SUBMITTED AS A PARTIAL REQUIREMENT TO OBTAIN DEGREE OF BACHELOR OF SCIENCE WITH HONOURS. MATHEMATICS WITH COMPUTER GRAPHICS PROGRAMME FACULlY OF SCIENCE AND NATURAL RESOURCES UNIVERSm MALAYSIA SABAH. 2014.

(2) ARKIB. UNIVERSITI MALAYSIA SABAH BORANG PENGESAHAN STATUS TESIS. A.. JUDUL:. REAL-1\1~~ VISlbN -2>A,StO. \1e~(\ \'. VSlNQ. IfjI~<aAA. roULHLFS>. 1IoN o t-. MOftJN~.....:@:uES=-.LTV%><--=,"----ll..L...:-:::..:..>o.<-u.w..-=--=-,--,--. METHOf)S. SESI PENGAJIAN:. SAYA:. "lOl'3/:;2.01L.\. (HURUF BESAR) Mengaku membenarkan tesis *(LPSMlSarjana/Doktor Falsafah) ini disimpan di Perpustakaan Universiti Malaysia Sabah dengan syaratsyarat kegunaan seperti berikut:1. 2. 3. 4.. Tesis adalah halemilik Universiti Malaysia Sabah. Perpustakaan Universiti Malaysia Sabah dibenarkan membuat salinan untuk tujuan pengajian sahaja. Perpustakaan dibenarkan membuat salinan tesis ini sebagai bahan pertukaran an tara institusi pengajian tinggi. Sila tandakan (I). '--_--'I. SULlT. '--_--'I. TERHAD. Z. (Mengandungi maklumat yang berdarjah keselamatan atau kepentingan Malaysia seperti yang termaktub di AKTA RAHSIA RASMI 1972) • (Mengandungi maklumat TERHAD yang telah ditentukan oleh organisasilbadan di mana Penyelidikan dijalankan). I TIDAK TERHAD. fERPUSTAKAAIt .. \~~lWfrS!TI ~';,\LAYSIA. SAMh. Disahk~lJRUlAIN BINTI ISMAil LIBRARIAN 4u.rwERSITI MALAYSIA SABAH. ~. (TA. DATAN~STAKA WAN). NAMA PENYELIA. 1).01 b I. Tarikh. Catatan :-. ~ 14. Tarikh:. ~:,. I" I a.o ('f. * Potong yang tidale berkenaan. -Jika tesis ini SULIT atau TERHAD. sila lampirkan surat daripada pihak berkuasa/organisasi berkenaan dengan menyatakan sekali sebab dan tempoh tesis ini perlu dlkelaskan sebagai SULIT dan TERHAD. *resis dimaksudkan sebagai tesis bagi [jazah Doktor Falsafah dan Sarjana Secara penyelidikan atau disertai bagi pengajian secara kerja kursus dan Laporan Projek Sarjana Muda (LPSM). PERPUSTAKAAN UMS. 111111111111111111111111 ·1000358292·.

(3) DECLARATION I affirm that this dissertation is of my own effort, except for the material referred to as cited in the reference section.. NG ING KYE (BS 11110414) 20 June 2014. ii.

(4) CERTIFIED BY Signature. SUPERVISOR. •. (Dr. Abdullah Bade). iii.

(5) ACKNOWLEDGEMENT First of all, I would like to express my appreciation to my supervisor, Dr. Abdullah Bade, who has given me guidance and providing academic supervision to me throughout the whole process of completing this thesis. Without Dr. Abdullah Bade's encouragement, timely feedback and constructive comments, I might not be able to complete this thesis smoothly.. In addition, I feel grateful for receiving kind assistance from all the lecturers from the course of Mathematics with Computer Graphics. They provide me valuable comments and suggestions whenever I have any problems in my study and in the preparation of my project and research as well. Besides that, I would like to thanks. my examiner, Madam Zaturrawiah Ali Omar for consistently providing me ideas and suggestion throughout the preparation process of my thesis.. Last but not least, I would also like to express my gratitude to my family and friends for their endless enormous support and encouragement in completing this thesis.. iv.

(6) ABSTRACT Motion gesture is one of the gesture recognition algorithms which provide touchless interaction within a system. The main objective of this project is to design a system that detects motion gesture of a virtual heart in medical imaging environment. This project encompasses a few gesture algorithms such as Gesture Estimation, Feedback and Gesture Recognition. The purpose of Gesture Estimation is to detect gestures and locates the screen coordinates. The Feedback is to produce response from the user regarding the motion data and the position of gestures. Gesture Recognition is to recognise the gestures and performs the basic transformation of the virtual heart. A Predefine Offset Screen Distance for Gesture Recognition is proposed. The flow of the algorithm can be divided into four steps. The first step is to calculate the offse~ line. Second, Recognition Threshold is declared to diminish the noise of the motion. Third, the range which is defined as left or right is used for gesture recognition. Lastly, the basic transformation such as rotation, translation and scale is executed if a data motion lies within the range. Four experiments are carried out to evaluate the prototype system. The experiments are Gesture Test, Lighting Test, Distance Test and Memory test. Besides, the proposed Predefine Offset Screen Distance is further analysed by measuring time complexity using Big-O Notation. The results showed that the proposed Predefine Offset Screen Distance is not very efficient and it has higher computational cost in time complexity with O(n 2 ). In this study, a prototype of Augmented Reality system that implements gesture motion is developed. The use of the gesture motion algorithm enables an interactive environment in medical imaging.. v.

(7) INTERAKSI MASA NYATA BERASASKAN VISI TANPA SENTUHAN TERHADAP JANTUNG MAYA MENGGUNAKAN KAEDAH INTEGRASIINTERAKSIISYARATPERGERAKAN ABSTRAK Gerakan isyarat adalah salah satu algoritma pengecaman isyarat yang boleh menghasilkan interaksi tanpa sentuhan dengan sistem. Tesis ini menitikberatkan pergerakan gerak isyarat dalam pengimejan perubatan. Objektif utama projek ini adalah merekabentuk satu sistem bagi mengesan pergerakan gerak isyarat pada model jantung maya. Projek ini akan mengabungkan tiga teknik iaitu Gerak Isyarat, Maklum Balas dan Pengecaman Gerak Isyarat Anggaran Gerak Isyarat adalah untuk mengesan gerak isyarat dan menempatkannya pada koordinat skrin. Selain itu, Maklum Balas adalah untuk menghasilkan tindak balas terhadap pengguna mengenai data pergerakan dan gerak isyarat kedudukan. Pengecaman gerak isyarat adalah untuk mengecam gerak isyarat dan melaksanakan transformasi asas bagi jantung maya. Dalam projek in;' satu teknik yang dikenali sebagai Teknik Penentuan Awal Jarak Skrin Offset untuk pengecaman gerak isyarat telah dicadangkan. Tatacara teknik ini secara amnya boleh dibahagikan kepada empat langkah. Langkah pertama adalah pengiraan imbangan garis. Langkah kedua adalah pengistiharan nilai ambang bagi mengurangkan gangguan daripada pergerakan. Langkah ketiga, nilai julat ditentukan di kanan atau di kiri skrin untuk pengecaman gerak isyarat. Akhir sekal;, jika data pergerakan terletak di dalam julat tersebut, transformas; asas seperti putaran, translas; dan pembesaran akan dilaksanakan. Empat eksperimen telah dija/ankan untuk menila; prototaip sistem. Eksperimen tersebut ada/ah ujian gerak kerja, ujian cahaya, ujian jarak dan ujian memori. Selain itu, ana/isis kompleksiti masa alogoritma Penentuan Awa/ Jarak Skrin Offset dikaji dengan menggunakan notasi Big-O. Hasil kajian menunjukkan bahawa algorithma yang dicadangkan adalah kurang cekap dan melibatkan kos yang tinggi dalam pengiraan kompleksiti masa iaitu O(n 2 ). Dalam kajian in;' sistem prototaip augmentasi nyata telah berjaya diimplementasikan dengan menggunakan pergerakan gerak isyarat telah dicipta.. vi.

(8) CONTENTS. Page DECLARATION. ii. CERTIFICATION. iii. AKNOWLEDGEMENT. iv. ABSTRACT. v. ABSTRAK. vi. CONTENTS. vii. UST OF TABLES. xii. UST OF AGURES. xiii. UST OF ABBREVIATIONS. xv. CHAPTER 1 INTROOUcnON. 1. 1.1. Overview. 1. 1.2. Motivation. 3. 1.3. Problem Background. 5. 1.4. Problem Statement. 8. 1.5. Aim. 9. 1.6. Objectives. 9. 1.7. Scope and Limitation. 9. 1.8. Justification. 10. 1.9. Thesis Organisation. 10. CHAPTER 2 LrnRATURE REVIEW. 12. 2.1. Introduction. 12. 2.2. Medical Imaging. 13. 2.2.1. 14. Radiographic Imaging. 2.2.2 Angiography or Contrast Agent. 15. 2.2.3. 16. Computed Tomography. 2.2.4 Nuclear Imaging 2.2.5 2.3. 16. Magnetic Resonance Imaging. 16. 2.2.6 Ultrasound. 17. Augmented Reality. 18 vii.

(9) 2.4. 2.5. 2.3.1 History. 18. Display System of Augmented Reality. 20. 2.4.1 Head-Mounted Display. 20. 2.4.2 Handheld Display. 21. 2.4.3. 21. Tracking System of Augmented Reality 2.5.1. 2.6. 2.7. Eye Glasses Display Sensor-Based Tracking Technique. 22. 2.5.3. 24. Hybrid Tracking Technique. User Interface of Augmented Reality. 24. 2.6.1 Vision-Based Interaction. 24. 2.6.2 Acoustic Interaction. 25. 2.6.3 Tangible/Haptic Interaction. 25. Gesture Recognition. 27. 2.7.1. Detection Layer. 27. a.. Colour. 27. b.. Shape. 28. c.. Motion. 28. 2.7.1 Tracking Layer. 29. 2.7.1. 2.9. 22. 2.5.2 Vision-Based Tracking Technique. a.. 2.8. 22. Template-based Tracking. 29. Recognition Layer. 29. a.. 29. Template Matching. Application Domain of Augmented Reality. 30. 2.8.1. 30. Medical Domain. 2.8.2 Industrial Domain. 30. 2.8.3. 31. Education Domain. 2.8.4 Game Domain. 31. 3D Modeling Technique. 31. 2.9.1. 32. Surface Model Technique. 2.9.2 Solid Model Technique. 33. 2.9.3. 33. Procedural Model Technique. 2.10. Processing. 34. 2.11. Discussion. 35. viii.

(10) 37. CHAPlER 3 MElHODOLOGY 3.1. Introduction. 37. 3.2. Project Framework. 38. 3.2.1. Phase I. 38. 3.2.2. Phase II. 38. 3.3. Architecture of System Design. 40. 3.3.1. Input/Output. 41. 3.3.2. Device API. 42. 3.3.3. Gesture Engine. 42. a.. Gesture Estimation. 42. b. c.. Feedback. 43. Gesture Recognition: Predefined Offset Screen. 44. Distance. 3.4. 3.3.4. Interaction with 3D Heart Model. 45. 3.3.5. Modeling of 3D Virtual Heart. 45. 47. Summary. CHAPTER 4 SYSTEM DESIGN AND INTERFACE. 48. 4.1. Introduction. 48. 4.2. Class Structure. 48. 4.2.1. Main Class and GSCapture Class. 50. 4.2.2. MultiMarker Class and objLoader Class. 4.2.1. g4p_controls and PFont Class. 4.2.1. interaction class and PImage. 51 51 51. 4.3. Sequence Diagram. 52. 4.4. Input Output Diagram. 53. 4.4.1. 4.4.2. Input Device. 54. a.. Camera. 54. b.. Mouse. 55. Device Tools. 55. a.. MCG Marker Generation. 5S. b.. New Marker Registration. 56. 4.4.3. Heart Model Data. 57. 4.4.4. Output. 58 ix.

(11) 4.5. Flow Chart. 59. 4.6. Menu. 60. 4.7. 4.6.1. Option Menu. 60. 4.6.2. Wording Display Menu. 61 62. Summary. CHAPTERS ANALYSIS OF MOTION INTERACTION TECHNIQUE. 63. AND COMPUTATIONAL OF SYSTEM 5.1. Introduction. 63. 5.2. Motion Gesture Creation. 63. 5.2.1. Motion Estimation. 64. 5.2.2. Feedback. 66. 5.2.3. Gesture Recognition: Predefine Offset Screen Distance. 67. a.. Rotation. 68. b.. Scale. 69. 5.3. 5.4. Experiment Set-up. 70. 5.3.1. Gesture Test. 70. 5.3.2. Ughting Test. 71. 5.3.3. Distance Test. 72. 5.3.4. Memory Test. 73. Experiment Results. 73. 5.3.1. Gesture Test. 74. a.. Translation. 74. b.. Rotation. 75. c.. Scale. 76. 5.3.2. Ughting Test. 77. 5.3.3. Distance Test. 79. 5.3.4. Memory Test. 80. 5.5. Analysis of Time Complexity Using Big-O Notation. 81. 5.6. Summary. 81. CHAPTER 6 CONCLUSION. 83. 6.1. Summary. 83. 6.2. Contributions. 84. x.

(12) 6.2.1 Construction of Heart Model and Register the Model. 84. into an Augmented Reality System. 6.2.2 The algorithm of Offset Screen Distance To Manipulate. 84. with the Virtual Heart. 6.2.3 The Use of Motion Gestures Create an Interactive. 85. Environment in Medical Imaging Application. 6.3. Future Work. 85. 6.3.1 Optimize the Motion Estimation Algorithm 6.3.2 Robust and Accurate of Motion Gesture Recognition. 85 85. Algorithm. 6.3.3 Alternative Methods for Motion Detection. 86. 6.3.4 Overcome the Factors that Affect the Detection of Marker 86 87. Reference. xi.

(13) LIST OF TABLES. Page. No. 1.1. Statistics of top ten diseases that cause death of Malaysian in. 4. 2006 5.1. The result of the gesture test on translation of 3D virtual heart. 75. 5.2. The result of the gesture test on rotation 3D virtual heart. 76. 5.3. The result of the gesture test on scale of 3D virtual heart. 77. xii.

(14) LIST OF FIGURES No.. Page. 1.1. A modem Computed Tomography (CT) (upper left), a MRI scanner (lower left) and a Mammogram Scanner (right). 3. 1.2. Overall process of Medical Augmented Reality System. 7. 1.3. 3D heart model that implement lie exactly in a virtual human Body. 7. 2.1. A generic block diagram of a typical modern electronic medical imaging system. 13. 2.2. Principle of an X-ray system with image intensifier. 15. 2.3. The output of using ultrasound on visualisation of liver. 17. 2.4. Milgram's Reality-Virtuality Continuum. 19. 2.5. An Optical See-Through HMD Conceptual Diagram. 21. 2.6. The Wrap920AR glasses provide immersive augmented reality for $1,995. 22. 2.7. The relationship between marker coordinates and the camera coordinates is estimated by image analysis. 23. 2.8. Haptics Demo from Magic Vision Lab, University of South AListralia. 26. 2.9. Tangible Tele-meeting System Overview. 26. 2.10. An Application Using Half-edge Topology Mesh to Create a Human Model. 32. 2.10. Rendering of Solid Table and Chair models in 3d Studio Max. 33. 2.11. Modeling of Rain Particle using Particle System in 3d Studio Max. 34. 3.1. Overall Framework Design of Real-time Vision-based Touchless Interaction of Heart Model Application System. 39. 3.2. Architecture of Proposed System Design. 40. 3.3. The proposed Predefined Offset Screen Distance. 44. 3.4. Summary of the proposed Predefined Offset Screen Distance. 45. 3.5. A cube with marked faces, edges and vertices and its boundary representation. 46. 4.1. Class Structure of the System. 49. 4.2. Display of main menu from a web camera. 50. 4.3. Sequence Diagram of the System. 52. 4.4. Input Output Diagram. S4. 4.5. MCG Marker Attached with Marker Paperboard. 5S. 4.6. MCG Marker Generated. S6 xiii.

(15) 4.7. Procedure for MCG Marker Generated. 56. 4.8. Function to load MCG marker into the system. 57. 4.9. 3D Virtual Heart Model in 3D Studio Max 2013. 57. 4.10. Data stored in Wavefront file (.obj) of Virtual Heart Model. 58. 4.11. Output of the System for Translation. 58. 4.12. Row Chart of the System. 59. 4.13. GUI Builder. 60. 4.14. Option Menu. 61. 4.15. Wording Display Menu. 62. 5.1. Pseudocode for Calculate the Difference of Frames by Euclidean Distance. 64. 5.2. Pseudocode for motion detection at certain position. 65. 5.3. Pseudocode for translation gesture recognition. 66. 5.4. Translation of Motion Gesture. 67. 5.5. Predefined Offset Screen Distance in this project. 68. 5.6. Pseudocode for rotation recognition algorithm. 68. 5.7. Rotation of Motion Gesture. 69. 5.8. Pseudocode for scale recognition algorithm. 69. 5.9. Scale of Motion Gesture (a) Reduce the size of virtual heart (b). 70. Enlarge of Virtual Heart 5.10. Experiment setup for lighting test. 72. 5.11. Experiment setup for distance test. 73. 5.12. Initial state and final state of gesture test on translation of 3D virtual heart. 74. 5.13. Initial state and final state of gesture test on rotation of 3D virtual Heart. 75. 5.14. Initial state and final state of gesture test on scale of 3D virtual Heart. 76. 5.15. Result of distance experiment at d (a) 15cm (b) 25cm (c) 35cm (d) 45cm (e) 55cm (f) 65cm (g) 75cm (h) 85cm. 5.16. Result of distance experiment at d (c) 45cm (d) 55cm (e) 65cm. 5.17. Memory test result using Java Profiler VisualVM 1.3.7. =. 78. =(a) 25cm (b) 35cm. 79. xiv. 80.

(16) LIST OF ABBREVIATIONS Augmented Reality. AR. ConnputedTonnography. CT. Digitally Substracted Angiography. DSA. Human Computer Interface. HC. Head-Mounted Device. HMD. Image-Guided Radiation Therapy. IGRT. Image-Guided Surgery. IGS. Magnetic Resonance Imaging. MRI. Virtual Environments. VE. xv.

(17) CHAPTER 1. INTRODUCTION. 1.1. Overview. Gesture is defined as a body movement that is used to express an idea or meaning, in which it generally involves the movement of hands and arms (Steel, 2012). For example, a "V" sign is a hand gesture symbolises the meaning of "peace". In computer science, gesture recognition plays a crucial role in interpreting the human gesture sign using mathematical algorithms. "Gesture" was first used in 1991 to replace a graphic display, keyboard and mouse (Christian & Berard, 2001). Gesture recognition is an approach chiefly classified into two types: two-dimensional (20) and three-dimensional (3~) methods. In 20 method, a hand can be represented by its geometric and non-geometric features. Geometric features comprise contour and fingertips (McAllister et aI., 2002). In contrast, non-geometric features include colour and texture (Oka et aI., 2002). Although 20 method solely detects gobal hand motion without determining the articulate motion on fingers, it can be applied in any real-time application because of its computational efficiency. On the other hand, 3D method is able to determine the articulate motion on fingers by identifying the location and position on the orientation of hand.. Gestures recognition is ubiquitous in the field of the research of interaction design. This is because the users usually prefer high accesssiblity devices and gestures also can provide this desired speed. With the aid of sensor on devices, motion gesture becomes more preferable among the gestures sign as every.

(18) movement or action has a connection with motion. Furthermore, motion gestures enable hand-free usage. For instance, Apple iPod implements the quick gesture with. its "shake and shuffle" motion. In this study, the recognition system is based on vision-based motion detection (camera-based). Camera-based motion data is calculated through the comparison between two frames: the previous frame and the current frame. The data is then used for gesture recognition to be applied in transformation of object.. This project begins with the study of touchless interaction based on gesture technology in medical imaging during the performance of image-based surgery. This research focuses on 20 motion gesture. In Image-Guided Surgery (IGS) practice, surgeon will rely on the computer-generated image that superimposes into the real world as a guide to conduct the operation. Hence, Augmented Reality (AR) will be used as our platform for this research as the user is partially immersed into the AR world. To allow real and virtual object to be aligned properly with each other, AR must be studied carefully by considering these three AR system characteristics:. i.. Procedure to combine real and virtual. ii. Interactivity iii. Procedure to register the 3D model (Azuma, 1997). The study is divided into several stages. Firstly, the set of requirements is defined for real-time interaction which- can be further segregated into two groups: practical and impractical. Practical includes detection, identification and tracking. In contrast, impractical describes the quality of the requirements. In this project, touch less interaction algorithm which is known as the motion gesture concentrates on three main steps: Motion Estimation, Feedback and Gesture Recognition. The goal of Motion Estimation is to detect the motion between two frames. Feedback is provided to test the user's. performance. The process of recognition is to. differentiate the gesture motion based on their task.. The current project aims to develop a real-time system which is implemented with efficient technique of motion gesture using vision-based for medical purpose. At the final stage of the study, the system will be evaluated by the performance of the. 2.

(19) proposed motion gesture algorithm. This project plays a vital role in supporting the development of a real-time medical visualization system.. 1.2. M"tiYation. Approximately one hundred years ago, human mankind survived without technology. Previously, medical study faced difficulty in improving the medical field as the education of medicine was merely based on practical knowledge and techniques. The invention of the first computer around fifty years ago has successfully enhanced the quality of the human's life. One of the major contributions of technology is medicine. Nowadays, the advancement of technology had reduced the death population compared to the past. ProfeSSionally qualified officers such as doctors, physiCians and specialists discover different vaccines. to save and improve the health of human. beings. With the help of technology, there is a drastic improvement in medical field where doctors can study and examine our body structure easily with the invention of technology. For example, Computed Tomography (Cl) scan shows details of every part of the human body, including bones, tissues, organs and blood vessels. Medical imaging also comprises Image-Guided Radiation Therapy (IGRT) and Open Magnetic Resonance Imaging (MRI). Figure 1.1 shows a modern Computed Tomography (Cl), a MRI scanner and a Mammogram Scanner. These devices are being used widely in medical field.. Figure 1.1 A modern Computed Tomography (Cl) (upper left), a MRI scanner (lower left) and a Mammogram Scanner (right) (Source: cr, MRI and Mammogram Scanner. 2013. In Mawar Renal Medical Centre.). 3.

(20) There are many diseases contribute to high mortality rate in human. Cancer, for instance, is a disease caused by the uncontrollable growth of damaged cells (tumour) which successfully affect the organs in human body including vital organs such as lung, brain and heart. A statistic shows that cancer disease is the third disease that cause death in Malaysia in 2006 (refer Table 1.1). Surgery is one of the oldest and most requisite treatments for cancer. The surgeons will extract the tumour out from the patient body during the operation. Therefore, surgeon needs to analyse the tumour thoroughly before conducting the operation procedure carefully in order to prevent surgery complication which will bring detrimental effect to the patient. With the improvement of technology, the eqUipment of medical can enhance the process with success without complication as well as minimise the operation time. Besides, surgeon can also provide remote surgery (telesurgery) in a long distance (Marescaux, 2001).. Table 1.1. Statistics of top ten diseases that cause death of Malaysian in 2006 (Source: Malaysia Ministry Health Department, 2006) Death Reason. No. Of Death. %. Septicemia. 6550. 16.87. Heart Attack. 5667. 15.70. Cancer. 4004. 10.59. Lung Infection. 2245. 5.81. Stroke. 3234. 8.49. Accident. 2099. 5.59. DS Malfunction. 1762. 4.47. Clearly, the life of the patients is fully depended on the surgeon's knowledge and experiences. So, the surgeon must be well-trained because each mistake made by the surgeon might cause the loss of the patient's life. In order to overcome the shortage, the developments in computer science especially in the realm of graphics are essential in helping the surgeons to perform surgery successfully. Image-Guided Surgery is used for the operation process where image is being generated so that it will indirectly lead the surgeon in operation procedure (Oertel. et a/., 2013). In. addition, a newbie surgeon too can utilize the visualisation technology to practise and horn their surgical skills in medical field. Even though the computer vision and. 4.

(21) human-computer interaction have gained improvement rapidly over the years, there are still rooms for improvement. Our approach is motivated by the fact that hand gestures perform more natural way of interaction than other interaction devices because every mankind use their hand to complete most of their task (Varona et a/.,. 2009). 1.3. Problem Background. As outlined in the previous section, there is an increase dependence of surgery on imaging system. In order to provide a real virtual environment as well as a successful operation through IGS, a few principals have to be considered. The general principals are the process of creating virtual images of the patients and visualisation of model, registration of model into AR, surgical simUlation and navigation.. The process of creating virtual images of the patients and visualisation of virtual model is the foremost important stage in IGS. Before the discovery of X-rays by Wilhelm Roentgen (1895), surgery procedure was carried out without the assistance of images. In other word, the surgeon could not directly observe the internal part of the patient's body. To overcome this limitation in medical field, a variety of imaging has been invented to improve problems such as Digitally Substracted Angiography (DSA), Computed Tomography (CT), Magnetic Resonance Imaging (MRI) and Ultrasound that have been invented to improve the problem. Similar to X-rays, the process undergoes scanning of the anatomy region of the patients and then uploads the anatomy region to a computer system. These devices have the same objective: to collect a set of 3D data from the patients in the end of the process. The most attractive part is that surgeon does not necessary need to visualize all the data which is collected during surgical procedure. For example, if a surgeon only wishes to view hard structure such as bones or the flow of arteries and veins through the heart, the raw data that is collected directly from the devices will have. \.0. undergo analysis and processing. The analYSis of the data can be visualised. and render into 3D model by using specialise software such as CHAI 3D and Netgen.. Nowadays, many researchers have been focusing on the way to obtain the most accurate data. CT scan is the most preferable device (Mischkowski et a/., 2007). 5.

(22) among the available scanning methods. Besides, some researchers also try to combine more than one scanning method by data fusion technique to produce more accurate data. Clearly, the fussed data is more informative than the original data collected from one source (Lawrence, 2004). The visualisation of the patient's anatomy structure of particular region is depended on the 3D data set obtain from the scanner. Therefore, researchers should take the challenge to improve the data acquire and storage to produce a better data set for clinical use in the future. Moreover, the efficiency to extract only data that the surgeon wishes to display is also another problem.. During the surgical procedure, a surgeon needs to read data of the patient frequently to update the medical report from time to time. Therefore, the application of AR technology is useful because of the three characteristic described in section 1.1 instead of using external monitor display. Considering AR system in surgical. lJi ocedure, it will create a more realistic surgical environment where part of the body from patient is being augmented into the real world such as the case of a patient who wishes to remove this gall bladder. An Imaging device is used to capture and collect data of the related organs (digestive system). The data are then used to model and render the virtual organs by graphics API. AR technology allows the virtual organs to be displayed and overlay with the real world objects. With the aid of display devices, the surgeon can allocate the gall bladder easily through the visualization of the virtual organs and the conduction of surgical procedure. The overall process of medical AR is illustrated in Figure 1.2 One of the key issue in AR approach is the accuracy of registering the real and virtual worlds (William et a/.,. 1996).. 6.

(23) Rool florid. V!rtual Iforld laltnrdo. BrD-Sf1ISOr. ARSysltm. Figure 1.2 Overall process of Medical Augmented Reality System (Source: MPEG Augmented Reality) In this context, AR is used to enable the surgeon to view "onto" and "into" the patient's anatomy within region of interest. For example, in cardiac surgery the display of virtual heart must be in the exact location of the patient's chest and be updated in time (refer Figure 1.3). Hence, the registration of the virtual images must lie exactly within the body. The accuracy of registration is important because a slight mislocation of images might cause difficulty for the surgeon to read the data. The system should react efficiently and be updated in time once the surgeon proceeds to the next step. Besides, the instruments that are being used by surgeon should be registered into the AR system too and alligned with the real world instruments. Tracking of the instruments and collision detection technique need to be applied for the manipulation of the surgery region.. Figure 1.3 3D heart model that implement lie exactly in a virtual human body. 7.

(24) After modelling and registration, a surgeon interacts with the virtual objects by using syrgical simulation. In fact, surgeons need to interact frequently with the medical system before or during surgery in order to review medical image and records. The physical contact of the computer and their peripheral devices such as keyboard, mouse and touch screen is difficult to be sterilised. This is because the devices have higher chance to transfer contaminated material such as bacteria. Generally, an assistant or nurse needs to stand by to operate the mouse and keyboard for the manipulation of the image during the operation procedure. This indirect. interaction. sometimes. will. cause. misunderstandings. because. of. communication problems (Johnson et aI., 2011). Another opportunity is the surgeon do it all by himself/herself. However, without assistance, the surgeon needs to move away from the patient to the located medical system for browsing and manipulation (Wachs et aI., 2006). Thus, direct and touchless interactions are being proposed because they are more intuitive and provide natural interactivity compared to contact-based input devices.. In general, there are two approaches in touchless interaction implementation. The first implementation uses vocal commands system whereas the latter uses gestures. The limitation of voice-based commands is that the system can hardly distingUish different people speaking in the same room. In other word, vocal commands are sensitive to the environment (Anon, 2013). The reason is, during a surgery, it is impossible to implement vocal commands for the system as there are plenty of people inside the surgical room, a surgeon and his assistants such as nurse and practical trainer. Somehow, gestures technology can overcome this by implementing more than one camera for recognition and provide body-worn sensors. However, there is also limitation for gesture where the paramount problem lies in the lower accuracy in difficult lighting, difficulty to differentiate between virtual object with background and occlusion by equipment or staff.. 1.4. Problem Statement. Sterility restriction makes touchless interaction an interesting and efficient solution for surgeons to interact with medical system directly. The main outcome lies on motion estimation and gestures recognition. An efficient and robust algorithm and. 8.

(25) REFERENCES Amine, A., Ghouali, S. & Mohammed Rziza. 2009. "Face Fetection in still Color Image Using Skin Color Information '; Proceedings ISCCSP2006, UC Berkley. Angel, E. Interadive Computer Graphics: A Top-Down Approach Using OpenGL. 2003. Third Edition, Pearson Education. Argyros, A. A. & Lourakis, M. I. A. 2004. Real-time tracking of multiple skin-colored objects with a possibly moving camera. In Proceedings European Conference on Computer Vision, pp. 368-379. Ashbrook, D. & Starner, T. 2010. MAGIC: A Motion Gesture Design Tool. Augmented Planet: A Different View On reality. www.augmentedplanet.com. Azuma, R. T. 1997. A Survey of Augmented Reality. In Presence of Teleoperators and virtual Environments 6, pp. 355-385. Barett, H. H. & Myers, K. J. 2003. Foundations of Image Science. Wiley, New York. Birk, H., Moeslund, T. B. & Madsen, C. B. 1997. Real-time recognition of hand alphabet gestures using principal component analysis. In Proceedings Scandinavian Conference on Image Analysis, Lappeenranta, Finland. Butz, A., Hollerer, T., Feiner, S., MacIntyre, B. & Beshers, C. 1999. Enveloping Users and Computer in a Collaborative 3D Augmented Reality. In !WAR '99, pp. 35-. 44. Card, S. K. & Moran, T. P. 1980. The Keystoke-Level Model for User Performance Time with Interactive Systems. Journals of Graphics and Image Processing, 23(7) :392-410. Christian von Hardenberg. & Berard, F. 2001. Bare-Hand Human-Computer Interaction, Proceedings of the 2001 Workshop on Perceptive User Interface. Comport, A. 1., Marchand, E., Pressigout, M. & Chaumette, F. 2006. "Realtime markrless tracking for augmented reality: virtual servoing Framework." IEEE Trans. on Visualization and Computer Graphics, 12(6):615-628. Cote, M., Payeur, P. & Comeau, G. 2006. Comparative Study of Adaptive Segmentation Techniques for Gesture Analysis in unconstrained Environments. International Workshop on Imaging Systems and Techniques, pp. 28-33. Cretu, A-M. 2003. 3D Object Modeling - Issues and Techniques. Preliminary Report.. 87.

(26) Mawar Renal Medical Centre. Retrieved Mar 30,2014, http://www.mawar.com.mY/hospital/index .ph p/ en/ medical-ervices/ diagnosticunit Douglas, C. N. Introduction to Medical Imaging. Edited by JF. web.eecs.umich.edu/""fessler/course/516/I/noll,516,intro-jf.ppt. Eck, U. & Sandor, C. 2013. HARP: A Framework for Visuo-Haptic Augmented Reality. Journal of Virtual Reality, pp. 16-20. Fessler, J. 2009. An Introduction to Medical Imaging Systems. 21:21. Filler, Aaron G. 2010. The History, Development and Impact of Computed Imaging in Neurological Diagnosis and Neurosurgery: CT, MRI, and 011. Internet Joumal of Neurosurgery, 1(1):5. Foley, J.D., van Dam, A., Feiner, S. K. & Hughes, J. F. 1996. Computer Graphics Principles and Practise. Adison-Wesley. Grifantini, K. 2010. Augmented Reality Goggles. Hall, R. 1991. Supporting Complexity and Conceptual Design in Modeling Tools, in State of the Art in Computer Graphics: Visualisation and Modeling. Haralick, R., Joo, H., Lee, C., Zhuang, X., Vaidya, V. & Kim, M. "Pose Estimation from Corresponding pointData", IEEE Trans. Systems/ Man and Cybernetics, 19(6):1426-1446. Henrysson, Billinghurst, M. & Ollila, M. 2005. Face to Face Collaborative AR on Mobile Phones. Proceendings of the 4h IEEEjACM International Symposium on Mixed and Augmented Reality (ISMAR 05), pp. 80-89. Herman, G. T. 2009. Fundamentals Of Computerized Tomography: Reconstruction From Projection. 2nd eddition. Springer. HIH:. Image. Hands-free Interaction in the Hospital. Retrieved April, 2013. http://www.healthcare.philips.com/pwc_hc/se_sv/about/assets/Docs/medica mundi/mm_voI49_no3/mm_49-3_Technology%20News.pdf.. Imaging Modalities.ppt. www.eng.tau.ac.il/ .... gannot/MI/Imaging .ppt. Inventor in the Reid Of Virtual Reality. http://www.mortonheilig.com/InventorVR.. 88.

(27) John, R., O'Hara, K., Sellen, A., Cousins, C. & Criminisi, A. 200 1: Explori~g the Potential for Touchless Interaction in Image-Guided Interventlonal Radiology. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; pp. 3323-3332. Johnson, T. 1986. Arteriograms, Venograms Are Angiogram Territory. Kato, H. & Billinghurst, M. 1999. Marker Tracking and HMD Calibration for a VideoBased Augmented Reality Conferencing System. Faculty of Information Sciences, Hiroshima City University Human Interface Technology Laboratory, University of Washington. Kato, H., Billighurst, M. & Poepyrev, I. 2000. ARToolkit Version 2.33: A Software Library for Augmented Reality Applications. Klepper, S. 2007. Augmented Reality - Display Systems. Klineker, G., Reicher, R. & Brugge, B. 2000. Distributed User Tracking Concepts for Augmented Reality Applications. ISAR'OO, pp. 37-44. Kooper, R. & Macintyre, Blair. 2001. Browsing the Real-World Wide Web: Maintaining Awareness of Virtual Information in an AR Information Space. International Journal of HG. Kwon, Yong.Moo. & Park, Jin.Woo. 2005. Tangible Tele-Meeting System with DVARPN (Augmented Reality Peripheral Network) Journal of Science & Technology, 39( 1) : 136-791. Lawrence, A. K. 2004. Sensor and data fusion: A Tool for Information Assessment and Decision Making. SPIE Press, pp. 51. Lawrenceville, G.A. 2011.Groundbreaking Augmented Reality- Based Reading Curriculum Launches Lee, K. 2012. Augmented Reality in Education and Training. Tech Trends, 56(2):1321. Lee, T. & Hollerer, T. 2007. Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertips Tracking. "Proc. 11th IEEE Int~1 Symp. Wearable Computers (ISWC ~07), pp. 83-90 Liang, Z-P. & Lauterber, P. C. 2000. Principles of Magnetic Resonance Imaging. SPIE, IEEE, New York. Lubrecht, A. 2012, Augmented Reality for Education "Digital Union", The Ohio State University.. 89.

(28) MacAllister, G., Mckenna, S. & Ricketts, 1. 2002. Hand Tracking for Behavior Understanding. Image Vision Comput, 20(12):827-840. Magnetic Resonance Imaging Information http://WW\N.magnetic-resonance.org/MagRes%20Chapters/21_02.htm Malaysia Ministry Health Department, 2006 WW\N.mohe.gov.my/ Marescaux, Jacques., Leroy, J. & Gagner, M.2001. A World First in TeleSurgery: The Surgical Act Crosses the Atlantic!. Press Conference, September, 2001, New York, United State of America. Merriam-Webster Online: Dictionary and Thesaurus. http://www.merriam-webster.com/medical/imaging . Milgram, P. & Kishino, F. 1994. Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Infonnation and System, pp. 1321-1329. Milgram, P., Takemura, H., Utsumi, Akira. & Kishino, Fumio. 1994. Journal of Telemanipulator and Telepresence Technologies, Vol. 2351.. Mirzaei, M. R., Ghorshi, S. & Mortazavi, M. 2012. Helping Deaf and Hard-of-hearing People by Combining Augmented Reality and Speech Technologies. Proceedings 9" IntI Conference Disability, Virtual Reality and Associated Technologies, pp. 149-158. Mischkowski, R.A., Zinser, MJ., Ritter, L., Neugebauer, J., Keeve, E. & Zoeller, J. E. 2007. Intraoperative Navigation in the Maxillofacial Area based on 3D Imaging Obtained By a Cone-beam Device. International Journal of Oral and Maxillofacial Surgery, 36:687-694 Mohammad Reza Mirzaei, Seyed Ghorshi & Mohammad Mortazavi. 2012. Audio-visual speech recognition techniques in augmented reality environments. Journal of the Visual Computer, pp. 1432-2315 Mortenson, M. E. 1997. Geometric Modeling, Second Edition, Wiley Computer Publishing. MPEG Augmente Reality http://wgll.sc29 .0rg/augmentedReality/?pageJd= 1154 Natterer, F. & Wubbeling, F. 2001. Mathematical Methods in Image reconstruction. Soc. Indust. Applied Math. Philadelphia. Newman, J., Ingram, D. & Hopper, A. 2001. Augmented Reality in a Wide Area Sentient Environment. In ISMAR '01, pp. 77-86. 90.

(29) Noelle, S. 2002. "Stereo augmentation of simulation results on a projection wall". Mixed and Augmented Reality. Proceedings of IsMAR 2002, pp. 271-322. Oertel, M. K., Jannin, P. & Collins, D. l. 2013. The State of the Art of Visualization in Mixed Reality Image Guided Surgery. Journal of Computerized Medical Imaging and Graphics, pp. 98-112. Oka, K., Sato, H. & Koike, H. 2002. Real-time Tracking of Multiple Fingertips and Gesture Recognition for Augmented Desk Interface System. Proceedings of IEEE International Conference on Automated Face and Gesture Recognition (FG/02J, 2002, pp. 411-416. Processing http://www .processing .org/ Pang, Y., Nee, A.Y.C., Kamal Youcef-Toumi, Ong, S.K. & Yuan, M.l. 2004. Assembly Design and Evaluation in an Augmented Reality Environment. Papagiannis, H. 2012. The Future Of Augmented Reality Is In Our Hands With Haptics & Touch Screens. Perrin, S. D., Ulano, A. & Hazelton, T. R. 2009. Revisiting The Pattern Approach To Interstitial Lung Disease On Chest Radiography Journal of Applied Radiology, Vol. 38. Perry, S. 2008. Wikitude: Android App With Augmented Reality: Mind Blowing. Rehm, M., Bee, N. & Andre, E. 2007. Wave Like an Egyptian - Accelerometer Based Gesture Recognition for Culture Specific Interactions. British Computer Society. Ruiz, J. & Li, Y. 2011. DoubleFlip: A Motion Gesture Delimiter for Mobile Interaction. Shapiro, V. 2002. Solid Modeling. In Handbook of Computer Aided Geometric Design, Elsevier. Siltanen, s. 2012. Theory and applications of marker-based augmented reality. Finland: Julkaisija Utgivare Publisher. Simms, Paul. 2009. History of Angiograms and Angiography. Singh, S. K., Chauhan, D. 5., Vatsa, M. & Singh, R. 2003. "A Robust Color Based Face Detection Algorithm," Journal of Science and Engineering, 6(4):227-234. Spitzer, M. B., Rensing, N. M., McClelland, R. & Aquilino, P. 1997. Eyeglass-Based Systems for Wearable Computing.. 91.

(30) Steel, M. 1997. New Oxford English-English-Malay Dictionary. Second Edition. Oxford Fajar, Selangor, 353ms. Varona, J., Antoni, Jaume-i-Capo., Gonzilez, J. & Perales, F. J. 2?09. To~ard Natural Interaction Through Visual Recognition of Body Gestures In Real-time. Journal. of Interacting with Computers, 21(1-2):3-10. Wachs, J., Stern, H., Edan, Y., Gillam, M., Feied, C., Smith, M. & Handler, J. 2006. A Real-time Hand Gesture Interface for Medical Visualization Applications. Application ofSoft-computing. Springer. Wagner, D. 2007. Handheld Augmented Reality. Wagner, D. & Schmalstieg, D. 2003. First Steps Towards Handheld Augmented Reality. Proceedings of the 7 h International Symposium on Wearable Computers, pp. 127 Webb, A. G. 2003. Introduction to Biomedical Imaging. IEEE, New York. William, A. H. & Khoi, N. 1996. Computer Vision-based Registration Techniques for Augmented Reality. Proceeding of Intelligent Robots and Computer Vision xv, Vol. 2904, pp. 538-548. You, S., Neumann, U. & Azuma, R. 1999. Hybrid Inertial and Vision Tracking for Augmented RealityRegistration. Zhou, F., Duh, .H. B. & ~illinghurst, ~. 2008. Trend in Augmented Reality Tracking, Interaction and ~Isplay: A Review of Ten Years of ISMAR. Proceedings of 7th Int'l Symp. on Mixed and Augmented Reality, pp. 193-202.. 92.

(31)

No results found