Interactive 3D visualization for tropical plant species

(1)Interactive 3D Visualization for Tropical Plant Species CY Ho Bioinformatics Programme Institute of Biological Sciences University of Malaya Kuala Lumpur, Malaysia kagami88@gmail.com. CH Tan Department of Mechanical Engineering Faculty of Engineering Kuala Lumpur, Malaysia cheehau86pl@gmail.com. HJ Yap Department of Mechanical Engineering Faculty of Engineering Kuala Lumpur, Malaysia hjyap737@um.edu.my. Siow-Wee Chang Bioinformatics Programme Institute of Biological Sciences University of Malaya Kuala Lumpur, Malaysia siowwee@um.edu.my. Abstract— Augmented Reality (AR) is gaining more popularity nowadays as AR enables the user to see, hear and feel the virtual elements in the real world. The advancement in AR has made an impressive remark in the field of game, education, tourism, exploration, and advertising. In this research, an interactive 3D visualization for tropical plant species is developed. It provides the user with better visualization of plant species through AR. It permits the user to view threedimensional rotatable models through a marker in a book, on a pamphlet or on signage. It enables the user to have a better idea of how the plants would look like in real life rather than plain twodimensional images. In order to track the 3D models, markers or image targets are developed to track various 3D plant models. The AR markers developed are rich in detail, good contrast and with no repetitive patterns to have the good tacking ability. This research consists of five main stages which are image acquisition, 3D model construction, marker design, integration of 3D model into marker, and graphical user interface (GUI) design. This application permits the user to access the 3D models of the topical plant species with the availability of the marker anywhere and anytime. In addition, it is portable and low-cost interactive learning material for school children as well as the public. Keywords—Augmented Reality, 3D Construction, Marker-based, Graphical User Interface, Tropical Plant Species I.. INTRODUCTION. Augmented Reality (AR) is becoming more popular nowadays because AR enables the user to see, hear and feel the virtual elements in the real world [13]. AR was originally a science-fiction concept and it. has become a science-based reality. The cost of developing AR applications is low and affordable [45]. AR is easily available such as on mobile apps and AR applications are easily developed with some opensource software [6]. In this research, an interactive 3D visualization tool for tropical plant species is proposed. It is an AR application that allows user to view three-dimensional rotatable models through a marker in a book, on a pamphlet or on signage. It enables the user to have a better idea of how the plants would look like in real life rather than plain two dimensional images. There are five stages in this research, first, image acquisition to acquire the images of the plants. Second, 3D model construction to build 3D models of the plants. Next, design the marker for tracking the 3D models in the real environment. Fourth, integrate the 3D models into the designed markers and lastly, design the graphical user interface for the application. Blender [7], Unity [8], Vuforia SDK [9] and ImageJ [10] were used to develop the proposed AR app. II.. METHODOLOGY. The dataset and methods used in this proposed study are described in the subsequent sections. A. DATASET This research utilized the myDAUN dataset which was collected in the University of Malaya, Kuala Lumpur, Malaysia [11-12]. myDAUN dataset consists of 88 tropical plant species which included 45 tropical shrub species and 43 tropical tree species. Only 74 species were selected to include in this research based on the quality of images available. The selected 74 species are listed in Table 1.. Proceedings of the Asia Pacific Industrial Engineering & Management Systems Conference 2019. M.

(2) TABLE 1 LIST OF THE TROPICAL PLANT SPECIES Species. Species. Species. Acacia auriculiformis. Duranta erecta. Mesua ferrea. Acalypha wilkesiana. Erythrina variegate. Mimusops elengi. Acalypha siamensis. Eucalyptus alba. Murraya paniculata. Adenanthera pavonina. Excoecaria cochinchinensis. Mussaenda erythrophylla. Albizia saman. Fagrea fragrans. Mussaenda philippica. Allamanda cathartica. Ficus microcarpa. Phyllanthus myrtifolius. Alstonia scholaris. Filicium decipiens. Plumeria rubra. Aquilaria malaccensis. Graptophyllum pictum. Polyalthia longifolia. Artocarpus integer. Hibiscus rosa-sinensis. Polyscias balfouriana. Barringtonia racemose. Hopea odorata. Pterocarpus indicus. Bauhinia blakeana. Hura crepitans. Saraca thaipingensis. Bougainvillea spectabilis. Hymenaea courbaril. Sauropus androgyrus. Bruntelsia calycina. Ixora javanica. Senna surattensis. Bucida molineti. Khaya senegalensis. Spathodea campanulata. Cassia fistula l.. Lagerstroemia floribunda. Sterculia foetida. Cinnomomum iners. Lagerstroemia indica. Strobilanthes crispa. Clinacanthus nutans. Lantana camara. Swietenia macrophylla. Cynometra malaccensis. Lawsonia inermia. Syzygium aqueum. Delonix regia. Loropetalum chinense. Syzygium campanulatum. Dillenia suffruticosa. Magnolia figo. Tabebuia rosea. Dipterocarpus grandiflorus. Malvaviscus arboreus. Terminalia catappa. Dracaena surculosa. Mangifera indica. Theobroma cacao. Dracaena reflexa. Melaleuca cajuputi. Tristaniopsis cf. whitiana. Dryobalanops aromatic. Manihot esculenta. Tabernaemontana divaricate. Melastoma malabathricum. Tibouchina urvilleana.

(3) B. METHODS The development of the proposed tool consists of five stages, which are image acquisition, 3D model construction, design marker, integrating the 3D model into marker and graphical user interface design. Figure 2 shows the flow chart for the development of interactive 3D visualization applications.. 2D images. Rendering process including photorealistic effects or non-photorealistic rendering. In this project, sapling and skin modifier methods were used to construct 3D models of plants [15-16]. (iii) Design Marker Marker represents the image that the Vuforia SDK is able to detect and track. There are some criteria in choosing a suitable marker, for example, it should be rich in details, good in contrast and no repetitive patterns in order to be recognized. Detection works when the SDK detects and tracks the natural features which are found in the marker and comparing these natural features against a known target resource database when the marker is in the field of view of device’s camera. (iv) Integrating 3D model into the marker The 3D model is aligned properly by dragging the 3D model along the x-axis, y-axis, and z-axis of the marker. When the marker is detected by using the developed tool’s camera, the 3D model associated with the target will appear. The scales used are x=0.5, y=0.5 and z=0.5 to position the 3D model exactly on top of the marker. (v) Graphical user interface (GUI) design GUI is a way that a user can interact with the application or a website. A good design GUI can improve user interaction experience. An AR app with interactive GUI is developed for the interactive 3D visualization for tropical plant species to ease user interaction. III. RESULT. Fig. 1. Flowchart for the development of Interactive 3D Visualization for Tropical Plant Species. (i) Image Acquisition Image acquisition plays an important role. The quality of the images determines the quality of the 3D model produced [13]. To ensure the quality of the images, some factors need to be considered such as the lighting condition, equipment, and shooting strategies. The optimum distance is fixed earlier to ensure that the object’s position in the middle of the photo and fills most of the frame. Restrict movement during image acquisition in order to obtain a highly focus quality image. Images such as whole tree, flower, bark, leaf, and fruit are taken for the purpose of material mapping in the 3D models. (ii) 3D Model Construction 3D model construction or 3D modeling is the process of creating a virtual 3D model of any physical object using a 3D modeling software. In this project, Blender [14] is used to construct the 3D models of tropical plants. 3D rendering is the computer graphics process of automatically converting 3D wireframe models into. An AR app for interactive 3D visualization for tropical plant species is developed and is named as myDAUN app. It consists of 74 tropical plant species from the myDAUN dataset which was collected in the University of Malaya, Kuala Lumpur, Malaysia. Images acquired through image acquisition were used for material mapping in 3D model construction. Next, 3D models of the selected 74 species were created by using either the sapling method or the skin modifier method. Sapling method is done by using Blender sapling tree generator add-on [16] whereas the skin modifier method [15] uses vertices and edges to create a skinned surface, using a per-vertex radius to better define the shape. Figure 2 shows the 3D models created using both methods. All the 3D models were created based on the photos taken..

(4) (a). (b). Fig. 2. 3D models created using (a) sapling method and (b) skin modifier method. Fig. 3. Examples of the leaf markers. Thirdly, markers were created by using the leaf image as well as the scientific name of each species. This is to create a good and unique detection for each of the. (a) Fig.4. A tropical tree is integrated with its respective marker. markers. Figure 3 shows the examples of some designed markers. (b).

(5) Fig.5. The developed myDAUN app. After the creation of markers, the fourth step is to integrate the 3D models into each of the designated markers. As shown in Figure 4(a), the arrow in green colour represents the y-axis, the blue colour represents z-axis and the red colour represents x-axis. By dragging the arrows to draw the asset into the desired direction. Figure 4(b) shows the final 3D model that integrated into marker. Lastly, myDAUN app was developed. The app contains three major tabs, which are “Camera”, “Gallery”, and “About us” as shown in Figure 5. When the user clicks on the camera tab and focuses it on the leaf marker, the 3D model which is associated with the marker will pop up. The user is able to interact with the 3D model by zooming in and out, in order to view the details of the 3D model. Figure 6 illustrates some examples of the 3D models by using the myDAUN app. Furthermore, the gallery tab shows the list of all 74 species with related information and images. Fig.6. Example of 3D models in myDAUN app. associated with each of the species i.e. whole tree, leaf, bark, flower, and fruit. IV. DISCUSSION The quality of the images acquired plays an important role to determine the quality of the 3D model construction. The sampled images are used to map the material of the 3D models in 3D construction. Tree Sapling and skin modifier methods are used to construct 3D models. Tree sapling is more difficult to model if compare to the skin modifier method. Furthermore, skin modifier method is able to create a more realistic tree than the sapling method. Markers created are able to detect based on the natural features of the markers. Scientific name and leaf shape are used to improve the uniqueness of the markers. Besides that, the 3D model placement is crucial in the step of integrating the 3D model into the marker. The 3D model has to be located in the center when.

(6) adjusting the position of the 3D model. This is due to the placement of the marker in the Unity editing scene will reflect how the models look like in real life. V. CONCLUSION In this research, an interactive 3D visualization for tropical plant species is developed using AR technology. Software such as Blender, Unity, Vuforia, and ImageJ are used to develop the AR application. With regard to that, myDAUN AR app was developed. This app allows the user to view the 3D model through a unique designed marker in the real-life environment. It enables the user to have a better view of how the plants would look like in real life rather than plain twodimensional images. This app permits the user to access the 3D models of the tropical plant species with the availability of the marker anywhere and anytime. In addition, it is also a low-cost interactive learning material for school kids as well as the public. The limitation of this research is it consists 74 tropical plant species only. Future works will include more tropical plant species such as flowery plants and medicinal plants. Besides that, more elements such as mini-game, virtual maps can be added to have a more comprehensive AR app.. [2] [3]. [4] [5]. [6]. [7] [8] [9] [10]. [11]. [12]. ACKNOWLEDGMENT This work was supported by the University of Malaya UMRG grant (RP038C-15AET) and RU Grant (ST013-2019). The funders had no role in study design, data collection & analysis, decision to publish, or preparation of the manuscript. REFERENCES [1]. Chi, H.-L., Kang, S.-C., & Wang, X., “Research trends and opportunities of augmented reality applications in architecture, engineering, and construction”, Automation in Construction, 2013, 33, pp.116-122.. [13]. [14]. [15]. [16]. Lee, K., “Augmented Reality in Education and Training”, TechTrends, 2012, 56(2), pp.13-21. Nicolau, S., Soler, L., Mutter, D., & Marescaux, J. (2011), “Augmented reality in laparoscopic surgical oncology”, Surgical Oncology, 2011, 20(3), pp.189-201. Azuma, R. T., “A survey of augmented reality. Presence: Teleoper”, Virtual Environment, 1997, 6(4), pp.355-385. Heilig, M., “Augmented Reality – The Past, The Present and The Future”, retrieved May 02, 2019, from https://www.interactiondesign.org/literature/article/augmented-reality-the-past-thepresent-and-the-future Moakley, B., “Introduction to Unity UI – Part 1”, Retrieved April 28, 2019, from https://www.raywenderlich.com/149464/introduction-tounity-ui-part-1 Pablo Vazquez & Francesco Siddi, Blender, Amsterdam, the Netherlands, 2019. Unity Technologies, Unity for Students, San Francisco, United States, 2019. Vuforia, Vuforia SDK for Android. The United States, 2019. Laboratory for Optical and Computational Instrumentation, ImageJ. University of Wisconsin-Madison, United States, 2019. Tan, J.W., Chang, S.W., Abdul-Kareem, S., Yap, H.J., Yong, K.T., “Deep Learning for Plant Species Classification using Leaf Vein Morphometric”, IEEE-ACM Transactions on Computational Biology and Bioinformatics, DOI:10.1109/TCBB.2018.2848653, 2019. Murat, M., Chang, S.W., Abu, A., Yap, H.J., Yong, K.T., “Automated classification of tropical shrub species: a hybrid of leaf shape and machine learning approach”, PeerJ, 2017, 5:e3792. Remondino, F., & El-Hakim, S., “Image-based 3D Modelling: A Review”, The Photogrammetric Record, 21(115), 2006, pp.269-291. Blender, Introduction, retrieved April 30, 2019, from https://docs.blender.org/manual/en/dev/render/cycles/introdu ction.html Blender, Skin Modifier, retrieved May 05, 2019, from https://docs.blender.org/manual/en/latest/modeling/modifiers/ generate/skin.html?highlight=skin%20modifier Lesterbanks, How to create animated trees with Blender Sapling. Retrieved May 10, 2019, from https://lesterbanks.com/2018/07/create-animated-treesblender-sapling/. R.

(7)

New documents

There are two main overarching models of triage: Police officers who have special mental health training often referred to as Crisis Intervention Teams – CIT, or a co-response model

Methods: Carers of people with long-term mental health difficulties n = 106 completed measures of adult attachment the Experiences in Close Relationships-Short Form questionnaire,

Conventional surface metrology methods such as, white light scanning interferometry WLSI based on coherence correlation interferometry CCI technique is mostly used as an offline

Abbreviations ASR: Acoustic startle reflex; CT: Childhood trauma; CTQ-SF: The Childhood Trauma Questionnaire-Short Form; CWAIS: The Wechsler Adult Intelligence Scale; EA: Emotional

The validity of Posttraumatic Stress Disorder Checklist for DSM-5 PCL-5 as screening instrument with Kurdish and Arab displaced populations living in the Kurdistan region of Iraq Hawkar

Comparing each two sets of the samples results, feature segmentation analysis and the pruning condition of 6 σ where Sq= 0.6 nm height and 15µm width have provided a clear evidence for

Abbreviations BMI: Body Mass Index; CBT: Cognitive Behavioural Therapy; CI: Confidence Interval; DSM-IV: Diagnostic and Statistical Manual of mental disorders, 4th edition; ES: Effect

The current study results also showed that among anxiety disorders, the highest prevalence belonged to generalized anxiety disorder 5.2% and obsessive-compulsive disorder 5.1% while the

Conclusion The segmentation analysis results and the theoretical model approach in this research paper, both appear to indicate that the major contributing factor for determining the

Keywords: Common mental disorders, Depression, Anxiety, Treatment seeking, Health service utilisation, Andersen behavioural model, Systematic review, Healthcare access, Barriers to