Augmented reality system development : from simple to advance

(1)

7 AUGMENTED REALITY THEORY AND

APPLICATIONS

Ajunewanis Ismail

Zakiah Noh

INTRODUCTION

he chapter covers the state-of-the-art in the field of Augmented Reality (AR), in which 3D virtual objects are integrated into a 3D real environment in real-time. It describes the theory of Augmented Reality and explores AR that can be applied in medical, manufacturing, visualization, path planning, entertainment, military and so on. This chapter will provide a starting point for anyone that is interested in doing research or using Augmented Reality.

This chapter will discuss about the theory of AR. Azuma (1997) mentioned three criteria that have to be fulfilled for a system to be classified as an AR system: they combine the real and the virtual, they are supposed should interactive in real-time in other meaning that the user can interact with the system and get response from it without delay, and they are registered and aligned in three dimensions. This section also will discuss about technologies in AR in order to develop AR system.

In the next section, we will describe about AR applications. AR applications can be found in diverse domains such as medical, education, military applications, entertainment and infotainment, technical support and industry applications and so on. However, this chapter is focusing on description of AR applications in medical, annotation and visualization, education, manufacturing and repair, robot path planning, entertainment and games.

(2)

Augmented Reality Theory and Applications 89

AUGMENTED REALITY’S THEORY

Definition

AR is an environment that supported by real and virtual objects in real time representation. Azuma (1997) defined AR as “an environment that includes both virtual reality and real-world elements. For instance, an AR user might wear translucent goggles; through these, he could see the real world, as well as computer generated images projected on top of that world”.

The goal of AR system is to add information and improve the user’s view to a real environment. This definition of AR system is not limited to display technologies only for instance Head Mounted Display (HMD) but it can be potentially applied to the visual common sense including hearing, smell and touch.

Milgram and Kishino (1994) elaborated the virtuality continuum of real environment to virtual reality in which AR is one part of Mixed Reality. The environment of Augmented Virtuality and Virtual Environment are virtual whereas the environment of AR is real.

(3)

The differences between VE and AR are based on immersion level. AR can give user an immersive experience however user still can see and feel the presence of real world. Some researchers describe AR with several features; i) combines real and virtual, ii) interactive in real-time and iii) registered in 3D. Registration is an accurate alignment of real and virtual object. The illusion between the virtual and real objects exist in the real space is severely compromising without accurate registration. The application of AR can be applied in medical, manufacturing, visualization, path planning, entertainment, education and others.

Technology of AR

To develop AR system, several displays technology is required to accomplish the combination of real and virtual environment. These technologies include optical see-through, video see-through, Virtual Retinal Systems and Monitor based AR.

Optical See-Through HMD

(4)

Figure 7.2 Conceptual diagram of an optical see through HMD (Azuma 1997)

(5)

Video See Through

Video see-through HMDs are able to give user a real world sight by combining a closed-view HMD with one or two head-mounted video cameras, due to this mixture give user a view of real world and virtual world in real-time through the monitors in front of the user's eyes in the closed-view HMD. Figure 7.4 shows a conceptual diagram of a video see-through HMD. Figure 7.5 shows a video see-through HMD. Video composition can be done using chroma-key or depth mapping (Silva, Oliveira & Giraldi 2003).

(6)

Figure 7.5 Video see-through HMD (Silva, Oliveira & Giraldi 2003)

Virtual Retinal Systems

Virtual Retinal Systems aim to produce a full color, wide field-of-view, high resolution, high brightness and low cost virtual display. This technology can be used in wide range of applications from head-mounted displays for military or aerospace applications to medical purposes.

(7)

Figure 7.6 Virtual Retinal System HMD (Silva, Oliveira & Giraldi 2003)

Monitor Based AR

(8)

Figure 7.7 Build of Monitor-based system (Azuma 1997)

(9)

AUGMENTED REALITY APPLICATIONS Medical

In medical applications, computer will generate 3D data. These data will be superimposed onto the surgeon’s view of the patient. Then, it provides a spatial data of the organ which belongs to the patient’s body or simply “X-ray” vision. Doctors have been implementing AR in surgery as a visualization and training aid. Non-invasive sensors such as Magnetic Resonance Imaging (MRI), Computed Tomography scan (CT), or ultrasound imaging has been used for the purpose of colleting 3D data of the patient. Those data are being rendered and combined with a view of the real patient in real-time. In context of AR system, those 3D data are combined with the view of real patient when the virtual view of 3D data is rendered in real-time. Therefore, doctors can easily interact with the AR system and get response without delay. As a result, this would give a doctor "X-ray vision" inside the patient. This is an efficient way to reduce high risks of operation during minimally-invasive surgery. A problem with minimally-minimally-invasive techniques is doctor has a lack of ability to see inside the patient, so it is making surgery more complicated. Thereby with AR technology, it might provide an internal view without the need for larger incisions.

(10)

Figure 7.9 Virtual pregnant patient (State et al. 1996)

(11)

Figure 7.10 These images show Liver-Surgery planning using AR. (a) Process 3D refinement that allows changing of an incorrectly segmented liver. (b) Input and output devices tracking process. (c) Simulation of non-standard tumor resection process. (d) Simulated calculation for quantitative analysis (Bornik et al. 2003)

(12)

LiverPlanner, has been introduced by Bornik et al. (2003). Figure 7.11 shows the images of planning process for LiverPlanner. They use AR technology to manage clinical process of planning liver tumor resections so that it becomes more simple and easier.

Education

A potential and challenges of using collaborative AR within the context of immersive virtual learning environments becomes a high motivation to propose AR in education. For example, the experiences made during the development of a collaborative AR applications specifically designed for mathematics and geometry education called Construct3D, produced by Kaufmann (2000).

(13)

Figure 7.11 A tutor assists a student while working on constructing geometry model (Kaufmann 2000)

Figure 7.12 Collaborative works of students within the

Augmented Reality application in constructing 3D object (Kaufmann 2000)

(14)

Augmented Reality Theory and Applications 101

Annotation and Visualization

AR can be used for annotation and visualization tasks, which is efficient to annotate objects and environments with information either private or public. AR annotation system will draw the information given to view the virtual slide to users. Annotation that uses AR will combine the real and the virtual, the interactive annotation system AR in real-time means that the user can interact with the system and get response from it without delay. According to Fitzmaurice (1993), when the user walks around the library, the information of library shelves will be provided by a hand-held display.

(15)

Figure 7.14 Annotation on engine model (Rose 1995)

According to Rose (1995), at the European Computer-Industry Research Centre (ECRC), when users point at parts of an engine model and AR system plays their role to display the name of the part once it is being pointed. Figure 7.14 shows where the user points at an engine model and the information appears as label of the part that being pointed.

(16)

Manufacturing and Repair

In industry, AR can be applied in various areas like repairing and maintenance of complex engines. For example, in the repairing and maintenance of complex engines, AR system can provide labels that aid mechanics in identifying engine components. Additional data such as maintenance reports, schematic manufacturer’s specifications, and repair procedures might be retrieved and displayed next to the specified component which observed in the real environment.

AR applications in manufacturing and repairing could be found in diverse domains, such as the assembly, maintenance, and repair of multipart machines. Instructions might be easier to comprehend, compared to the manuals instructions that contain texts and pictures. It seems to be more difficult. But, AR produces a virtual instruction in 3D drawn, showing the task need to be done and how to be handled in easier way, perfect in step-by-step. These virtual instructions can be animated and thus, providing the directions even more clear

(17)

Several research projects have confirmed prototypes in this area. As stated by Feiner (1993), they built a laser printer repairs application for printer maintenance. As shown in Figure 7.16, the user's view which the existing wireframe is guiding user on how to remove the paper tray. Almost applications also prove successful in Caudell (1992), Tuceryan (1995) and Sims (1994). Moreover, according to Tuceryan (1995), AR might be used for any complex machines such as automobile engines.

Robot Path Planning

The complex problem is the operation of a robot, especially with long delays in means of communication when the robot is moving far away. In effects, especially to conduct the robot directly without delay, it may be to instead run a virtual version of the robot. Manipulation of the local virtual version is needed in order to plan the robot's actions in real-time that are specified by users. The outputs are directly displayed on the real world. Then, user will inform the real robot to run the plan once the plan is tested and determined.

(18)

Figure 7.16 Virtual lines illustrate a designed motion of a robot arm (Milgram 1993)

Entertainments and Games

(19)

Figure 7.17 Invisible Train PDA Games (Wagner et al. 2005)

As introduced by Wagner et al. (2005) on their successful project delivery, The Invisible Train, which was the first real multi-user AR application for PDA devices. Figure 7.18 shows AR application for mobile games that can be played using PDA technologies. The Invisible Train is a mobile, collaborative multi-user AR game, in which players control virtual trains on a real wooden tiny rail track as shown in Figure 7.19. Virtual trains are only visible to players through their PDA's display as they actually do not exist in real world. According to Wagner et al. (2005), this type of user interface is usually called the "magic lens metaphor".

(20)

The social communication aspect can be clearly observed with non-computer based multi-player board-games like Mah-Jongg, Trivial Pursuit, as stated in Gervautz et al. (1998), their research on collaborative gaming in AR. As shown in Figure 7.20, other collaborative AR applied in games, which multi-user interact each other to play the game in virtual mixed with real world.

Figure 7.19 Collaborative AR in Mah-jong game (Gervautz et al. 1998)

(21)

Figure 7.20 Nonverbal communication application (Thalmann et al. 1998)

CONCLUSION

AR is a field of computer research which deals with the combination of real-world and computer-generated data. AR research is evolved from the areas of virtual reality, wearable and ubiquitous computing and human computer interface. Human factors have been studied in order to know the information representation in such a way that the users are able to distinguish between what is real and what is virtual information. Recently, the field of AR is evolving as its own discipline, with strong ties to these related research areas.

(22)

REFERENCES

Azuma, R. 1997, “A Survey of Augmented Reality.” Presence: Teleoperators and Virtual Environments 6, no.4, pp. 355-385.

Azuma, R , Baillot, Y, Behringer, R, Feiner, S, Julier, S and MacIntyre, B., 2001, “Recent Advances in Augmented Reality.” IEEE Computer Graphics and Applications (6), pp. 34-47.

Bornik , A, Beichel ,R, Reitinger ,B, Gotschuli ,G, Sorantin, E, Leberl, F and Sonka, M 2003, “Computer aided liver surgery planning: An augmented reality approach.” Medical Imaging 2003: Visualization and Display, Proceedings of SPIE, volume 5029.

Caudell, Thomas , P and Mizell, Dw 1992, “Augmented Reality: An Application of Heads-Up Display Technology to Manual Manufacturing Processes,” Proceedings of Hawaii International Conference on System Sciences, pp. 659-669. Chinthammit, Winyu, Burstein, R , Seibel, E and Furness, T., 2001,

‘Head tracking using the Virtual Retinal Display,’ Second IEEE and ACM International Symposium on Augmented Reality, October 29-30, New York, NY.

Drascic, D, Grodski, JJ, Milgram,P, Ruffo,K, Wong,P and Zhai, S 1993, “ARGOS: A Display System for Augmenting

Reality”, Video Proceedings of INTERCHI '93: Human Factors in Computing Systems (Amsterdam, the

Netherlands, 24-29 April 1993). ACM SIGGRAPH

Technical Video Review, Volume 88. Extended abstract in Proceedings of INTERCHI '93, pp. 521.

(23)

Fitzmaurice and George, 1993, “Situated Information Spaces: Spatially Aware Palmtop Computers.” CACM 36, 7 (July 1993), pp. 38-49.

Gervautz, M, Szalavári, Z and Eckstein, E, 1998, “Collaborative gaming in augmented reality.” ACM Symposium on Virtual reality software and technology, Taipei, Taiwan. Kancherla, Anantha, R, Rolland,Jp, Wright, Dl and Burdea, G

1995, “A Novel Virtual Reality Tool for Teaching

Dynamic 3D Anatomy.” Proceedings of Computer Vision, Virtual Reality, and Robotics in Medicine '95 (CVRMed '95) (Nice, France, 3-6 April 1995), pp. 163-169.

Kaufmann, H, Schmalstieg, D and Wagner, M, 2000, “Construct3D: A Virtual Reality Application for Mathematics and Geometry Education,” Education and Information Technologies 5:4 (December 2000), pp. 263-276.

Milgram ,P and Kishino,F 1994, “A Taxonomy of Mixed Reality Visual Display.” IEICE Trans. Information System, vol.E77-D, no.12, pp. 1321-1329.

Milgram, P, Zhai,S, Drascic, D and Grodski, Jj 1993,

“Applications of Augmented Reality for Human-Robot Communication.” Proceedings of International Conference on Intelligent Robotics and Systems (Yokohama, Japan, July 1993), 1467-1472.

Roussos, M, Johnson, A, Moher, T, Leigh, J, Vasilakis, C and Barnes, C 1999, “Learning and Building Together in an Immersive Virtual World”, PRESENCE 8(3), pp. 247-263. Rose, Eric, Breen,D, Ahlers, K, Crampton,C, Tuceryan,M,

Whitaker, R and Greer, D 1995, “Annotating Real-World Objects Using Augmented Reality.” Proceedings of Computer Graphics International '95 (Leeds, UK, 25-30 June 1995), pp. 357-370.

Sims, Dave 1994, “New Realities in Aircraft Design and

(24)

State, Andrei, Mark, Al, Gentaro , H, William, Fg, Mary Cw, Henry, F and Pisano, Ed 1996, “Techniques for

Augmented-Reality Systems: Realizing Ultrasound-Guided Needle Biopsies.” Proceedings of SIGGRAPH ‘96, pp. 439-446.

Silva, R, Oliveira, Jc and Giraldi, Ga 2003, “Introduction to Augmented Reality.” viewed 12 August 2007,

www.lncc.br/~jauvane/papers/RelatorioTecnicoLNCC-2503.pdf

Thalmann ,N, Thalmann, D, Tolga, K and Pandzic, Is 1998, “Nonverbal Communication Interface for Collaborative Virtual Environments.”, Proc. Computer Graphics International 1998.

Tuceryan, Mihran, Greer,Ds, Whitaker, Rt, Breen,D, Crampton,C, Rose,E and Ahlers, Kh 1995, “Calibration Requirements and Procedures for Augmented Reality” IEEE Transactions on Visualization and Computer Graphics 1, 3 (September 1995), pp. 255-273.

Wagner, D, Pintaric,T, Ledermann,F and Schmalstieg, D 2005, ‘Towards Massively Multi-User Augmented Reality on Handheld Devices,’ Vienna University of Technology. Wursthorn, S, Coelho, Ah and Staub, G 2004, “Applications for

Mixed Reality.”, XXth ISPRS Congress, Istanbul, Turkey. Yamamoto, H 1999, ‘Case Studies of Producing Mixed Reality