AUGMENTED REALITY METHODS AND ALGORITHMS FOR HEARING AUGMENTATION
by
Julie Carmigniani
A Thesis Submitted to the Faculty of The College of Engineering and Computer Science In Partial Fulfillment of the Requirements for the Degree of
Master of Science
Florida Atlantic University Boca Raton, Florida
iii
ACKNOWLEDGEMENTS
I would like to begin by offering my sincerest gratitude to my advisor Dr. Borko Furht. His persistence, encouragements, and advices proved vital in the steps leading up to this point in my education as well as career. Additionally, Dr. Oge Marques who taught me more in more than one class proved himself a remarkable professor and for that I am very appreciative.
I would also like to express my deepest appreciation to my parents Vincent and Marie-Christine Carmigniani whose constant support, love, and trust propelled me to this point in my life.
Finally, I would like to thank my friend, Tegan Story-Llewellyn, for her friendship, support, and advices that have meant more for me than I could ever say.
iv ABSTRACT
Author: Julie Carmigniani
Title: Augmented Reality Methods and Algorithms for Hearing Augmentation
Institution: Florida Atlantic University Thesis Advisor: Dr. Borko Furht
Degree: Master of Science
Year: 2011
While new technologies are often used to facilitate regular people’s lives, they often fail to see their potential in helping disabled people. Augmented reality, one of the newest state-of-the-art technologies, offers users the opportunity to add virtual information to their real world surroundings in real time. It also has the potential to not only augment the sense of sight, but also other senses such as hearing. Augmented reality could be used to offer the opportunity to complement users’ missing sense.
In this thesis, we study augmented reality technologies, systems and applications, and suggest the future of AR applications. We explain how to integrate augmented reality into iOS applications and propose an augmented reality application for hearing augmentation using an iPad2. We believe mobile devices are the best platform for augmented reality as they are widespread and their computational power is rapidly growing to be able to handle true AR applications.
v
AUGMENTED REALITY METHODS AND ALGORITHMS FOR HEARING AUGMENTATION
LIST OF TABLES ... viii
LIST OF FIGURES ... ix
CHAPTER 1: INTRODUCTION ...1
1.1. Background ... 1
1.1.1. Introduction to Augmented Reality ... 1
1.1.2. History and Background of Augmented Reality ... 3
1.2. Motivation ... 5
1.3. Thesis Contribution ... 6
1.4. Thesis Organization ... 7
CHAPTER 2: AUGMENTED REALITY TECHNOLOGY, SYSTEMS AND APPLICATIONS ...9
2.1. Augmented Reality Technologies ... 9
2.1.1. Computer Vision Methods in Augmented Reality ... 9
2.1.2. AR Devices ... 13
2.1.2.1. Displays ... 13
2.1.2.2. Input Devices ... 16
2.1.2.3. Tracking ... 18
vi 2.1.3. AR Interfaces ... 19 2.1.3.1. Tangible AR Interfaces ... 19 2.1.3.2. Collaborative AR Interfaces ... 20 2.1.3.3. Hybrid AR Interfaces ... 20 2.1.3.4. Multimodal AR Interfaces ... 21 2.1.4. AR Tool Kits ... 21
2.2. Augmented Reality Systems ... 23
2.2.1. AR General Systems ... 23
2.2.2. AR Mobile Systems ... 28
2.2.2.1. Socially Acceptable Systems ... 29
2.2.2.2. Personal and Private Systems ... 31
2.2.2.3. Tracking Technology ... 32
2.2.2.4. Mobile AR Systems Design in this Thesis ... 33
2.3. Augmented Reality Applications ... 34
2.3.1. Advertising and Commercial AR Applications ... 34
2.3.2. Entertainment and Education AR Applications ... 40
2.3.3. Medical AR Applications ... 45
2.3.4. Mobile (iPhone) AR Applications ... 49
2.4. Augmented Reality’s Future Direction ... 52
CHAPTER 3: USING THE IOS PLATFORM TO BUILD AN AUGMENTED REALITY APPLICATION ...57
3.1. Motivation ... 57
vii
3.2.1. Picture View Controller ... 61
3.3. Final Phase Design ... 64
3.3.1. AV Foundation Framework Implementation ... 66
CHAPTER 4: DESIGN OF AN AUGMENTED REALITY HEARING APPLICATION 69 4.1. Motivation ... 69
4.2. Design ... 69
4.3. OpenEars – Speech Recognition System ... 75
4.3.1. ARPA Language Model ... 76
4.3.2. Lmtool Language Modeler ... 78
4.3.3. Pocketsphinx Library ... 79
4.4. OpenCV Face Detection Algorithm ... 82
4.4.1. OpenCV Library – Brief Introduction ... 84
4.4.2. Face Detection Algorithm ... 87
4.4.2.1. Viola-Jones Method ... 89
4.5. Prototype Results and Screenshots ... 94
CHAPTER 5: SUMMARY AND FUTURE RESEARCH DIRECTION ...99
5.1. Summary ... 99
5.2. Future Work ... 102
viii
LIST OF TABLES
Table 2.1: Comparison of different techniques for different types of displays ... 17
Table 2.2: Comparison of common tracking technologies ... 19
Table 2.3: Abbreviations ... 25
Table 2.4: AR System Comparison ... 26
Table 2.5: Systems preferred type of technologies ... 27
Table 4.1: OpenCV Portability Guide for Release 1.0 [BrKa08] ... 85
ix
LIST OF FIGURES
Figure 1.1: Milgram's Reality-Virtuality Continuum [MiKi94] ... 1
Figure 1.2: Ivan Sutherland's Head-Mounted Display [IS09] ... 5
Figure 2.1: Point constraints for the camera pose problem [AbMa08] ... 11
Figure 2.2: HMD Display [ReSc03] ... 14
Figure 2.3: Handheld displays [WaSc06] ... 15
Figure 2.4: Example of SAR [SaMo] ... 16
Figure 2.5: MINI Advertisement ... 35
Figure 2.6: right picture: Picture of a virtual motorcycle prototype (on the right) next to a physical motorcycle prototype (on the left) in the real environment; left picture: Virtual prototype in a virtual environment [SaMo]. ...36
Figure 2.7: Virtual factory prototype [SaMo] ... 36
Figure 2.8: Virtual office prototype [SaMo] ... 37
Figure 2.9: Virtual testing of a lighting system [SaMo] ... 37
Figure 2.10: User trying on virtual shoes in front of the Magic Mirror [SaMo] ... 38
Figure 2.11: Cisco's AR commercial where a client is trying on clothes in front of a "magic" screen ...40
Figure 2.12: Augmented reality view of Dashuifa [HuLi09] ... 41
x
Figure 2.14: Mobile phone-enabled guidance in a museum [BrBr07] ... 42
Figure 2.15: ARCC [CoKe04] ... 43
Figure 2.16: Bichlmeier et. al. system for viewing through the skin [BiWi07] ... 45
Figure 2.17: Sequential screenshots of knot tying task [AkRe06]... 47
Figure 2.18: Wikitude Drive ... 50
Figure 2.19: Firefighter 360 app ... 50
Figure 2.20: Le Bar Guide app ... 50
Figure 2.21: From top to bottom and left to right: DARPA's contact lens project [Wi08], MIT's Sixth Sense [MiMa09], Contactum's AR solutions [Co06] ...53
Figure 2.22: From top to bottom and left to right: Examples of futuristic augmented reality and Babak Parviz's contact lens [Pa09] ...54
Figure 3.1: iTranslatAR first phase design picture translation general system ... 59
Figure 3.2: iTranslatAR - Screenshots ... 60
Figure 3.3: Image operation set up snapshot ... 62
Figure 3.4: Image processing for OCR snapshot ... 63
Figure 3.5: iTranslatAR final phase design video feed translation general system ... 66
Figure 3.6: AV Foundation framework video frame analyzes set up snapshot ... 68
Figure 4.1: iHeAR General System Design ... 72
Figure 4.2: Speech detected snapshot ... 73
Figure 4.3: Speech detected and frame analyzed for face snapshot ... 73
Figure 4.4: Frame processing snapshot ... 74
xi
Figure 4.6: General Speech Recognition System ... 80
Figure 4.7: Face detection method ... 83
Figure 4.8: Creating an OpenCV image object from Objective-C image object ... 84
Figure 4.9: OpenCV Basic Structure ... 86
Figure 4.10: OpenCV Detector Algorithm Architecture ... 88
Figure 4.11: Haar features from the cascade files that ship with OpenCV Haarcascade Frontalface Default [Ha10] ...89
Figure 4.12: Haar Features ... 90
Figure 4.13: Rectangle with coordinates A(x1, y1), B(x2, y2), C(x3, y3) and D(x4,y4) ...91
Figure 4.14: AdaBoost classifier cascade as a chain of filters [He07] ... 93
Figure 4.15: iHeAR ready to listen with "This is my application" speech detected ... 95
Figure 4.16: iHeAR recognized "This is working well" ... 95
Figure 4.17: iHeAR - Screenshots: Demonstration of face detection. ... 96
Figure 4.18: iHeAR – Screenshots: : Demonstration of translation. ... 97
1
1. INTRODUCTION
1.1. BACKGROUND
1.1.1. INTRODUCTION TO AUGMENTED REALITY
Augmented Reality, also commonly referred to as AR, is defined as a real-time direct or indirect view of a physical real-world environment that has been enhanced or
augmented by adding virtual computer-generated information to it [CaFu11]. AR is both
interactive and registered in 3D as well as combines real and virtual objects. Milgram’s Reality-Virtuality Continuum is defined by Paul Milgram and Fumio Kishino as a continuum that spans between the real environment and the virtual environment, and comprises Augmented Reality and Augmented Virtuality (AV) in between, where AR is closer to the real world while AV is closer to a pure virtual environment [CaFu10]. The Milgram’s Reality-Virtuality Continuum can be seen in Figure 1.1.
Figure 1.1: Milgram's Reality-Virtuality Continuum [MiKi94]
Augmented Reality aims at simplifying the user’s life by bringing virtual information not only to his immediate surroundings, but also to any indirect view of the real-world environment, such as live-video streams with recorded and live football games
2
where different color lines help the viewer better understand the changes in perception to see where players are located with regard to the important positions. AR enhances the user’s perception of and interaction with the real-world. While Virtual Reality (VR) technology or Virtual Environment as referred to by Milgram completely immerses the users in a synthetic environment without having any contact with the real world, AR technology offers the user a dual world with both real and synthetic environment where the virtual objects are added in such a way as to augment the user’s perception in real time so as to provide them with a better understanding of their surroundings. Virtual objects added to the real environment show information that the user cannot directly detect with his/her senses. The information passed on by the virtual object can help the user in performing daily work tasks, such as guiding workers through electrical wires in an aircraft by displaying pertinent digital information or by guiding maintenance workers through displaying which units might need repair or are due for maintenance. The information can also have a simple entertainment purpose, such as Wikitude and many mobile augmented reality applications. AR applications cover a wide range of classes such as medical visualization, entertainment, advertising, maintenance and repair, annotation, robot path planning, etc.
Augmented Reality is considered by many [CaFu11, AzBa01] to not be restricted to a particular type of display technology such as head-mounted display (HMD), nor is it considered to be limited to the sense of sight. AR can potentially be applied to all senses, augmenting smell, touch and hearing as well. In this effect, AR can also be used to augment or replace a user’s missing sense by sensory substation, such as augmenting hearing for deaf users by the use of visual cues as will be discussed later on in this
3
research work. Similarly, many [CaFu11, AzBa01] also consider AR applications that require removing real objects from the environment and are commonly referred to as
mediated or diminished reality, in addition to adding virtual objects. In fact, removing
objects from the real environment corresponds to covering the object with virtual information that matches the background in order to give the user the impression that the object is not there.
1.1.2. HISTORY AND BACKGROUND OF AUGMENTED REALITY
The first appearance of Augmented Reality (AR) dates back to the 1950s when Morton Heilig, a cinematographer, thought of cinema as an activity that would have the ability to draw the viewer into the onscreen activity by taking in all the senses in an effective manner. In 1962, Heilig built a prototype of his vision named Sensorama, which he described in 1955 in “The Cinema of the Future” and predated digital computing. [CaFu11]. Next, Ivan Sutherland invented the head mounted display in 1966 as seen in Figure 1.2. In 1968, Sutherland was the first one to create an augmented reality system using optical see-through head mounted display [IS09]. In 1975, Myron Krueger creates the Videoplace, a room that allows users to interact with virtual objects for the first time. Later, Tom Caudell and David Mizell from Boeing create the term Augmented Reality while helping workers assemble wires and cables for an aircraft [CaFu11]. They were also the first to start discussing the advantages of augmented reality versus virtual reality, such as requiring less power since fewer pixels are needed [IS09]. In the same year, L.B Rosenberg developed one of the first functioning AR systems, called Virtual Fixtures and demonstrated its benefit on human performance while Steven Feiner, Blair MacIntyre and Doree Selgmann presented the first major paper on an AR system prototype named
4
KARMA. The reality virtuality continuum seen in Figure 1.1 is not defined until 1994 by Paul Milgram and Fumio Kishino as a continuum that spans from the real environment to the virtual environment. Augmented reality and augmented virtuality are located somewhere in between with AR being closer to the real environment and AV being close the virtual environment. In 1997, Ronal Azuma writes the first survey in AR providing a widely acknowledged definition of AR by identifying it as combining real and virtual environment while being both registered in 3D and interactive in real time [IS09]. The first outdoor mobile AR game, ARQuake, is developed three years later by Bruce Thomas and demonstrated during the International Symposium on Wearable Computers. In 2005, the Horizon Report predicts that AR technologies will emerge more fully within the next 4-5 years; and, as to confirm that prediction, camera systems that can analyze physical environments in real time and relate positions between object and environments are developed the same year. This type of camera system has become the basis to integrate virtual objects with reality in AR systems. In the following years, more and more AR applications are developed especially with mobile applications, such as Wikitude AR Travel Guide launched in 2008, but also with the development of medical applications in 2007. Nowadays, with the new advances in technology, an increasing amount of AR systems and application have emerged, notably MIT sixth sense prototype, and it is expected that more and more of these mobile AR applications will surface with the frequent appearances of newer and more advanced portable devices such as latest Android phones, iPhones and iPads that have more and more the capability of handling the computational needs required by augmented reality applications.
Figure 1.2: Ivan Sutherland's Head
1.2. MOTIVATION
While new technologies are often used to facilitate regular people’s lives, they often fail to see their potential i
offers users the opportunity to add virtual information to their real world surroundings in real time. It also has the opportunity to not only augment the sense of sight, but al
senses such as hearing. I opportunity to complement
used to help deaf people in “hearing” what an interlocutor might be telling them
needing to know how to read lips or without needing the interlocutor to need to know sign language by outputting the person’s speech next to their head in an intuitive, easy to interact with, and easily understandable way. As an addition, such a system can also be used as a convenient translator by travelers.
5
.2: Ivan Sutherland's Head-Mounted Display [IS09
While new technologies are often used to facilitate regular people’s lives, they often fail to see their potential in helping disabled people. Augmented reality currently offers users the opportunity to add virtual information to their real world surroundings in
It also has the opportunity to not only augment the sense of sight, but al
hearing. In this sense, augmented reality could be used to offer the complement a user’s missing sense. For instance, augmented reality can be used to help deaf people in “hearing” what an interlocutor might be telling them
to know how to read lips or without needing the interlocutor to need to know sign language by outputting the person’s speech next to their head in an intuitive, easy to interact with, and easily understandable way. As an addition, such a system can also be used as a convenient translator by travelers.
[IS09]
While new technologies are often used to facilitate regular people’s lives, they n helping disabled people. Augmented reality currently offers users the opportunity to add virtual information to their real world surroundings in It also has the opportunity to not only augment the sense of sight, but also other n this sense, augmented reality could be used to offer the For instance, augmented reality can be used to help deaf people in “hearing” what an interlocutor might be telling them, without to know how to read lips or without needing the interlocutor to need to know sign language by outputting the person’s speech next to their head in an intuitive, easy to interact with, and easily understandable way. As an addition, such a system can also be
6
Using an iPad2 with a camera as the interaction device, this thesis introduces a system with speech recognition and optional language translation and display of the resulting string in an easy and natural way to use by detecting a face and its corresponding position present in the frames and outputting the resulting string next to the detected face in a cartoon-like bubble. The use of this system, dubbed iHeAR, consist in having the user simply angles the device towards the person’s face and once both text and face are detected, the final string is outputted on the screen without requiring any additional steps from the user.
This system presents one of the different aspects of augmented reality that does not only involve augmenting the sense of sight, but also involves hearing augmentation. In addition, the following conditions and limitations are assumed:
•
The user does not know sign language or how to read lips,•
The environment is quiet and free of background noise,•
The system will be used for one-on-one conversation,•
Speech recognition needs to happen on the device so as to not depend on network availability.1.3. THESIS CONTRIBUTION
The contributions of this research are as follows:
•
A detailed description of augmented reality technologies, systems, and applications, notably mobile applications, and the future of augmented reality technologies.7
•
The design and implementation of a picture translator, dubbed iTranslatAR, using the iOS platform, Tesseract OCR and Google Translate as a way of introducing the iOS platform and integration of foreign libraries and APIs within the iOS platform.•
The design of an augmented reality video frame translating application using the iTranslatAR original design as a starting point. Although iTranslatAR augmented reality design was not implemented, a detailed explanation of the design and steps to implementing iOS AV Foundation framework used in the main focus of this research is provided along with this work.•
The design of an augmented reality hearing application dubbed iHeAR using the iOS platform, OpenEars open source as the means for speech recognition, OpenCV as the means for image/frame processing, and Google Translate as the means for language-to-language translation. A detailed explanation of the OpenEars speech recognition system and the OpenCV face detection algorithm is provided along with this work.•
The implementation of the augmented reality hearing application design on iPad2.1.4. THESIS ORGANIZATION
Chapter 1 of this thesis provides an introduction and historical background to the state-of-the-art technology that is augmented reality and the objective of this research using augmented reality. Chapter 2 provides a more in-depth explanation of augmented reality current systems, tools and applications along with a better understanding of mobile applications challenges and the future direction augmented reality is taking. In
8
Chapter 3, an augmented reality application for picture/video frame translation’s design is presented along with some of the critical algorithm used during the first phase implementation of the design. Chapter 4 introduces an augmented reality application for hearing augmentation’s design in conjunction with its critical algorithms used for speech recognition and face detection during the implementation of the design. Results of the prototype developed for this research are also discussed in Chapter 4. Chapter 5 concludes this body of work while presenting new directions for potential future research and further implementation of the proposed two augmented reality applications designs.
9
2. AUGMENTED REALITY TECHNOLOGY, SYSTEMS AND APPLICATIONS
2.1. AUGMENTED REALITY TECHNOLOGIES
Augmented reality being a state-of-the-art technology requires state-of-the-art algorithms and devices to be feasible. The main tools to augmented reality are computer vision, devices capable of handling heavy computations, and intuitive interfaces to provide the user with a natural and easy-to-use way to interact with the AR system. These tools are described in this section.
2.1.1. COMPUTER VISION METHODS IN AUGMENTED REALITY
Computer vision renders 3D virtual objects from the same viewpoint from which the images of the real scene are being taken by tracking cameras. Augmented reality image registration uses different methods of computer vision mostly related to video tracking. These methods are usually divided into two stages: tracking and reconstructing/recognizing. During tracking, fiducial markers, optical images, or interest points are detected in the camera images using various image processing methods such as feature detection or edge detection. Most of the computer vision tracking techniques can be divided into two classes: feature-based tracking and model-based tracking. Feature-based methods consist of discovering the connection between 2D image features and their 3D world frame coordinates. Model-based methods make use of the model of the desired tracked objects’ features such as CAD models or 2D templates of the item using
10
distinguishable features. Once a connection is made between the 2D image and the 3D world frame, the camera pose can be calculated by projecting the 3D coordinates of the feature into the observed 2D image coordinates and by minimizing the distance to their corresponding 2D features. Camera pose estimation most often uses point features to determine the appropriate constraints. The reconstructing/recognizing stage makes use of the data obtained during the first stage to reconstruct the real world coordinate system [CaFu11].
Assuming a calibrated camera and a perspective projection model, if a point has coordinates
(
x ,,y z)
T in the coordinate frame of the camera, its projection onto the image plane is(
x/z,y/z,1)
T.In point constraints, there are two principal coordinate systems, illustrated in Figure 2.1: the world coordinate system W and the 2D image coordinate system. Let
(
)
Ti i i
i x y z
p , , , where i=1,...,n, with n≥3, be a set of 3D non-collinear reference points in the world frame coordinate and qi
(
xi',yi',zi')
T be the corresponding camera-space coordinates, p and i q are related by the following transformation: iT Rp qi = i + (1) where = T T T r r r R 3 2 1 and = z y x t t t T (2)
11
Let the image point hi (ui, vi, 1)T be the projection of pi on the normalized image plane. The collinearity equation establishing the relationship between hi and pi using the camera pinhole is given by:
(
Rp T)
t p r h i z i T i + + = 3 1 (3)The image space error gives a relationship between 3D reference points, their corresponding 2D extracted image points, and the camera pose parameters, and corresponds to the point constraints [AbMa08]. The image space error is given as follows: 2 3 2 2 3 1 + + − + + + − = ∧ ∧ z i T y i T i z i T x i T i p i t p r t p r v t p r t p r u E (4) where T i i i u v m ∧ ∧ ∧ 1 ,
, are the observed image points.
12
In augmented reality, most methods of computer vision assume the presence of fiducial markers or objects with known 3D geometry in the environment, and make use of those data. Static systems of known initial positions have the 3D scene structure calculated beforehand, while dynamic systems make use of Simultaneous Localization And Mapping (SLAM) technique for mapping fiducial markers or 3D models relative positions. In the rare but growing cases when no assumptions about the 3D geometry of the scene can be made, Structure from Motion (SfM) method is used. SfM method is divided into two parts: feature point tracking and camera parameter estimation.
Augmented reality tracking methods depend greatly on the type of environment and the type of AR system built. Knowing that the environment might be indoor, outdoor or a combination of both will dictate the lightning control capability of the system beforehand, which is an important factor in computer vision. Likewise, whether the AR system built is meant to be static or dynamic, or whether the environment it will be used in is known or unknown, will also dictate the type of computer vision methods to use.
Although visual tracking now has the ability to recognize and track a lot of things, it mostly relies on other techniques such as GPS and accelerometers. For example, for a computer to detect and recognize something as familiar and obvious as a car is very quite a challenge. The surface of most cars is both shiny and smooth and most of the feature points come from reflections and thus are not relevant for pose estimation and even sometimes recognition. The few stable features that one can hope to recognize, such as window corners or wheels are extremely difficult to match due to reflection and transparent parts. While this example is a bit extreme, it shows the difficulties and
13
challenges faced by computer vision with most objects having an irregular shape: food, flowers, and most artifacts.
A recent new approach for advances in visual tracking has been to study how the human brain recognizes objects, also referred to as the Human Vision System (HVS), as it is possible for humans to recognize an infinite number of objects and persons in fractions of seconds without making any effort. If the way of recognizing things by the human brain can be modeled, computer vision will be able to handle the challenges it is currently facing and keep moving forward [CaFu11].
2.1.2. AR DEVICES
Augmented reality’s main concerns when selecting appropriate devices are displays, input devices, tracking, and computer.
2.1.2.1.DISPLAYS
There are three major types of displays used in Augmented Reality: head mounted displays (HMD), handheld displays and spatial displays.
HMD can be worn directly on the head or as part of a helmet and place both images of the real and virtual environment over the user’s view. There are four types of HMD: video-see-through and optical-see-through and each can be either monocular or binocular display optic. Video-see-through requires the user to wear two cameras on their head as well as the processing of both cameras to provide the “real” part of the augmented scene and the virtual objects with unmatched resolution. Optical-see-through, on the other hand, employs half-silver mirror technology to allow the view of the physical world to pass through the lens and graphically overlay information to be
14
reflected in the users’ eyes. While video-see-through systems are more demanding than optical-see-through systems due to the processing, video-see-through systems allow much more control over the result since the augmented view is already composed by the computer. This can help in synchronizing the virtual image with the scene before displaying to the user and thus reduces the effect of jittering of the virtual objects.
Figure 2.2: HMD Display [ReSc03]
Handheld displays consist of small computing devices with a display that the user can hold in their hands. They use video-see-through techniques to overlay graphics onto the real environment and can often be directly use as the main platform. This research considers three distinct classes of commercially available handheld displays that are being used for augmented reality: smart-phones, PDAs and Tablet PCs. With most of the recent advances, smart-phones such as iPhones and Tablets such as iPads are becoming the preferred platforms for developing augmented reality applications as they do not require additional hardware to use, they present a combination of powerful CPU, camera, GPS, and solid state compasses, and they are extremely portable and widespread.
15
Figure 2.3: Handheld displays [WaSc06]
Spatial augmented reality, also known as SAR, makes use of video-projectors, optical elements, holograms, radio frequency tags, and other tracking technologies to display graphical information directly onto physical objects without requiring the user to wear or carry the display. Spatial displays separate most of the technology from the user and integrate it into the environment, thus offering the ability to naturally scale up to groups of users and offering collaboration between users. This also increases the interest for such augmented reality systems in universities, labs, museums, and in the art community. There are three different approaches to SAW which mainly differ in the way they augment the environment: video-see-through, optical-see-through and direct augmentation. Video-see-through displays in SAR are screen-based and are a common technique used if the system does not have to be mobile as they are cost efficient since only off-the-shelf hardware components and standard PC equipment is required. Spatial optical-see-through displays generate images that are aligned with the physical environment and often comprise spatial optical combiners, such as planar or curved mirror beam splitters, transparent screens, or optical holograms. Neither screen-based video-see-through spatial systems nor spatial optical-see-through systems support mobile applications due to spatially aligned optics and display technology. Lastly, projector-based spatial displays apply front-projection to seamlessly project images directly onto
16
physical object’s surfaces [BiRa07]. Table 2.1 shows a comparison of different types of display techniques for augmented reality.
Figure 2.4: Example of SAR [SaMo]
2.1.2.2.INPUT DEVICES
There are many types of input devices for augmented reality systems. Developers can make use of gloves [ReSc03], wireless wristbands [FeMu05], Wii wireless remotes, etc. but the author of this thesis believes that smart-phones and tablets such as iPhones and iPads are currently the most useful type of input devices as they can also be used as the main platform for the augmented reality system. It is important, however, to point out that the input devices chosen also greatly depend on the type of application the system is being developed for as well as the display chosen. For instance, if an application requires the user to be free handed to use it, the input device will need to be one that still allows the user to perform the necessary tasks without requiring extra unnatural gestures or to be held by the user. Examples of such input devices include gaze interaction in [LeLe10] or the wireless wristband used in [FeMu05]. Similarly, if a system makes use of a handheld display such as an iPhone, the developer can utilize the touch screen input device.
1
7
Table 2.1: Comparison of different techniques for different types of displays
Types of
Displays HMD Handheld Spatial
Techniques Video-see-through Optical-see-through Video-see-through Video-see-through Optical-see-through Direct Augmentation Types of Displays HMD Handheld Advantages complete visualization control, possible synchronization of the virtual and real environment employs a half-silver mirror technology, more natural perception of the real environment portable, widespread, powerful CPU, camera, accelerometer, GPS, and solid state compass portable, powerful CPU, camera, accelerometer, GPS, and solid state compass more powerful cost efficient, can be adopted using off-the-shelf hardware components and standard PC equipment more natural perception of the real environment displays directly onto physical objects' surfaces Disadvantages requires user to wear cameras on his/her head, requires processing of cameras video stream, unnatural perception of the real environment time lag, jittering of the virtual image small display becoming less widespread, small display more expensive and heavy
does not support mobile system does not support mobile system not user dependent: everybody sees the same thing (in some cases this
disadvantage can also be considered to be an advantage)
18 2.1.2.3.TRACKING
Tracking devices consist of digital cameras and/or other optical sensors, GPS, accelerometers, solid state compasses, wireless sensors, etc. Each of these technologies has a different level of accuracy and depends greatly on the type of system being developed. Table 2.2 shows a comparison of common tracking techniques adapted from [PaSi08] and [DiHo07], where range corresponds to the size of the region that can tracked within, setup is the amount of time for instrumentation and calibration, precision represents the granularity of a single output position, time is the duration for which useful tracking data is returned before it drifts too much, and environment corresponds to the location the tracker can be used: indoors or outdoors.
2.1.2.4.COMPUTER
Augmented reality systems require powerful CPUs and considerable amount of RAM to process camera images. Most mobile computing systems employ a laptop in a backpack configuration, but with the recent advances in technology, we can hope to see this backpack configuration replaced by a lighter and more sophisticated looking system. Stationary systems can use a traditional workstation with a powerful graphics card.
19
Table 2.2: Comparison of common tracking technologies
Technology Range (m) Setup time (hr) Precision (mm) Time (s) Environment
Optical: marker-based 10 0 10 ∞ in/out
Optical: marker less 50 0-1 10 ∞ in/out
Optical: outside-in 10 10 10 ∞ in
Optical: inside-out 50 0-1 10 ∞ in/out
GPS ∞ 0 5000 ∞ out WiFi 100 10 1000 ∞ in/out Accelerometer 1000 0 100 1000 in/out Magnetic 1 1 1 ∞ in/out Ultrasound 10 1 10 ∞ in Inertial 1 0 1 10 in/out Hybrid 30 10 1 ∞ in/out UWB 10-300 10 500 ∞ in
RFID: active 20-100 when needed 500 ∞ in/out
RFID: passive 0.05-5 when needed 500 ∞ in/out
2.1.3. AR INTERFACES
One of the most important aspects of augmented reality is to create appropriate techniques for intuitive interaction between the user and the virtual components of augmented reality applications. This research has identified four main ways of interaction in AR applications: tangible AR interfaces, collaborative AR interfaces, hybrid AR interfaces, and the emerging multimodal AR interfaces.
2.1.3.1. TANGIBLE AR INTERFACES
Tangible interfaces support direct interaction with the real world by exploiting the use of real, physical objects and tools such as touch screen display in smart phones or the use of sense tables as with TaPuMa [MiKu08]. TaPuMa is a table-top tangible interface,
20
also called sense table, which uses physical objects to interact with digital projected maps using real-life objects the user carries as queries to find locations or information on the map. The advantage of this type of application is found in using objects as keywords thus eliminating language barriers in conventional graphical interfaces that are often mistranslated. While objects can be interpreted in different ways based on culture and age, they still can be considered more meaningful to users.
Other examples of tangible objects include the use of wristbands such as in [CoKe04] and other various physical objects to interact with augmented reality systems. Tangible interfaces are currently the most common form of augmented reality interfaces. 2.1.3.2. COLLABORATIVE AR INTERFACES
Collaborative interfaces include the use of multiple displays to support remote and co-located activities. Co-located sharing uses 3D interfaces to improve physical collaborative workspace while remote sharing effortlessly integrates multiple devices with multiple locations to enhance teleconferencing. Such interfaces could also be integrated with medical applications for performing diagnostics, surgery, or even maintenance routine.
2.1.3.3. HYBRID AR INTERFACES
Hybrid interfaces combine an assortment of different, but complementary interfaces as well as the possibility to interact through a wide range of interaction devices. They also provide a flexible platform for unplanned, everyday interaction where it is not known in advance which type of display or device will be used. An example of such a system is Sandor. et al. augmented reality system implemented to support end
21
users in assigning physical interaction devices to operation as well as virtual objects on which to perform those procedures, and in reconfiguring the mappings between devices, objects and operations as the users interact with the system [SaOl05].
2.1.3.4. MULTIMODAL AR INTERFACES
Multimodal interfaces combine real objects’ input with naturally occurring forms of language and behaviors such as speech, touch, natural hand gestures, or even gaze. These types of interfaces have more recently emerged and examples of it include MIT’s sixth sense [MiMa09] wearable gesture interface, dubbed WUW, as it supplies the user with information projected onto surfaces, walls, and physical objects through natural hand gestures, arm movements, and/or interaction with the object itself. This type of interaction is now being largely developed and is sure to become one of the preferred types of interaction for future augmented reality application as it offers a relatively robust, efficient, expressive and highly mobile form of human-computer interaction that represents the user’s preferred interaction style. In addition, they have the capability to support users’ ability to flexibly combine modalities and choose from interaction methods such as speech or gesture, or to switch from one input mode to another depending on the task or setting. This freedom to choose the mode of interaction is crucial to wider acceptance of pervasive systems in public places [CaFu11].
2.1.4. AR TOOL KITS
Augmented reality being such a “hot” topic but not such an easy thing to accomplish, AR tool kits are a necessity and a convenience for developers that might not have the necessary background in image processing but wish to develop an augmented
22
reality system or application. One of the most well-known computer vision libraries for creating augmented reality applications is the ARToolKit developed in 1999 by Hirokazu Kato from the Nara Institute of Science and Technology and released by the University of Washington HIT Lab. It uses video tracking capabilities to calculate in real time the real camera position and orientation relative to physical markers. Once the real camera position is known, a virtual camera can be placed at the same exact position and 3D computer graphics model can be drawn to overlay the markers. The extended version of ARToolKit, ARToolKitPlus added many features over the ARToolKit, notably class-based APIs, but it is no longer being developed and already has a successor: Studierstube Tracker [CaFu11].
Studierstube Tracker’s concepts are very similar to ARToolKitPlus; however, its code-base is completely different and it is not an open source, thus not available for download. It has the ability to support mobile phone with Studierstube ES, as well as PCs, making its memory requirements very low: 100KB, which corresponds to 5 to 10% of the memory requirements of the ARToolKitPlus; and its processing very fast: about twice as fast as ARToolKitPlus on mobile phones and about 1ms per frame on a PC [ScFu00]. Studierstube Tracker is highly modular and developers can extend it in any way by creating new features for it. When first introducing Studierstube, the designer had in mind a user interface that “uses collaborative augmented reality to bridge multiple user interface dimensions: multiple users, contexts, and locales as well as applications, 3D-windows, hosts, display platforms, and operating systems” [ScFu00].
23 2.2. AUGMENTED REALITY SYSTEMS 2.2.1. AR GENERAL SYSTEMS
This research divides augmented reality systems into five categories: fixed indoor systems, fixed outdoor systems, mobile indoor systems, mobile outdoor systems, and mobile indoor and outdoor systems. We define mobile system as a system that does not constrain the user to one setting and thus allows for movement through the use of a wireless system. Fixed systems on the other hand do not allow to be moved while in use and the user must use these systems wherever they are set up without having the flexibility to move unless the whole system set up is relocated. Choosing what type of system to build is the first thing the developer should do as it will dictate the tracking system, display device and possibly the interface to select. For instance, fixed systems will not need to make use of GPS tracking while outdoor systems will most likely need to.
This research conducted a study using 25 randomly selected recent papers that were classified according to their systems, and determined what tracking techniques, display type and interface were used for each. Tables 4 and 5 show the results of the study and Table 3 shows the meaning of the abbreviation used in Tables 4 and 5. The papers used for the study were all published between 2002 and 2010 with a majority of papers published between 2005 and 2010.
The results from this study can be used to show the most popular type of systems developed so far since the papers were more or less randomly chosen from recent papers. However, it should be taken into account that mobile systems have been on the rise lately and while mobile indoor and outdoor systems represent only 12% of the papers studied,
24
due to the recent advances in technology, developers are looking more and more into this type of system as they have the most chance of making it into the market. Note that in the mobile indoor and outdoor systems, one of the system studied (Costanza’s eye-q [CoIn06]) does not use any tracking techniques, while others use multiple type of tracking techniques. This is due to the fact that this system was developed as a personal, subtle notification display. It was also noted that there was only one paper studied that discussed fixed outdoor systems because this type of system is not popular due to its inflexibility.
Although these results cannot be used as a general rule when building an augmented reality system, they can serve as a pointer of what type of tracking techniques, display, or interface is more popular for each system type. Developers should also keep in mind that these choices depend also on the type of applications, although as can be seen, the application does not necessarily guide the type of system.
From this study, we see that optical tracking is mostly preferred in fixed systems, while a hybrid approach is most often preferred for mobile systems. HMDs are often the preferred type of display choice; however, we predict that they will need to become more fashionably acceptable for systems using them to reach the market. When it comes to interfaces, the most popular choice is tangible interfaces, but we predict that multimodal interfaces will become more famous with developers within the next years as we believe that they also have a better chance to reach the public industry.
25
Table 2.3: Abbreviations
Abbreviation Meaning Abbreviation Meaning
Adv Advertising IO inside out
Ent. and Ed. Entertainment and Education OI outside in
Med. Medical MB marker-based
Nav. and Info. Navigational and Informational ML marker less
2
6
2
7
28 2.2.2. AR MOBILE SYSTEMS
Augmented reality mobile systems include mobile phone applications as well as wireless systems such as MIT’s sixth sense [MiMa09] and eye-q [CoIn06]. Users interact with mobile AR systems through the use of wearable mobile interfaces in a natural and socially acceptable way. Using mobile phones for developing augmented reality applications present both advantages and drawbacks. Most mobile devices of interest are equipped with cameras, and nowadays, these cameras are getting better and better with auto focusing lenses and the possibility to capture high level images. In addition, most cell phones provide accelerometers, magnetometers and GPS from which AR can benefit. However, in spite of rapid advances in mobile phones, their computing platform for real-time imaging applications is still rather limited if done using the cell phone’s platform. As a result, many applications send data to a remote computer that does the computation and send the results back to the mobile device, but this approach is not well adapted to AR due to limited bandwidth. Nevertheless, considering the rapid development in mobile devices computing power, it can be considered feasible to develop real-time augmented reality application that locally process information in the near future. In fact, the main focus of this research makes use of an iPad2 to create a hearing augmented reality application that processes both frames and speech-listening locally. While the augmented reality processing is not quite up-to-speed, the author was able to optimize the processing of the algorithms to only occur when necessary. Furthermore, rumors about upcoming iPhone devices claim that they will be equipped with the current iPad2 dual-core processor and there is good reason to believe that the
29
iPad3 will be equipped with an even more powerful processor. Likewise, upcoming Android mobile devices are claimed to be equipped with dual-core processors.
This thesis defines a successful augmented reality mobile system as an application that enables the user to focus on the application or system rather than on the computing device, interact with the device in a natural and socially acceptable way, and provide the user with information that can be shared if necessary. This suggests the need for lightweight, wearable or mobile devices that are fashionably acceptable, private and possess robust tracking technology.
2.2.2.1. SOCIALLY ACCEPTABLE SYSTEMS
Mobile phones and PDA’s reminders, messages, calls, etc. have been judged as distractions and are mostly considered not to be socially acceptable as they not only disrupt the person whose phone or PDA is receiving an interruption of more or less importance, but also the other persons present in the same room, whether they are having a conversation with the disruptor (as this is how the person whose device is disturbing the room will be seen as) or the disruptor is in a public place, such as a bus. As a result, many research groups such as [FeMu05], [CoIn06] have decided that interaction with augmented reality systems implemented on mobile devices needs to be subtle, discrete and unobtrusive so as to not disrupt the user if s/he is under a high load of work and the disruption is not of high priority level. A system that is subtle, discrete and unobtrusive thus becomes socially acceptable. The main issue with social acceptance comes from the level of disruption portable devices create in public places and during conversations and meetings. In [CoIn06], the authors studied the peripheral vision and adapt their mobile device’s information cue, eye-q which is a pair of glasses, so that it does not occlude the
30
foveal field of view, which is the main focus of the human field of view. The cues become less and less visible depending on the level of concentration and work-load of the user, making it naturally adaptive to the user’s cognitive workload and stress. Moreover, since the cues are only visible to the user and will only disrupt the user depending on his level of concentration, these cues can be considered to be socially acceptable. This research also considers multimodal interfaces to be crucial to wider acceptance of pervasive systems in public places as they offer the user the freedom to choose from a range of interaction modes giving the user the opportunity to select the most appropriate and socially acceptable mean of communication with their devices.
Socially acceptable systems also need to consider a natural way of interaction between the user and the device. If the interface and way of interaction between the user and the device is unnatural, it will appear awkward of use in public places. For instance, in [FeMu05], the authors have created an augmented reality system that uses a wireless wristband including an RFID reader, 3-axis accelerometer and RF communication facilities, a cell phone and a wireless earpiece to allow the user to interact with services related to objects using the RFID tags through implicit touch-based gestures. Once an object is detected in the user’s hand, the user can interact with information about this object using natural slight wrist gestures while previous commercial interfaces that supported hands-free and eyes-free operations required speech recognition, which not only suffers from medium performance and noisy conditions, but is also not considered to be socially acceptable.
Mobile augmented reality systems that wish to step from the laboratories to the industry will also be facing fashion issues as the users are unlikely to want to wear HMDs
31
or other visible devices that might be deemed as “geeky”. As a result, it is up to the developers of AR mobile systems to take into account fashion trends as they might be a big obstacle to overcome. Groups such as MIT Media Lab, constantly try to reduce the amount of undesirable visible devices or arrange them in different design choice. WUW’s first development stage integrated the camera and projector into a hat and the second development integrated it into a pendant. They were also researching a way to replace the need to wear colored marker caps on finger tips [MiMa09]. Although these are not quite up to the current fashion standard, they are a good step in the direction of “fashionably acceptable” and since fashion changes often, they might even be capable of bridging the gap between “geeky” and fashionably acceptable.
2.2.2.2. PERSONAL AND PRIVATE SYSTEMS
Augmented reality mobile systems need to be personal, meaning that the displayed information should only be viewed by others if the user allows it. MIT’s SixthSense technology [MiMa09] although very advanced, does not offer such privacy to its user due to the use of direct augmentation technique without using any viewing device for protecting the information. Anyone can see the same thing as the user at any time. This poses a dilemma as not needing to wear or carry any extra viewing device for the WUW is an advantage for “fashionably acceptable” devices; however, it is a problem when it comes to privacy. Systems, such as Costanza et. al. eye-q [CoIn06] or Babak Parviz’s contact lens [Pa09], offer such privacy to the user with information that can only be viewed by the user. These systems can also be considered socially acceptable as they are discrete and subtle as well as fashionably correct. However, these systems do not offer the ability of sharing information if the user desires to. A successful AR mobile
32
system should provide the user with private information that can be shared when the user wishes to.
In addition, AR mobile systems need to be careful not to violate other users’ and non-users’ privacy in new ways. Indeed, information that is available and not considered private on social networks, for instance, can be considered private in everyday life. As a result, technologies such as WUW [MiMa09], that make use of online available information about other persons to display them for the user, might face privacy issues due to the way the information is being disclosed.
2.2.2.3. TRACKING TECHNOLOGY
It is well known that for AR to be able to trick the human senses into believing that computer-generated information coexists with the real environment, very accurate position and orientation tracking is required.
As was seen in the AR systems section, the most common type of tracking systems for mobile systems is by combining a few complimentary tracking techniques to comprise the advantages of both and support the disadvantages of the other, which creates hybrid tracking. Outdoor systems mostly make use of GPS and inertial tracking technique with the use of accelerometers, gyroscopes, electronic compasses and/or other rotation sensors, along with some computer vision tracking techniques. GPS systems, although lacking in precision, provide an easy tracking system for outdoor systems that allow for better estimating the position of the user and its orientation once coupled with some inertial sensors. In this way, the user’s interest point is narrowed down and allows for easier visual tracking with fewer options. Indoor systems where GPS cannot be used unite visual tracking with inertial techniques only. Visual tracking achieve the best results
33
with low frequency motion, but is highly likely to fail with rapid camera movement, such as the ones that will occur with HMDs. On the other hand, inertial tracking sensors perform best with high frequency motion, while slow movements do not provide good results due to noise and bias drift. The complementary nature of these systems leads to combining them together in most hybrid systems.
Other systems, such as [LeLe10] and [ArPe09], rely on computer vision for tracking, but most are indoor systems with which the environment can be somewhat controlled. When it comes to visual outdoor tracking, a couple of factors, such as lightning, make tracking extremely difficult. Moreover, some objects present tracking difficulties. One of the most advanced visual tracking mobile systems is Google Goggles; however, this application can only track objects of regular form such as barcodes and books, or places thanks to its GPS and accelerometer that help the application recognize where the user is standing and the user’s orientation to narrow down the options. Google Goggles cannot recognize things of irregular shapes such as leaves, flowers or food. 2.2.2.4. MOBILE AR SYSTEMS DESIGN IN THIS THESIS
The mobile applications developed in the specs of this research use iPhone and iPad2 as the interaction and platform devices. These devices have been proven to be fashionably acceptable and easy to interact with, and it is up to the developer to keep the natural way of interaction present in iOS devices to keep the interface simple. While it is simple to avoid audio cues in most of these applications, in the case of the hearing augmented reality, the application was designed for hearing impaired people to interact with one other person in a natural and easy to use fashion, and not to use in public places.
34
2.3. AUGMENTED REALITY APPLICATIONS
While there are many possibilities for using augmented reality in an innovative and original way, this research has cornered four types of applications that are most often the main focus in AR research: advertising and commercial, entertainment and education, medical, and mobile applications of often trivial use and which we chose to focus on iPhone and iPad devices. In the following chapter section, this research studies why augmented reality could bring a better solution to some areas, a cheaper one to other areas, or simply create a new type of service. The challenges augmented reality is facing to go from the laboratories to the industry are also discussed.
2.3.1. ADVERTISING AND COMMERCIAL AR APPLICATIONS
Marketers have started to use a form of augmented reality to promote their new products online. Most techniques use markers such as QR Codes that the users present in front of their webcam either using specific software or simply on the advertising company’s website. An instance of such use was seen in December 2008, when MINI, the famous car company, ran an augmented reality advertisement in several German automotive magazines. The reader simply had to go to the MINI website, show the ad in front of their webcam and a 3D MINI appeared on their screen as if on top of the magazine, as seen in Figure 2.5. Beyond Reality [BR] decided to use a marker-less technique when they released a 12-page advertisement magazine with each page being recognized by their special software downloadable on their website. This AR advertisement magazine was published as the starting point to their Augmented Reality Games. They see that with such a system, they could add a “paid” option on the software that would allow the user to access additional content, such as seeing a movie trailer and
then being able to click on a link to view the full movie, thus turning a ma movie ticket [BR].
Augmented reality also offers a solution to the expensive problem of building prototypes. Indeed, the industrial companies are faced with the costly need to manufacture a product before commercialization to figure out if any changes should be made and see if the product
should be made, which is more often the case than not, a new prototype has to be manufactured and additional time and money are wasted. Creating a virtual prototype and introducing it into the real wo
and money as prototypes would be able to be changed quicker and created at lower cost since they would no longer involve materials. A group of the Institute of Industrial Technologies and Automation (IT
[SaMo] in Milan works on augmented reality and virtual reality systems as a toll for supporting virtual prototyping. The ITIA
contexts and application usi
development, and evaluation. Some examples of applied research projects where the
35
then being able to click on a link to view the full movie, thus turning a ma
Figure 2.5: MINI Advertisement
ty also offers a solution to the expensive problem of building prototypes. Indeed, the industrial companies are faced with the costly need to manufacture a product before commercialization to figure out if any changes should be made and see if the product meets the expectations. When it is decided that changes should be made, which is more often the case than not, a new prototype has to be and additional time and money are wasted. Creating a virtual prototype and introducing it into the real world would save companies considerable amounts of time and money as prototypes would be able to be changed quicker and created at lower cost since they would no longer involve materials. A group of the Institute of Industrial Technologies and Automation (ITIA) of the National Council of Research (CNR) of Italy ] in Milan works on augmented reality and virtual reality systems as a toll for supporting virtual prototyping. The ITIA-CNR is involved in the research for industrial contexts and application using VR, AR, realtime 3D, etc. as a support for product testing, development, and evaluation. Some examples of applied research projects where the then being able to click on a link to view the full movie, thus turning a magazine into a
ty also offers a solution to the expensive problem of building prototypes. Indeed, the industrial companies are faced with the costly need to manufacture a product before commercialization to figure out if any changes should be meets the expectations. When it is decided that changes should be made, which is more often the case than not, a new prototype has to be and additional time and money are wasted. Creating a virtual prototype and rld would save companies considerable amounts of time and money as prototypes would be able to be changed quicker and created at lower cost since they would no longer involve materials. A group of the Institute of Industrial of Research (CNR) of Italy ] in Milan works on augmented reality and virtual reality systems as a toll for CNR is involved in the research for industrial ng VR, AR, realtime 3D, etc. as a support for product testing, development, and evaluation. Some examples of applied research projects where the
above technology have been applied include motorcycle prototyping ( layout of a factory and an office
2.9), and virtual trial of shoes using the Magic Mirror system discussed next.
Figure 2.6: right picture: Picture of a virtual motorcycle prototype (on the right next to a physical motorcycle prototype (on the left) in the real environment;
picture: Virtual prototype in a virtual environment
Figure
36
above technology have been applied include motorcycle prototyping (Figure an office (Figures 2.7 and 2.8), virtual light simulation ( ), and virtual trial of shoes using the Magic Mirror system discussed next.
: right picture: Picture of a virtual motorcycle prototype (on the right next to a physical motorcycle prototype (on the left) in the real environment;
irtual prototype in a virtual environment [SaMo
Figure 2.7: Virtual factory prototype [SaMo]
Figure 2.6), virtual ), virtual light simulation (Figure ), and virtual trial of shoes using the Magic Mirror system discussed next.
: right picture: Picture of a virtual motorcycle prototype (on the right) next to a physical motorcycle prototype (on the left) in the real environment; left
Figure
Figure 2.
Shoes are the accessories that follow fashion trends the most and are renewed annually, especially for those who live in fashion capitals, s
Paris. For these people, it is often more important to wear trendy shoes than comfortable shoes. With the Magic Mirror system, t
augmented reality system which, combined with high
measurement, enables the user to virtually try on shoes prior to buying/ordering them. The user is able to see his/her reflection in the Magic Mirror with a virtual model of the pair of shoes s/he would like to try on
37
Figure 2.8: Virtual office prototype [SaMo]
2.9: Virtual testing of a lighting system [SaMo
Shoes are the accessories that follow fashion trends the most and are renewed annually, especially for those who live in fashion capitals, such as Milan, New York, and Paris. For these people, it is often more important to wear trendy shoes than comfortable shoes. With the Magic Mirror system, the ITIA-CNR of Milan [SaMo
augmented reality system which, combined with high-tech footwear technology measurement, enables the user to virtually try on shoes prior to buying/ordering them. The user is able to see his/her reflection in the Magic Mirror with a virtual model of the pair of shoes s/he would like to try on (Figure 2.10). The advantage of such a system over
sting of a lighting system [SaMo]
Shoes are the accessories that follow fashion trends the most and are renewed uch as Milan, New York, and Paris. For these people, it is often more important to wear trendy shoes than comfortable CNR of Milan [SaMo] has created an ootwear technology for measurement, enables the user to virtually try on shoes prior to buying/ordering them. The user is able to see his/her reflection in the Magic Mirror with a virtual model of the The advantage of such a system over
