The field of visual augmented reality has recently experienced rapid growth, fueled by the growing popularity and technological advances of smart phones and virtual reality headsets [106]. Visual augmented reality overlays visual information, partic- ularly virtual objects, on real-world scenes. This concept of mixed real and virtual environments has been expanded to include haptics as well. Jeon et al. created a taxonomy to describe the possible combinations of real and virtual vision and haptics that comprise the realm of haptic augmented reality [47]. Users can touch a com- pletely real object, a completely virtual object, or a real object with virtual haptic sensations added to augment how it feels. A key challenge lies in creating a scene where the visuals and haptics work seamlessly together.
Many researchers have focused on the challenge of adding virtual objects to a visually real scene. In the majority of these systems, the user feels and interacts with the virtual objects through a force-feedback haptic device. Vallino and Brown created one of the first systems of this type, which allowed users to manipulate virtual objects that were visually overlaid on a physical environment [105]. A Phantom force-feedback device was used to display the shape and weight of the virtual objects. Aleotti et al. created a system for manipulating virtual objects using a Falcon force- feedback device [4]. This system allowed for both virtual and real objects in the
augmented environment, and it handled the real-virtual interactions between objects. Ye et al. created the VisHap system to interact with virtual objects using a Phantom force-feedback device [117]. Unlike many systems that include force-feedback devices, however, the user did not hold the Phantom. Instead, the tip of the Phantom followed the motion of the user and contacted his or her hand only when he or she made contact with a virtual object. This paradigm allowed for more distinct virtual objects because the user felt external sensations only when touching the virtual objects.
Haptic augmented reality systems have also been developed to virtually simulate specific tasks. Bianchi et al. created a haptic augmented reality ping-pong game [14]. This system involved a head-mounted camera to capture the real scene. The virtual object was superimposed on the image, and the augmented scene was then shown using a head-mounted display. Bayart et al. created an augmented reality system for virtual painting using an Omni force-feedback device to control the virtual tool and provide haptic feedback [8]. The Omni was visually hidden, and a virtual tool (i.e., a pencil or a paintbrush) was displayed on the screen. The user then virtually drew or painted on physical objects. Harders et al. created a haptic augmented reality medical simulator using a physical mock-up of a patient’s leg [40]. A portion of the physical leg was missing, which was then created virtually through the addition of graphics and haptics. Open surgery was simulated by allowing users to cut into the virtual portion of the leg using a Phantom force-feedback device.
feels through the addition of haptic feedback. For example, researchers have created systems that change the perceived friction between a tool and the physical surface. Jeon et al. modulated the friction of a physical surface using a Phantom force-feedback device [51]. The Phantom device included a force sensor attached at the tip, which measured the friction as the user dragged the tip across a physical surface. The Phantom created an additional lateral force to reach a target friction force. In another approach, Wintergerst et al. created a set of styli that could actively alter the friction between the stylus tip and a touch screen [114]. An electromagnetic coil was used to attract a rolling ball to the body of the stylus, which would increase the apparent surface friction. This friction control system was used to provide feedback during dragging gestures, including guiding the user, simulating virtual mass, and enhancing drawing by simulating different tools.
Researchers have also created haptic augmented reality systems to change the perceived hardness of physical objects. Jeon and Choi modulated the stiffness of a physical object using a Phantom force-feedback device [47]. A force sensor was at- tached to the tip of the Phantom to measure the user’s applied force. The Phantom was then used to generate additional force to increase the object’s perceived stiffness. This stiffness augmentation was extended to allow for three-dimensional interactions with physical objects [48,49], and it was further extended to allow the perceived stiff- ness to be changed while squeezing a physical object using two Phantom devices [50]. Hachisu et al. altered the perceived hardness of a physical surface by displaying tap-
ping transients using a voicecoil actuator [37]. Innate vibrations from tapping on the physical surface were removed by using an elastic sheet so that the physical hardness was affected only by the tapping transients.
Haptic augmented reality systems have also been created to change the perceived texture of physical objects. Bau and Poupyrev created the REVEL system to change the perceived texture of objects touched with a bare finger [7]. Texture was added using reverse electrovibration, which involves injecting a weak AC electrical signal into the user’s body, creating an oscillating electrical field around the user’s fingers. The user then touches physical objects that have been coated with an insulator- covered electrode. The properties of the AC signal determine the surface texture that is felt. Yoshimoto et al. altered perceived texture by displaying electrical stimuli through electrodes attached to the user’s finger [118]. Asano et al. changed the perceived roughness of textured surfaces by playing vibrations through a voicecoil actuator attached to the user’s finger [5]. The perceived roughness could be increased or decreased depending on the properties of the vibrations that are played.
Past work in haptic augmented reality has tended to focus on changing the per- ception of only one haptic attribute. This Chapter presents a novel haptic augmented reality system that alters both the perceived texture and friction of a physical object.