Multi-sensory augmented reality

User experience (UX) is defined as “a person's perceptions and responses that result from the use or anticipated use of a product, system or service” ([53]). User experience is about how a person feels about using a system. The usability of the system is only one thing that affects user experience. The user and the context of the use influence UX as well. UX includes all the users' emotions, beliefs, prefer-ences, perceptions, physical and psychological responses, behaviour and accom-plishments that occur before, during and after use.

An AR system can expand the user experience by providing stimulus for other senses in addition to visual augmentation. A system can improve the immersivity of a mixed reality application with augmented 3D sound, scent, sense of touch, etc. In this section, we discuss the state-of-the-art of non-visual augmented reality and multi-sensory augmentation in mixed reality.

2.5.1 Audio in augmented reality

Audio has mainly been used in two different ways in augmented reality: as part of the user interface or for aural augmentation.

For example in our AR assembly demo [7], audio was used as one modality of the multimodal user interface. The user was able to give audio commands and the system gave feedback with audio signals (beeps). Interactive sound effects are used in the mobile phone version of the Dibidogs (Dibitassut) demo (mentioned in the previous section), for example, where the dog starts to growl if the user gets too close. This kind of non-directional audio is trivial from the technological point of view of audio processing. Yet even the simple use of audio brings a new dimen-sion to mobile applications.

For the visually impaired augmented audio can give a better understanding of the environment. A good example is LookTel [54], which is a smartphone applica-tion for the visually impaired (Figure 20). Augmented audio and the audio interface are only parts of its functionality. The system uses optical character recognition (OCR) and computer vision techniques to detect objects and read texts. The user may point at objects with the device. The application then reads the information aloud.

Figure 20. The LookTel smartphone application for the visually impaired recog-nises objects and characters, and reads aloud things at which the user points using the mobile phone (image courtesy of LookTel).

Similarly, the Hyperfit hybrid media application reads aloud nutritional information, which the system locates in a database [55]. Another similar application for the visually impaired is vOICe for Android [56], which adds sonic augmented reality overlay to the live camera view in real time. The vOICe technology is compatible with a head-mounted camera, in which case the system shares the view with the user. Sometimes the boundary between hybrid media and augmented reality is blurred. Hybrid media connects digital information with printed media or physical objects. Depending on how the connection is made and how the information is then presented it may be considered a sort of augmented reality.

Audio information can be geotagged in a similar way as any other information and then used for location-aware services such as Toozla [57]. Toozla can be described as an audio augmented reality browser. It works in a similar way to Layar’s Wikitude in that it uses location services (GPS) and then gives users audio commentary on the subscribed channel (similarly to Wikitude’s visual layers).

Possible channels are e.g. the Touristic channel for information about nearby landmarks and the Service channel for promotions and information about shops and businesses, a Weather Channel and a Chat channel. Toozla works on several platforms and phone models.

3D sound is a research topic of its own, and numerous 3D sound recording, creation and playing systems exist. In movie and home theatre systems, 3D sounds are an ordinary feature. However, there is a fundamental difference be-tween 3D sound in a film and that in augmented reality. In film, the desired relative position of the sound source is known beforehand and is fixed. In mixed reality applications, the user may be situated in any direction of the desired sound source position and in any orientation. This means in practice that the desired sound direction is known only after the user’s pose is calculated for each time step.

3D sounds are more explored in the virtual reality end of the Milgram’s mixed reality continuum than in augmented reality. Nevertheless, some studies cover the use of audio in AR, e.g. [58].

2.5.2 Sense of smell and touch in mixed reality

In closed-space environments, it is possible to control the environment and enrich the user experience by involving other senses such as the sense of smell, touch and warmth. Heilig invented the first multi-sensory simulator, called Sensorama, in 1962. Sensorama was a motorcycle simulator with visuals, sound, vibration and smell [59].

A more recent example of a multi-sensory environment is Pömpeli, a video space with multi-sensory user experience (see Figure 21) created at the Laurea University of Applied Sciences [60]. One setup is installed at Helsinki Airport, where tourists can look at videos of Finland. The visual experience is augmented with variety of scents, smells and wind blow that match with what is seen. In addi-tion, temperature and a lighting atmosphere adapt to scenes and actions in the video [61].

Figure 21. Pömpeli multi-sensory space with multi-touch video, audio, smell, wind and lights at Helsinki Airport.

These kinds of multi-sensory augmentations are more attractive for virtual envi-ronments than for mobile augmented reality, for example, where it is challenging to control the environment.

Building up a gustatory display is challenging because the perception of gusta-tory sensation is affected by other factors, such as vision, olfaction, thermal sensa-tion and memories. However, people have also tested augmented flavours. Users can be tricked into tasting a non-existent flavour using visual and olfactory clues.

An example of this kind of augmented flavour system is Meta Cookie [62], where users are given neutral tasting sugar cookies with marker decoration.

Figure 22. Meta Cookie setup: instead of neutral tasting sugar cookies, users see augmented cookies and are given corresponding smells image form [62].

The cookie is detected and virtually replaced with the flavour of cookie chosen by the user, e.g. chocolate, and the corresponding scent is emitted. The Meta Cookie system air pumps have seven kinds of scented air, which can be controlled in 127 increments. In addition, the system has the ability to emit fresh air.

Augmented flavours are still a future technology. Before smell becomes a fea-ture in a larger scale in augmented reality, the user interface must be improved:

the wearable scent producing system must be miniaturised for mobile applications (see Figure 22).