Research method

The research method is described first in the context of two senses. Considering resource allocation using bi-modal interactions allows better understanding of the experimental framework and facilitates the subsequent introduction of other sensory modalities. The use of visual and audio cues in bi-modal VEs is quite common and the majority of the research studies presented in chapter 3 make use of these senses. Since olfaction presentation is more primitive that audio and visual stimuli, a significant amount of work was conducted to enable the simulation and delivery of olfactory impulses. The intensity of olfatory stimuli is also examined at perceptual level to investigate di↵erences between the range of available concentrations that can be delivered by our available olfactory display and are perceptually distinguishable to the users. Finally, this section concludes by describing the research methodology of an experimental study that encompasses all the three senses and builds upon the experimental layout and observations obtained from the previous two experimental methods.

4.3.1 Research methodology for audio-visual interactions

The objective of this research is to study user allocation preferences when distribut- ing visual and audio resources while the available computational budget varies. Efficient schemas for carefully allocating computational resources are of significant importance both in industry and academia as they allow the rendering process to be adjusted depending on the importance of each of the delivered senses at any given time.

compared to audio, when experiencing a virtual world. This hypothesis is investi- gated through an experimental study. In this study, resource allocation is measured over a series of experimental scenarios and budget sizes whereby participants are asked to assign the given budget to experience better audio or visual cues. There- fore, the e↵ect of two independent variables, namely, experimental scenario and budget size, are considered on the percentage of the total budget that is assigned for visual quality. This percentage is the dependent variable of the experimentE1. The budget percentage given to update the audio quality can be deterministically given by the percentage devoted to graphics.

The objective of this experiment is twofold. Firstly, to investigate how the visual-audio importance relationship scales across di↵erent budget sizes that can be available to the user and secondly, to utilise the obtained experimental data for building a prediction model capable of estimating resource allocation depending on an input budget size and scenario.

The experiment was designed using rendered images and audio where the quality level of each of the two senses could be adjusted interactively using the controls of a Graphical User Interface (GUI). Physically-based simulations were used for the computation of both audio and visual stimuli. In order to make use of the obtained experimental data in actual applications, a statistical model was designed. This model takes the computational budget and scenario as inputs and provides an audio-visual ratio estimation. A second, complementary experiment, was conducted with a new set of participants and untested budgets to validate the model’s performance against actual participant selection preferences.

The process of rendering visual and auditory stimuli is described in chap- ters 5 and 6 (see also Figure 4.1) of this thesis along with the rendering background required for the explanation of the computation process followed in either of the two modalities. The description is separated in two di↵erent chapters for better clarity. Chapters 5 and 6 also define the metrics used for assessing quality for each of the two considered senses. The selected quality metrics are associated with cost functions for estimating the computational cost required for obtaining better quality levels in vision and audio. Before implementing the experimental studyE1, it was important to have a predefined set of stimuli that perceptually di↵ered from each other at each of the two considered senses. All the audio-visual stimuli needed to be presented in perceptually uniform steps that could elicit distinct visual/auditory di↵erences and would have di↵erent computational costs. In both vision and hearing there are methodologies to evaluate whether two stimuli are perceptually di↵erent either in the form of a computational model or using previously estimated JND thresholds

for some characteristic of the considered stimulus (e.g. contrast or brightness in vision and loudness or pitch in acoustics).

4.3.2 Introducing multiple senses in the experimental framework As explained in the previous chapters (see section 3.6), previous research has concen- trated mostly on exploiting perceptual interactions between the senses of sight and hearing to selectively render VEs with lower computational cost. The introduction of other sensory modalities has remained relatively unexplored in this area. Many senses in a VE can significantly increase the level of immersion and give a more genuine representation of real life scenarios. Efficient resource allocation schemas in multi-sensory VEs is more a necessity rather than a requirement as rendering all these sensory stimuli in high quality is considered computationally prohibitive and o↵ers no perceptual benefit to the users.

The introduction of more senses in the resource allocation framework includes challenges both at a technical and perceptual level. A technical challenge has to do with delivering physically correct stimuli intensities to the users synchronously with other defined modalities using specialised hardware. On perceptual level, every sense has its own properties that are required for identifying distinct di↵erences between delivered stimuli of increasing computational cost. Measuring quality for senses other than vision and hearing might not be as straightforward as quality is mainly evaluated subjectively and not objectively by the users. Also, there are no well defined standards that can assist a metric selection. In this thesis, extension of the resource allocation framework from bi-modal to tri-modal VEs is achieved by considering the sense of olfaction along with the senses of sight and hearing that were introduced earlier.

The sense of smell was included for two reasons. Firstly, smell intensity es- timation can be achieved through the simulation of smell transport in a VE via Computational Fluid Dynamics (CFD). CFD techniques simulate the physical pro- cesses that govern smell transport and estimate odour concentration at any point of the computational domain. Physically accurate concentration values are then directed to an olfactory display for delivery to the VE users. Smell transport sim- ulation and delivery issues are addressed in more detail in chapter 8 of this thesis. Secondly, the availability of an olfactory display assisted to estimate olfactory per- ceptual thresholds and search for an olfactory quality metric before introducing the possibility of delivering smell impulses in the resource allocation framework.

There is a lot of research in the visual and auditory domains regarding whether two given stimuli (images or sound tracks) are perceived perceptually dif-

ferent for the average viewer/listener [MDMS05, Gel09, JJS93]. In the olfactory domain, although there are references that point out the importance of the sense of smell in real life environments [HE96, YN06], the knowledge of the HOS’s per- ceptual characteristics is limited compared to the visual and auditory sensory sys- tems [BMVK14]. Perceptual thresholds for many odorous compounds have been estimated in previous work [SW55, MC80] focusing mostly in estimating detection thresholds and not JND thresholds.

The work presented in chapter 9 (see also Figure 4.1) examines whether numerical accuracy of an odour transport simulation has any perceptual impact on the perceived smell intensity sensed by the users of a VE. The hypothesis that better numerical accuracy in a CFD simulation does not elicit perceptual di↵erences to the users is examined through the implementation of a psychophysics experiment for investigating the JND intensity threshold for a reference smell. Specifically, in order to identify discernible di↵erences, the numerical accuracy of the CFD simulation is adjusted using finer spatial discretisation of the geometrical domain thus requiring further computational resources (see chapter 8). The selected tool for checking whether better spatial discretisation is necessary is the JND threshold. This is estimated for a range of concentration values via pairwise comparisons of smell stimuli delivered to the participants of the experiment using an olfactory display. 4.3.3 Research methodology for audio-visual-olfactory interactions The experimental study E2 extrapolates the resource allocation methodology to tri-modal VEs. The initial hypothesis suggests that better spatial discretisation refinements of the computational domain in smell transport simulations do not elicit any perceptual di↵erence to the average user. This hypothesis is experimentally validated with four experimental scenarios.

According to Jiang et al. [JCT95], odour intensity cannot be considered a valid measure of smell quality because higher concentration olfactory impulses do not necessarily mean better quality. This study argues that odour intensity can be classified to a scale system at which very intense smell stimuli are described as “in- tolerable” when people are sniffing them. This result is also intuitively understood from every day experience where very strong odours are not favoured by people despite how pleasant the smell is. Another limiting factor is also the fact that hu- mans evaluate olfactory stimulations in a highly subjective manner a↵ected mostly on previous experiences and familiarity with the exposed odour. This behaviour is called hedonic assessment and has been described in detail in section 2.4.2.

tion, as allocating resources is based on the quality of the stimulus that is delivered to the user and better quality stimuli should have higher cost allocated from the user’s total budget. As odour intensity does not imply better quality, the experimen- tal framework of the tri-modal experimental study builds on the same methodology proposed inE1 for visuals and auditory and it extends it by introducing the abil- ity to deliver olfactory impulses to the user in an ON/OFF fashion depending on his/her preferences. When the user “turns” the control for smell ON, a one second smell burst is delivered to the nose using an olfactory display device.

The methodology of the third experimental study (E3), considers the e↵ect of two independent variables, namely, scenario and budget size on three dependent variables. These are the percentage of the total budget devoted for graphics and audio quality (numeric variables) and the decision to receive physically accurate smell impuses (categorical variable) when the related control is enabled. The results of this experimental study are analytically given in chapter 10.

As is the case with experimentE1, a prediction model is proposed that can be used for allocating resources in tri-modal VEs. This model is designed using the collected experimental data and validated against participants’ selection preferences in a subsequent experimental study, denoted asE3V. The results of the validation

stage are also given in chapter 10.

4.4

Summary

This chapter describes the research methodology of the experimental studies pre- sented in this thesis. The implementation of a resource allocation experiment using two senses is considered first. This framework includes the delivery of audio-visual cues where the key idea is to allocate a given budget of resources depending on the required stimulus fidelity in each of the senses of vision and hearing. This exper- iment was denoted as E1. The process of evaluating numerical accuracy of smell transport simulations using human perceptual criteria is introduced next. The es- timation of the JND threshold for a reference odour is used to assess whether high spatial discretisation of the VE is necessary in smell dispersion simulation prob- lems. This study is denoted as E2 in this thesis. Finally, the sense of olfaction is considered along with visual and auditory stimuli in a multi-sensory VE setup. ExperimentE3 describes the implementation of a resource allocation experimental study where smell impulses are delivered to the user in a binary manner depending on a user’s choice to select the smell ON or OFF when distributing the given budget to the three modalities.