Prototyping and evaluation

A major aspect of the methodology of this thesis is the prototyping and evaluation of ideas resulting from literature review or novel concept. Figure 2.1 shows how other elements of research feed into prototyping and evaluation. It also shows the methods that are used at successive stages of prototyping and evaluation. As the arrows in the diagram suggest, the process of prototyping and evaluation is pro- gressive. Concepts first undergo a feasibility study before they are implemented, and then may progress through to later stages of evaluation. A particular 3D re- construction approach will not necessarily proceed through all the listed steps, in fact it is unlikely that it will; concepts can be dropped at any of stages listed because they fail to meet requirements or expectations.

Feasibility study

A feasibility study precedes the implementation of a concept. Where the liter- ature review, or a novel idea has yielded a method or concept that seems to fit the requirements and meet some or all of the objectives, the feasibility is studied in more detail in the context of a final 3D telepresence system. The feasibil- ity of implementing a concept is considered in terms of the requirements of the concept that can be deduced from the literature and how well they fit with the available resources. Concepts evidently failing to meet the overall aim or objec- tives would generally be ruled out during the literature review, but this may not become clear until feasibility is considered in more detail. Available resources were an important consideration as this guided the direction of research based on what was achievable without significant outlay on new equipment. For exam- ple, a 3D reconstruction method that appears to fulfil the aim and meet some of the objectives, but requires 100 cameras might be ruled out during the feasibility study. Other resources analysed during the feasibility assessment were: estimated processing overhead, memory requirements and network bandwidth where appli- cable. Another means by which the feasibility study can rule a concept out before implementation is where the literature survey does not provide sufficient imple- mentation details. The feasibility study relies heavily on figures quoted in the literature and implementation overhead is only estimated. Therefore the outcome of the feasibility study cannot considered to be totally reliable.

Concept implementation

Concept implementation takes place when a method or concept has been deemed feasible by the feasibility study. The concept is implemented based on details derived from the literature or in some cases from novel thought and deduction. Apart from at the earliest stages in the research there is often a framework in place that can be reused for parts of the concept implementation. For example, all of the 3D reconstruction approaches that were implemented required the basics of obtaining images from cameras, and therefore such methods became core re-

CHAPTER 2. METHODOLOGY 23 usable components of a research platform that matured during the course of the overall project. Correct implementation of a concept requires an adequate degree of detail derived from the literature survey or deduced by logical thought. In some cases it does not become apparent until attempting concept implementation that there are insufficient details to complete the implementation. Often further literature searches using disparate sources to that from which the concept was derived were required to bridge these gaps and implement the complete concept.

Simulation

Figure 2.2: An early screen shot of the simulated setting through which many approaches to 3D reconstruction were evaluated. Shown here performing a volu- metric reconstruction of a synthetic dog model from 6 virtual cameras.

Given the nature of the research problem being investigated, and the time taken to set up and calibrate real cameras, simulation became an invaluable part of the research methodology. A simulator was implemented as an integral part of the developed research platform. The simulator allows virtual cameras to be placed

around a synthetic object and for implemented algorithms to be tested using im- ages derived from these virtual cameras, as shown in Figure 2.2. Many real-world problems are not present in the simulated setting: virtual cameras do not exhibit lens distortion, camera image noise is non-existent, and they are trivial to cali- brate compared with real cameras. Therefore, incorporation of a simulator into the research platform enabled rapid prototyping of both algorithms and camera placement that would be much more time consuming in a real-world setting. The first point of testing for algorithms progressing from the concept implementation stage was in simulation.

Comparative study

Comparative study results from the simulated testing of implemented algorithms. Algorithms can be evaluated and compared based on implementation details such as memory or processor overhead, or their ability to meet the spatial, visual or temporal qualities of the end system. Most algorithms could be effectively eval- uated during simulation or comparative study, only progressing to the next stages of evaluation if simulated testing and comparison did not rule them out.

Laboratory experiment

Laboratory experiment denotes the stage at which algorithms begin to be tested in a real-world setting. Real cameras replace the virtual cameras used in the sim- ulation setting, and consequently real-world problems become apparent; camera lens choice can lead to image distortion which needs to be corrected before recon- struction is attempted. Camera image noise can make background subtraction less reliable and predictable. Lighting and shadows add to the background subtraction problems. Differences in camera image exposure or colour balance become ap- parent. Real cameras are much more difficult to calibrate than virtual cameras, and calibration of them can take some time to get right. The simulated setting ef- fectively divorced the algorithm from all of these problems to do with the camera

CHAPTER 2. METHODOLOGY 25 acquisition stage of a 3D reconstruction system. It could be argued that this re- search could be conducted in an entirely simulated setting, focussing on algorithm performance and therefore temporal quality, and theoretically enabling improved spatial and visual quality. It would, however, be impossible to test with anything but synthetic human subjects, and consequently real human phenomena such as eye gaze could not be investigated.

Within the scope of this thesis, only one algorithm, EPVH was tested under labo- ratory experiment conditions. A number of variants of the algorithm were tested (sequential, parallel execution on CPU, parallel execution on GPU), but aside from differences in performance their output was identical.

Case study

A case study was used where an algorithm had been validated in real-world set- tings using laboratory experiment conditions, and progressed to being used as part of a true experiment. In this research, an experiment was conducted into the abil- ity of people to estimate the eye gaze of a 3D reconstruction of a real person. A number of participants manipulated the 3D reconstructed person until they felt it was looking directly at them. The angle of presentation to the viewer of the body, the head, and the direction of gaze of the eyes in the head, were varied over a number of gaze poses to determine which of these had an impact on gaze estima- tion. The impact of camera placement on form reconstruction and texturing was also investigated. Further details of this study are presented in Chapter 6.