Software

The proposed system also includes software development. For the un-warping and stitching of the captured images, commercially available software was initially used. Photoshop is used for the conversion from polar coordinates to rectilinear coordinates for the one-shot system. The multi-shot system images can be stitched with Panotools and Hugin/PTGui. All of the post-processing after the commercial software stages is completed using custom algorithms in Matlab. A graphical user interface (GUI) was also developed for ease of use and faster algorithm development and testing. The GUI includes abilities for importing images from the hardware system. The images are then processed using signal processing techniques. Processes such as colour correction, image stitching and 3D surface reconstruction are included. An interest point detector was the first algorithm developed. Interest points are used throughout the system, and therefore the most appropriate interest point detector is necessary. A modified Harris detector is used in this system. The modifications allow the choice of the number of interest points returned. Colour correction algorithms are also necessary. Part of the proposed GUI is able to detect and edit colour changes across panoramic

images using diagonal, diagonal plus affine, linear and linear plus affine transforms. The GUI is also able to process stereo information to produce 3D surfaces. The GUI includes feature based and area based methods. Fig 4.8 shows a diagram of the software framework for easy reference. Fig 4.7 shows an example of part of the software framework.

Figure 4.7: Screenshot of software framework.

4.3 Correspondence

As discussed in Chapter 2, correspondence is an important consideration for the project, indeed it is an ongoing problem in most computer vision research. With-

Figure 4.8: A diagram of the software framework

out the correct correspondence information it is impossible to obtain accurate 3D information. Many approaches have been proposed and used for the correspondence problem. The robustness and computing time are the main challenges for the prob- lem. To develop a correspondence method robust to capture device, viewpoint and lighting variation, an interest point based method is proposed for the system. Corre- spondence between the panoramas is the most difficult and most important part of the 3D panoramic process. Correspondence is the linking of a point in one image with its corresponding point in the other image. There are two main methods of corre- spondence searching, area based methods and feature based methods. For this work a new interest point feature based correspondence method has been used. Interest points are points in the image where the pixel information changes two-dimensionally, for example at a corner, a t-junction, or a change in texture. Interest points can be detected by using equation 4.1:

∧ C =     ∧ I_x2 (I_x∧I_y) ∧ (I_xI_y) I∧_y2     R (x, y) = det(C)∧ −k.trace2(C)∧ R (x, y) > T hreshold⇒ Corner (4.1)

Where I_x and I_y are image intensity from horizontal and vertical direction Gaus- sian low pass filters.

Interest points will be used for image stitching. Their robustness to changes in lighting, viewpoint, scaling and rotation is well proven [45].

4.4 Content Based Colour Correction

One problem often encountered when capturing multiple images for stitching together is that of colour changes between images. Variables out of the users control such as lighting or capture device settings change. For example the two images shown in Fig. 4.9 are from consecutive images from a panoramic sequence and require, for aesthetic reasons, the same lighting results. The device settings have changed and therefore the colours have changed. A discussion of colour correction methods and the presentation of a new method are discussed in chapter 5. The colour correction methods discussed in chapter 5 are used in this system to combat any colour changes in the multi-shot system captured images.

4.5 3D Reconstruction

The 2D images that are captured using either the one- or multi-shot system do not have any depth information with the pixel data. The data stored includes colour information in a 2D array. The 2D array is filtered through a Bayer pattern and interpolated to form the RGB image. From this 2D information the system must be able to produce 3D results. This is achieved by using more than one 2D array of

Figure 4.9: Crops from consecutive images from a panorama sequence showing colour changes due to capture devices settings changes

image information. If a point in the scene can be acquired by more than one camera, and some information about the camera is known, then that point’s 3D position can be established. The more images that are captured, the more accurate the results will be, and also points hidden to some cameras from some positions will be visible in other cameras, therefore giving a larger 3D representation. A discussion of a new 3D stereo imaging technique is discussed in Chapter 7. Chapter 7 also investigates efficiency of interest point based 3D surface reconstruction.