Infrastructure

As a large number of CV models are needed for the validation and verification of their AR RAL suitability, the Augmented Reality Remote Access Laboratory Development Console (see Construction of the Augmented Reality Remote Access Laboratory Development Console for details) was constructed. The AR RAL Development Console

provides the necessary infrastructure to perform all the necessary CV model trials. All CV models were designed and implemented using the C# programming language. 6.1.1. (a) Model Structures

Approximating mathematical models, function and algorithm in software can be problematic. Reliably reproducing the purpose of a CV model is one issue, with the other on the interpretation of functionality when applied to the two-dimensional image data sets. Each CV model involved in this research has been interpreted and written into software code.

6.1.1. (b) Convolutions

Convolution theorem provides an efficient short-cut in the complex and time- consuming image transforms within the Time-Space domain, and allow for simple multiplications in the Frequency domain space [3]. A significant portion of the CV filters from this research are first, second or partial derivatives which must be applied to the two-dimensional digital image data set. Representation of common derivatives can be performed quickly using convolution theorem. The improvement in computational cost can be seen by the time complexity improvements, where applying CV filters in the Time Space Domain has a time complexity cost of _O

( )

yet only

(

)

O log for convolutions in the Frequency Domain. The AR RAL Development Console includes a range of generic convolution functions to apply kernel matrixes to the current image memory array.

6.1.1. (c) Matrixes

Mathematical transforms with digital images require a kernel as an operator in the convolution process. Kernels for convolution are matrixes, and are implemented in this research as arrays of object type double. The Laplacian kernel shown in Figure 6-9 is

represented in C# code shown in Figure 6-1. Matrixes of the form used in this research, are compatible with memory mapped images, so as to improve the performance of all CV processes.

A large number of CV processes are supported by the AR RAL Development Console in matrix format. Computer Vision image filters such as: Gaussian, Laplacian, and Mean as well as edge detection systems; Sobel, Prewitt, Kirsch, LoG and other derivatives are all represented as matrixes in C# code. A series of other image processing tools are represented in software matrixes as well, such as blur and sharpening functions.

6.1.1. (d) Model Attributes

Many CV models require adjustment or fine-tuning with user selectable values. As hybrid models are considered as part of this research, the stacked nature of hybrid models means that a single set of attributes are insufficient. Each CV model also contains its own complete list of user selectable attributes. The ability to quickly and instinctively adjust model parameters was also important. For this reason, a parameter interface was added to the console, and shown in Construction of the Augmented Reality Remote Access Laboratory Development Console Figure K-2. Values stored within the parameter interface are associated with the current active model and session of tests. 6.1.1. (e) Software Operations

The AR RAL Development Console provides a software framework for the realisation of the CV models listed in Computer Vision Model Summary Table F-. Testing each CV model occurs through the construction of a program. The program is a sequential list of CV models (Actions) to be applied to the current image. Each element (CV model) of the program list is dispatched to the underlying processing sub-systems, in sequence. Figure 6-2 shows a program of three ‘stacked’ actions, two of which involve convolution operations. Processing sub-systems call upon generic modules to perform the majority of image filtering or analysis such as Convolution processes.

Image or frame processing is summarised in Figure 6-3 and involves first converting the bitmap representing the current frame or test image into a byte array within main memory. This provides at least an order of magnitude speed increase when reading and writing the image attributes such as pixel colours. All operations and tests on the current image/frame are performed against the image memory array. Which CV model is to be

applied against the data array is decided by a large series of logic switches. Each model type of operation is grouped into enumeration families, shown by Computer Vision Model Summary Table F-. Functionality allows multiple re-entries into the logic switch, based on the CV models (Actions) listed within the current session’s program.

Once all selected Actions have been applied to the image data array, the array is converted back to a bitmap format for displaying. Testing CV models will generally return the results of the model, as a visual verification of the model’s effect. Operationally, the result should continue to be the original image, and within an AR environment, the returned image would include any visual enhancements. Additionally, to the resultant bitmap image, data may be available such as tracking information, or key reference/feature points.

6.1.1. (f) Model Attributes

Figure 6-3. AR RAL Image data processing pathway Figure 6-2. Standard experiment view (Shows 'stacked' program)

Default parameters are initially loaded when the application loads, and variations for different model selections (such as the weighting for a 3 x 3 or a 5 x 5 Gaussian convolution kernel) are automatically changed. Manual adjustments are instantly applied to the current image/frame, which provides an instant view of the results of such a change. Parameters are parsed to the various CV models (Actions) listed in the program of the session object, which follow through the logic switch.

Every function accessed through the logic switch consists of a set of generic parameters and function specific parameters. The ability to stack multiple models relies on each function returning a byte array consisting of the memory mapped bitmap image. Each CV function requires, as a minimum, pointer to the memory mapped image (sourcebuffer) and the attributes of the image including the dimensions, the colour depth (bytesPerPixel) and number of bytes per row (stride).

Function specific parameters are supplied by the attribute collection associated with the current test.

6.1.2 Validation and Verification

This section describes sound methods for performance evaluation of the CV image analysis models. Experimentation aspires to discover Computer Vision models which are sufficiently robust as a foundation for supporting AR functionality within the RAL framework, to identify and track objects, and which do not rely upon prior knowledge such as fiducial markers

6.1.2. (a) Methodology

Trials for all CV models (and combinations) follow the same pathway for their creation and execution. Attributes required for each trial must be applied prior to the execution of the trial. The Attribute panel of the AR RAL Development Console is preconfigured with the most conventional model settings, but may require adjustment for some trials. As each trial is created, the session is saved to store the configuration of the trial. The setup and execution stages are described in terms of the operation of the AR RAL Development Console operation.

byte[] functionName(byte[] sourceBuffer, int width, int height, int bytesPerPixel, int stride)

6.1.2. (b) Setup

Figure 6-4 defines the stages to create a new trial. To create a new trial, the user is required to select File | New Session from the AR RAL Development Console. The list of actions is now available for selection. The user should now select the file name from the File | Load Image menu selection. The image will be displayed in the console. The complete list of attributes for each CV image analysis trial are listed in Configuration. The user is expected to select from Configuration Table H- for each trial.

Each test consists of a program: built from individual CV image filters or analysis models to be applied to the current image. The user must now construct that program and select the necessary Actions to be applied. As the user double clicks the required Action (see Figure 6-4), the action is added to the test program.

At this stage, the session is saved to maintain the trial parameters. The filename for the saved session is as listed in Schedule of Tests with a suffix of *.AR RAL.