Design

4.4.2.1 Research Question R1

To address question R1 we perform four series of experiments using 9 local feature detection algorithms {Harris, Harris-Laplace, Harris-Affine, Hessian, Hessian-Laplace, Hessian-Affine, Maximally-Stable-Extremal-Regions (MSER), Difference-of-Gaussians (DoG), Laplacian of Gaussians (LoG) } and the SIFT descriptor algorithm:

1. The first experiment investigates the quantity of features detected with scale. 2. The second experiment series investigates detection repeatability over the whole

scale range when the algorithms are trained at different distances. For six training distances scale variant algorithms were compared against scale and affine invariant algorithms.

3. The third experiment series investigates 2D rotation of an object in the camera plane (rz) and rotation invariance in feature descriptors by comparing the

descriptor matching performance of rotation variant and rotation invariant algorithms under 2D rotation.

4. The fourth set of experiments addresses the general case of local feature detection performance with 3D object rotation (rx,ry) by comparing the detector matching

repeatability and descriptor matching performance of rotation variant and rotation invariant algorithms under 3D rotation.

4.4.2.2 Research Questions R2 to R4

R2 to R4 look at the performance of final four detection algorithms: Lab colour histograms, Mag-Lap multi-dimensional histograms for texture, shape context and SIFT for features on objects. We address R2 and R4 by performing another two series of experiments:

1. The first series evaluates the average detection performance over all objects of each algorithm, when the objects are scaled.

2. The second series evaluates the average detection repeatability for SIFT local features over all objects, and the average detection performance of the other three algorithms over all objects, when the objects are rotated.

4.4 SCALE AND ROTATION EXPERIMENTS

We investigate if scale and rotation is an issue, and compare the results between algorithms to identify whether some detection methods are more robust than others. From this we also gain insights into the training knowledge required to detect objects, addressing R3.

4.4.2 Procedure

For the local feature experiments, we define a procedure for calculating detection repeatability and descriptor matching performance as explained below.

To compare the relative performance of interest point detectors we use the repeatability criterion described by Mikolajczyk and Schmid [Mikolajczyk and Schmid 2004]. For this we compute the percentage ratio between the number of point or region correspondences (found between the current image from the scale set and the features found on the object in the training image) and the minimum number of points detected in both images. The correspondences are established by projecting points from each image into the other using a manually-annotated ground truth homography. We establish a correspondence when the following two criteria are met:

1. If the point locations are less than 1.5 pixels apart when projected.

2. Additionally, for scale and affine-invariant points, when the image region we project has an overlap intersection error εs < 0.4. This error corresponds to 40%

overlap error and is chosen as according to Mikolajczyk et al. as regions with 50% overlap error can still be matched successfully with a robust descriptor [Mikolajczyk, Tuytelaars et al. 2005]. The intersection error of the regions (εs) is

defined as the intersection and union of the regions:

) ( ) ( 1 A A A A b T a b T a s μ μ μ μ ε ∪ ∩ − = (4.1)

where μa and μb are the regions (represented as elliptical regions for affine-invariant

features or circles for scale-invariant features, and defined by xTμx = 1), A is the locally linearised ground truth homography relating the images and AT its transpose [Mikolajczyk, Tuytelaars et al. 2005]. The error can be computed numerically, or in our case by counting the number of pixels in the union and the intersection of regions when one region is projected into the second using the homography.

Relative matching performance of the descriptors is evaluated by establishing correspondences between descriptors in the test image and descriptors in the training image using nearest-neighbour Euclidean distance. Correct matches are determined by the overlap error (as for detector repeatability), but here we assume a match is correct if the overlap error is < 50% of the region (εs < 0.5). The final matching percentage score

(known as Recall) is calculated as the number of correct matches with respect to the total correspondences: ances Correspond # matches Correct # Recall= (4.2)

4.4 SCALE AND ROTATION EXPERIMENTS

4.4.2.1 Research Question R1

1. For the first series of experiments the 9 local feature algorithms were run on each image from the object appearance library scale set (see section 4.3.1) and the number of features detected inside the ground truth object location bounding box was recorded.

2. The second series is divided into 6 sub-experiments, corresponding to 6 algorithm training distances. Each local feature algorithm was trained at meter intervals between 1 and 6m (inclusive) using images from the object appearance library scale set. For training, feature detection was constrained to only detect features on the object by using the manually annotated ground-truth bounding box. Features were then detected in the remaining images in the scale set and the detection repeatability between the training and test images calculated using the method described above.

3. For each of the images of objects at 3m distance in the object appearance library scale set (see section 4.3.1) a 2D object rotation was simulated by rotating images in 10° increments between 0° and 350° using an affine transform. Simulation was used for this experiment as it was the easiest way to ensure accurate and repeatable rotations across all the objects without the need for specialized hardware to physically rotate the object. The SIFT descriptor [Lowe 2004] is computed for each interest point detected by the local feature detection algorithms in all images. The detection algorithms were trained with images of all objects at 0° rotation and matched against images of the objects at the other rotation angles to compare non-rotation invariant descriptors against rotation invariant descriptors.

4. The experiment is divided into 4 sub-experiments. The 4 sub-experiments are for 4 algorithm training distances, at meter intervals between 3 and 6m. In each sub- experiment the 9 algorithms were trained with the 0° rotation images of each object at the respective distance from the object appearance library rotation set (see section 4.3.1). These training images were matched to the -70° to +70° rotation images, comparing both scale invariant against non-scale invariant detector repeatability and non-rotation invariant descriptors against rotation invariant descriptor matching performance.

4.4.2.2 Research Questions R2 to R4

For the final 4 detection algorithms we define a procedure for calculating detection performance as detailed below.

For colour histogram detection we use a variation of Bradski’s CAMSHIFT approach [Bradski 1998]. The object histogram is first back-projected into the camera image and the bounding box of the largest blob is detected, representing the most likely object location. The CAMSHIFT algorithm is then used to refine the bounding box size and location based on the original object histogram.

For texture detection with multi-dimensional histograms we use an exhaustive search method, dividing each scale image into a grid of scale-adapted 2D windows of uniform size, with 25% partial overlap between each window. Each area’s histogram is calculated and matched against the object’s histogram. For object location use a mean- shift [Comaniciu and Meer 2002] clustering approach similar to Liebe and Schiele’s scale-adaptive method [Leibe and Schiele 2004] when match result maxima are greater

4.4 SCALE AND ROTATION EXPERIMENTS

than a pre-determined threshold. Hence, the 3D mean-shift clustering function (X,Y,Scale) acts as a Parzen window probability density estimation for the position of the object, allowing a more accurate location hypothesis. We increase the clustering window size as scale increases above 1.0 as the histogram matching results are spread over a larger area.

For shape detection we use a similar approach to texture, but restricted to single scale, due to the scale-invariance of our shape context algorithm.

For colour histogram, shape context and multi-dimensional histograms algorithms, correct detection was assumed when the detection bounding box had <50% overlap error with the ground truth bounding box from the test image library. For SIFT local features correct detection was assumed when a minimum of 8 features were matched to the training image using nearest-neighbour Euclidean distance matching and >50% of feature correspondences were correct.

1. The first experiment set investigates scale-invariance in the 4 detection algorithms and aims to quantify in what scale range we can repeatably detect an object. We performed 6 sub-experiments, each with a different training distance. All of the algorithms were trained with an image at meter intervals between 1 and 6m (inclusive), resulting in the 6 sub-experiments. The remaining images of the object appearance image library (99 images) were used for testing. The results for all four cues are averaged over all objects to give the percentage of all detected objects over the scale range (every 5cm).

2. The second set of experiments investigates 3D rotation. For each object we trained the algorithms using a single 0° image from the object appearance library. The remainder of the images between -80° (anti-clockwise rotation of object from 0°) and +80° (clockwise rotation) were used to evaluate the percentage of objects detected with each algorithm. We both trained and tested the algorithms with images of the objects at 3m distance as this is the centre of our working range.

4.4.3 Apparatus

A 3.4GHz dual core Pentium-4 computer running Windows XP SP2 was used for all experiments in this thesis. The detection algorithms were implemented in C++ using Intel OpenCV API for image processing

4.4.4 Results

We first present results for the first four experiments to investigate local feature algorithm performance with scaling and rotation, followed by results of the detection performance for the final four selected detection algorithms for colour, texture, shape and features.

4.4.4.1 Research Question R1: Local Features with Scaling and Rotation

Although we performed the experiments for all objects in the object appearance library, we only present the results for the Mediacube object below as the general trends for all objects are the same, hence, these results are representative of the other objects. Additionally, we only present results for the Harris-based family of detectors as the result trends are also representative of the relative Hessian-based detector performances.

4.4 SCALE AND ROTATION EXPERIMENTS 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 0 50 100 150 Distance (m) N um ber of F eat ur es

Number of Features Detected by Different Harris-Based Detectors

Harris Harris-Laplace Multi-Scale Harris-Laplace Scale-Invariant Harris Affine-Invariant 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 0 10 20 30 40 50 60 70 80 90 100

Repeatability of Interest Point Detection for Different Harris-Based Detectors

Distance from Camera (m)

R ep ea ta bilit y ( up t o 4 0 p er ce nt o ve rla p e rr or ) f or 4 .0 m T ra in in g I m ag e Harris Harris-Laplace Multi-Scale Harris-Laplace Scale-Invariant Harris Affine-Invariant

Figure 4.5 (left) Number of features detected on Mediacube object by Harris-based algorithms, (right) Harris-based algorithm detector repeatability when trained at 4m

As can be seen in Figure 4.5 (left), the number of points detected by all Harris based detectors decreases with distance, as the object appearance gets smaller in the image. The single-scale Harris detector always detects the least interest points as only the original image scale is used. In contrast, the Harris multi-scale detector detects all points where the Harris operator reaches a maximum in scale-space, leading to between 3 and 5 times more interest points throughout the whole object distance range. The number of points detected by the Harris-Laplace scale-invariant detector falls between the two, as it only detects points where the Laplacian also reaches a maximum. The affine-invariant Harris detector returns fewer points than the scale-invariant detector as the algorithm discards points which are not also invariant to affine transformations.

The difference in repeatability of the Harris-based detectors when trained with a single image of the object at 4m distance can easily be seen in Figure 4.5 (right). Here the single-scale Harris only has a small 2m range (3m to 5m) where points are repeatably detected, centred on the training distance. In contrast, multi-scale Harris- Laplace and scale-invariant Harris-Laplace have an almost constant performance across the scale range. Harris-Affine displays a general improving trend from 1m towards 6m, with a peak around the 4m training distance. For a 4m training image the scale factor ranges from 1:4 at 1m to 1.43:1 at 6m, with 4m being 1:1.

Figure 4.6 (left) shows that for the single scale Harris algorithm to achieve equivalent detection repeatability approaching the multi-scale and scale-invariant Harris-Laplace algorithms (over 60% throughout the test range), we must train and match the algorithm at 4 separate distances. The training images distances of 1.1m, 1.85m, 3m and 5m were determined empirically to have both minimum scale overlap between detectors and to maximise the repeatability across the whole scale range. This is equivalent to running the Harris detector algorithm 4 times independently on the image, with the consequent quadrupling of runtime.

The mean repeatability of the Harris-based algorithms over the whole distance range while varying the training data distance is shown in Figure 4.6 (right). Here we can see that both the multi-scale and scale-invariant Harris-Laplace algorithms have a stable mean repeatability around 91.22% and 75.12% respectively when the training image distance is varied. However, the training image scale makes a large difference both for Harris and Harris-Affine. The Harris algorithm has the highest repeatability (41.10%) when trained at 5m, and Harris-Affine reaches a peak of 74.98% when trained with images of the object at 6m.

4.4 SCALE AND ROTATION EXPERIMENTS 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 0 10 20 30 40 50 60 70 80 90 100

Harris Detector Repeatability with Varying Training Image Distances

Distance from Camera (m)

R ep ea tabi lit y ( up t o 40 per ce nt ov er la p er ro r) 1.1m Training Image 1.85m Training Image 3m Training Image 5m Training Image 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 0 10 20 30 40 50 60 70 80 90 100

Harris-based Detector Mean Repeatability with Varying Training Distances

Training Image Distance from Camera (m)

M ea n P er cen tage R epe at ab ili ty ( up t o 4 0 p er ce nt ov er la p e rr or ) Harris Harris-Laplace Multi-Scale Harris-Laplace Scale-Invariant Harris-Affine

Figure 4.6(left) Harris detector with multiple training images 1.1m,1.85m,3m,5m, (right) Mean repeatability with varying training distance for Harris-based detector algorithms

Figure 4.7 (left) compares rotation-invariant descriptors with rotation-variant descriptors, both with and without multiple training images. As shown by the red dashed line, the rotation-variant Harris-Laplace-SIFT matching result with only a single training image is only ever above 50% within 50° of the training image 0°/360° angle. In contrast, the rotation-invariant descriptor performance was more constant, with a mean average around 60% for the whole 360° rotation. Rotation-variant matching performance was also evaluated relative to multiple training images. In this case 60° rotation increments were determined empirically to produce performance similar to rotation-invariant descriptors, while using the least training images.

For 3D objects such as the Mediacube the number of features detected did not change significantly with 3D rotation, as seen in Figure 4.7 (right), due to textured surfaces on multiple faces being visible at each rotation angle. However, for 2D objects such as the notepad or objects where some surfaces lacked strong texture the number of points detected depended greatly on viewpoint. This led to large changes in the detector repeatability and descriptor matching percentages with 3D object rotation.

0 50 100 150 200 250 300 350 0 10 20 30 40 50 60 70 80 90

Mediacube: Matching Percent (up to 50 percent overlap error) with object at Distance 3m

Angle of Rotation (Degrees) with object at Distance 3m

M at chi ng P er cen ta ge ( up t o 5 0 pe rc en t ov er la p er ro r)

Harris-Laplace Rotation-Invariant SIFT Harris-Laplace-SIFT with 60° Ground Truths Harris-Laplace-SIFT -80 -60 -40 -20 0 20 40 60 80 20 30 40 50 60 70

80 mediacube: Number of Features with object at distance 4m

Angle of Rotation (Degrees) with object at Distance 4m

N um ber of F eat ur es Harris-Laplace Scale-Invariant Harris-Laplace Multi-Scale Harris Affine-Invariant

Figure 4.7 (left) SIFT Descriptor matching percentages with 2D rotation, (right) Number of features detected with 3D rotation

4.4 SCALE AND ROTATION EXPERIMENTS -80 -60 -40 -20 0 20 40 60 80 0 10 20 30 40 50 60 70 80 90 100

mediacube: Detection repeatability (up to 40 percent overlap error) with object at Distance 4m

Angle of Rotation (Degrees) with object at Distance 4m

R ep ea tabi lit y ( up t o 40 per cent ov er la p er ro r) Harris-Laplace Scale-Invariant Harris-Laplace Multi-Scale Harris Affine-Invariant -80 -60 -40 -20 0 20 40 60 80 0 10 20 30 40 50 60 70 80

mediacube: Matching Percentage (up to 50 percent overlap error) with object at Distance 4m

Angle of Rotation (Degrees) with object at Distance 4m

M at chi ng P er cen ta ge ( up t o 5 0 pe rc en t ov er la p er ro r) Harris-Laplace Rotation-Invariant Harris-Affine Rotation-Invariant Harris-Laplace Rotation-Variant Harris-Affine Rotation-Variant

Figure 4.8 (left) Detector repeatability for 3D rotation, (right) Descriptor matching percentages for 3D rotation

The repeatability of interest point detection with viewpoint transformation is shown in Figure 4.8 (left). As can be seen, for all algorithms the repeatability is generally at a maximum near the training image orientation (0°) and decreases with increasing rotation in either direction. In this graph the multi-scale Harris-Laplace algorithm performs best. Affine-invariance only leads to a small improvement in mean repeatability over scale-invariance of 37% against 34% respectively, over the whole rotation range. On average, between 20° and 40° from the training image orientation the repeatability has decreased to 50% or under.

The matching performance of SIFT descriptors with 3D object rotation is shown in Figure 4.8 (right). As can be seen, all algorithms have almost identical performance irrespective of whether the descriptor is rotation-variant or invariant.

4.4.4.2 Research Questions R2 to R4: Detection with Scaling and Rotation

To address questions R2 to R4 with the final 4 detection algorithms we first present the detection results for scale over all objects, followed by the results for 3D rotation:

Figure 4.9 (left) shows detection performance over all objects for SIFT local features. It is clearly visible that the percentage of objects detected falls below 50% after 4.5m distance when trained at 1m. For the chosen scale range, the average percentage of objects detected is highest when trained at 2m (M=82.31, SD=14.34). However, the detection percentage varies least across the scale range when trained at 6m (M=74.44, SD=9.32).

Figure 4.9 (right) shows detection performance over all objects for the Shape Context algorithm. Detection percentage is lowest when we train at 6m, never increasing above 30% across the scale range. The performance when training at both 1m and 3m both show a downward trend with similar detection performances, however, the highest percentage of objects were detected at when trained 1m (M=56.10, SD=19.71).