Princeton Shape Benchmark Data set Results

The PSB data set contain models obtained from the World Wide Web and as such are prone to problematic meshes. As a result, the EGI Oct and both Hough methods are missing from these results as the quality of the models adversely eected the descriptors such that meaningful results were unable to be obtained.

Table 5.4 shows the ratio of between class variance and mean within class variance. The higher the value, the better as it means that members of the same class are grouped tightly together, but dierent classes are spread apart. It can be seen that the ShapeD2 and MD2 both give a very high ratio compared to other descriptors. The Cord Hist 5 and Cord Histogram (and to some extent Cord Hist 4) also give higher ratios than other descriptors, but the ratio is much closer to them. We would expect the Shape D2 and MD2 to perform better than the other descriptors. We would also expect the Area Volume (with the smallest ratio) to perform worse than the other descriptors.

The rst experiment compares the relative performance of the descriptors on the PSB data set. The Euclidean distance metric has been used as the distance metric. Table 5.5 shows the statistics (see Section 2.9 for descriptions of these) and Figure 5.4 shows the corresponding precision-recall curves. The best performing descriptors are the Shape D2 and the MD2 descriptors. The Combined Cord Histogram and Cord Hist 4 are the next best. The worst descriptors are the Cord Hist 1 and Area Volume descriptors, although they are still better than performing a random retrieval. The precision-recall graph show a similar ranking of results to the statistics.

Comparing these results to those in the Princeton Shape Benchmark (Shilane et al., 2004), we can see that the Shape D2 is one of the poorer performing descriptors in their comparison where as it is one of the better ones in our comparison. This means there is denite room for achieving greater retrieval performance with alternative descriptors. It is also worth noting that the two EGI based techniques perform better than the Shape D2

Chapter 5 3-D Content-Based Retrieval 83 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Precision Recall Area Volume

Cord Histogram (Lengths) Cord Histogram (First Principal Axis)

Cord Histogram (Second Principal Axis)

Cord Histogram (Second Kind)

Cord Histogram (Third Kind)

Combined Cord Histogram

ShapeD2

Modified Shape D2

EGI Sphere

MRG Random

Figure 5.4: PSB Data set: Descriptor Comparison using the Euclidean Distance

Descriptor Nearest Neigh-

bour

First

Tier SecondTier MeasureE- DCG Area Volume 0.042 0.142 0.178 0.074 0.384 Cord Hist 1 0.131 0.162 0.200 0.088 0.399 Cord Hist 2 0.239 0.206 0.266 0.124 0.453 Cord Hist 3 0.202 0.205 0.261 0.124 0.446 Cord Hist 4 0.280 0.224 0.284 0.135 0.470 Cord Hist 5 0.282 0.221 0.277 0.132 0.463 Cord Histogram 0.290 0.225 0.287 0.136 0.471 EGI 0.181 0.186 0.224 0.102 0.418 MD2 0.341 0.242 0.325 0.154 0.492 ShapeD2 0.336 0.246 0.327 0.156 0.492 MRG 0.250 0.184 0.218 0.103 0.417 Random 0.019 0.018 0.034 0.018 0.307

Chapter 5 3-D Content-Based Retrieval 84 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Precision Recall Bhattacharyya City Block Euclidean Histogram Intersection Chi Quadratic Kullback-Liebler (symmetric) Kullback Liebler(non-symmetric) Random

Figure 5.5: PSB Data set: Average Distance Metric Performance

Metric Nearest Neigh-

bour

First

Tier SecondTier MeasureE- DCG Bhattacharyya 0.120 0.160 0.199 0.087 0.400 Chi 0.224 0.199 0.250 0.117 0.440 City-block 0.215 0.193 0.244 0.114 0.435 Euclidean 0.205 0.189 0.238 0.110 0.431 Intersect 0.144 0.153 0.195 0.088 0.401 Kullback 0.156 0.168 0.211 0.091 0.401 Kullback-ns 0.205 0.189 0.238 0.110 0.431 Quadratic 0.207 0.189 0.239 0.111 0.431

Table 5.6: PSB Data set: Average Distance Metric Performance

in their comparison. One dierence between implementations is that our EGI descriptors store surface area whereas the PSB implementations do not.

The second experiment compares the relative performance of the distance metrics on the PSB base test data set averaged over all descriptors. Table 5.6 shows the statistics and Figure 5.5 shows the corresponding precision-recall curves. It can be seen that the metrics fall into two groups. The higher performing group contains the Chi, City-block, Euclidean, Kullback-Leibler (non-symmetric) and Quadratic distances and the lower per- forming group contains the Bhattacharyya, Histogram Intersection and Kullback-Leibler

Chapter 5 3-D Content-Based Retrieval 85 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Precision Recall Area Volume

Cord Histogram (Lengths) Cord Histogram (First Principal Axis)

Cord Histogram (Second Principal Axis)

Cord Histogram (Second Kind)

Cord Histogram (Third Kind)

Combined Cord Histogram

ShapeD2

Modified Shape D2

EGI Sphere

MRG Random

Figure 5.6: PSB Data set: Best Distance Metric for Descriptor

(symmetric) distances. The Chi distance gives the best average performance overall and the Bhattacharyya distance giving the lowest average performance overall.

Figure 5.6 shows the precision-recall curves and Table 5.7 shows the corresponding statis- tics for the best metric for each descriptor ranked on highest DCG score. In some cases multiple distance metrics scored exactly the same. The table shows that the Bhat- tacharyya and Chi distances are the best performers typically, with the City-block dis- tance the only other metric and that appears only once. This contrasts with the previ- ous results in Table 5.6 and Figure 5.5 where the Bhattacharyya distance was the worst overall distance metric. Although increases in performance can be observed compared to Table 5.5, the increase is only slight.

Table 5.8 shows the descriptor and metric combination that gave the list of classes the best performance (DCG). As can be seen, no one combination comes out on top for all cases. Most combinations are only good for a single class, and only a few combinations have a sizable number of good classes. The Shape D2, MD2 and MRG descriptors have more classes in general than the other Cords based descriptors. However, the Cords based descriptors have many classes spread out over the distance metrics. This suggests that for these descriptors, no one metric works bests, where as for the Shape D2 descriptor a smaller number of distance metrics have been best matches indicating that some metrics are better than others for this descriptor.

Chapter 5 3-D Content-Based Retrieval 86 Descriptor Metric Nearest

Neigh- bour

First

Tier SecondTier E-Measure DCG Area

Volume Bhattach-aryya / Chi

0.043 0.142 0.178 0.075 0.385 Cord Hist 1 Bhattach-

aryya 0.163 0.172 0.217 0.098 0.413

Cord Hist 2 Chi 0.246 0.217 0.278 0.131 0.465 Cord Hist 3 Cityblock 0.228 0.208 0.266 0.126 0.448 Cord Hist 4 Chi 0.303 0.234 0.297 0.142 0.477 Cord Hist 5 Chi 0.324 0.242 0.310 0.148 0.484

Cord

Histogram Chi 0.330 0.247 0.308 0.148 0.489

Shape D2 Chi 0.215 0.190 0.232 0.105 0.425

Modied

Shape D2 Bhattach-aryya 0.354 0.267 0.348 0.165 0.507 Shape D2 Bhattach-

aryya 0.352 0.269 0.348 0.166 0.509

MRG N/A 0.251 0.185 0.219 0.103 0.417

Table 5.7: PSB Data set: Best Metric for Descriptor - Based on highest DCG

Figure 5.7 shows an example tier image for the Shape D2 descriptor using the Euclidean distance metric (see Section 2.9 for a description). We can immediately see the diagonal is a solid black line showing the nearest neighbour is correctly matched as itself. We can also see while the human arms out class and the ghter jet class both show good within class matches, there is also a large number of between class matching, indicating that the Shape D2 descriptor has trouble distinguishing between the two. There is a large number of matches outside the diagonal class boundaries, showing that there is a large amount of confusion between classes and that there is still room for improvement. Table 5.9 shows the statistics for a few selected classes. There are far too many classes to present them all. The upper section of the table shows the statistics for some classes the the Shape D2 performed well on, and the lower half some classes the Shape D2 performed badly upon. A large dierence in performance can be seen between those classes that obtain good retrieval performance and those classes that obtain poor retrieval performance. The higher performing classes are reected in the Tier Image (Figure 5.7) as densely coloured class squares, where as the poorer classes are reected by empty class squares.

Chapter 5 3-D Content-Based Retrieval 87

Descriptor Distance Metric Classes

Area Volume Kullback Barren

Cord Hist 1 Bhattacharyya Satellite, Hand, Shelves

Cord Hist 1 Chi Mailbox

Cord Hist 1 Euclidean monster_truck Cord Hist 1 Intersect Helicopter Cord Hist 1 Kullback-ns monster_truck

Cord Hist 1 Quadratic Staircase

Cord Hist 2 Chi Stealth_bomber

Cord Hist 2 City-block Flying_bird, School_desk

Cord Hist 2 Euclidean Jeep

Cord Hist 2 Intersect Snake, Wheel, Gear

Cord Hist 2 Kullback-ns Jeep

Cord Hist 3 Bhattacharyya One_story_home

Cord Hist 3 Chi Commercial

Cord Hist 3 City-block Book

Cord Hist 3 Euclidean Pail

Cord Hist 3 Intersect Flying_saucer, Satellite_dish

Cord Hist 3 Kullback-ns Pail

Cord Hist 3 Quadratic Pail

Cord hist 4 Chi Sink

Cord hist 4 City-block Train_car

Cord Hist 5 Bhattacharyya Vase

Cord Hist 5 Chi Human, Fish, Axe, Face, Head Cord Hist 5 City-block Rectangular, hat

Cord Hist 5 Kullback Biplane

Cord Hist 5 Quadratic Billboard, Race_car

Cord Histogram Chi Enterprise_like, Skull, Skyscraper, Ship, Sedan, Semi Cord Histogram City-block Knife, Single_leg

EGI Sphere Bhattacharyya Two_story_home

EGI Sphere Chi Bench

EGI Sphere City-block Church, Slot_machine EGI Sphere Kullback-ns Tie_ghter

EGI Sphere Quadratic Ant

EGI-Sphere Kullback Hot_air_balloon, Standing_bird MD2 Bhattacharyya Chess_set, Human_arms_out, Sea_turtle,

Computer_monitor, Door, Cabinet, Ladder MD2 Chi Chess_set,Gazebo, Submarine

MD2 City-block Rabbit, Umbrella

MD2 Intersect Rabbit, Umbrella

MD2 Kullback Walking, One_peak_tent, Eyeglasses, Street_light

MD2 Kullback-ns Buttery

MD2 Quadratic Barn

MRG Dog, Geographic_map, Glass_with_stem, Newtonian_toy, Potted_plant, Conical, Large_sail_boat

Shape D2 Bhattacharyya Fighter_jet, Glider, Sword, city, Desk_chair, Handgun, Electric_guitar

Shape D2 Chi Shovel, Desktop, Dining_chair Shape D2 Intersect Desktop, Dining_chair Shape D2 Kullback Hammer, Horse, Fire_place, Hourglass

Chapter 5 3-D Content-Based Retrieval 88

Figure 5.7: PSB Data set: Tier Image for Shape D2 using Euclidean Distance