Performance Evaluation

In this work, the overall ROC curve [13] is used to present the performance of the proposed method. In order to validate the feasibility of the proposed method, we compare it with the traditional averaging method in conjunction with two different SPN extraction methods, i.e., Basic [6] and BM3D [8]. In this work, we consider the situation that the number of reference images per camera is inadequate (i.e.

N <50), which is a case most current works do not take into account. Therefore, we estimate the reference SPN for each camera by only usingN = 15 and N = 30 images from the reference dataset. For each camera, the SPNs extracted from the 120 testing images of this camera are used as the positive samples and the SPNs obtained from the rest 1800 testing images of the other 15 cameras are deemed as the negative samples. Therefore, we would have 120×16 positive and 1800×16 negative samples from all the 16 cameras in total. To get convincing results, all these positive and negative samples are used together to draw the overall ROC curve.

False positive rate

10-3 10-2 10-1 100

True positive rate

0 0.2 0.4 0.6 0.8

1 Overall ROC curves, 256x256 pixels, N=15

Basic+Proposed Basic+Averaging

False positive rate

10-3 10-2 10-1 100

True positive rate

0 0.2 0.4 0.6 0.8

1 Overall ROC curves, 256x256pixels, N=15

BM3D+Proposed BM3D+Averaging

Figure 3.4: The overall ROC curves of difference methods with 15 reference images based on images with size of 256×256 pixels.

False positive rate

10-3 10-2 10-1 100

True positive rate

0 0.2 0.4 0.6 0.8

1 Overall ROC curves, 256x256 pixels, N=30

Basic+Proposed Basic+Averaging

False positive rate

10-3 10-2 10-1 100

True positive rate

0 0.2 0.4 0.6 0.8

1 Overall ROC curves, 256x256 pixels, N=30

BM3D+Proposed BM3D+Averaging

Figure 3.5: The overall ROC curves of difference methods with 30 reference images based on images with size of 256×256 pixels.

False positive rate

10-3 10-2 10-1 100

True positive rate

0 0.2 0.4 0.6 0.8

1 Overall ROC curves, 512x512 pixels, N=15

Basic+Proposed Basic+Averaging

False positive rate

10-3 10-2 10-1 100

True positive rate

0 0.2 0.4 0.6 0.8

1 Overall ROC curves, 512x512 pixels, N=15

BM3D+Proposed BM3D+Averaging

Figure 3.6: The overall ROC curves of difference methods with 15 reference images based on images with size of 512×512 pixels.

False positive rate

10-3 10-2 10-1 100

True positive rate

0 0.2 0.4 0.6 0.8

1 Overall ROC curves, 512x512 pixels, N=30

Basic+Proposed Basic+Averaging

False positive rate

10-3 10-2 10-1 100

True positive rate

0 0.2 0.4 0.6 0.8

1 Overall ROC curves, 512x512 pixels, N=30

BM3D+Proposed BM3D+Averaging

Figure 3.7: The overall ROC curves of difference methods with 30 reference images based on images with size of 512×512 pixels.

The overall ROC performance of different methods with respect to different image sizes and different numbers of reference images are shown in Fig. 3.4 - Fig. 3.7. In this experiment, Basic/BM3D+Proposed indicates that SPNs are extracted by using Basic/BM3D, and reference SPN is estimated by using the proposed SPN estimator; and Basic/BM3D+Averaging means that reference SPN is estimated by the traditional averaging method. In the real-world applications, a low false positive rate is usually required so as to ensure a low probability of wrong accusation. Therefore, in order to show the details of the ROC curves with a low FPR, the horizontal axis of all the overall ROC curves are plotted in the logarithmic scale. As shown in Fig. 3.4 - Fig. 3.7, the proposed method (red curves) outperforms the traditional averaging method regardless of the image size, the SPN extraction method and the number of reference images. It indicates that the proposed method is more reliable than the traditional averaging method on estimating reference SPN from a noisy image. Moreover, by comparing Fig. 3.6 with Fig. 3.7, we can see that the proposed method is more superior to the averaging method when the number of reference images is small (i.e., N = 15). These observations suggest that the proposed method can bring additional performance gains to a verification system when reference images are contaminated by scene details and the number of reference images is limited. It is worth mentioning that the BM3D method consistently outperforms the Basic method on the overall ROC performance. As mentioned in Section 2.2.1, this is because the BM3D method is superior to the Basic method on suppressing the impact of scene details.

Table 3.3 shows the TPR of different methods at a low FPR of 10−3. Sim- ilar to the observation in the overall ROC curve analysis, the proposed method

Table 3.3: The TPR (with the FPR fixed at 10−3) of different methods with respect to different number of reference images on different image sizes.

Method 256×256 512×512 15 30 15 30 Basic+Averaging 0.165 0.244 0.337 0.401 Basic+Proposed 0.234 0.329 0.416 0.439 BM3D+Averaging 0.276 0.348 0.442 0.498 BM3D+Proposed 0.319 0.405 0.555 0.549

consistently achieves higher TPR than the averaging method under all conditions. Moreover, we can see that when the number of reference images decreases from 30 to 15, the TPR of proposed method drops more slowly than the averaging method especially on the large image size (i.e., 512×512 pixels). For example, on the 512×512 pixels images, the TPR of Basic+Averaging degrades by 15.96% (from 0.401 atN = 30 to 0.337 atN = 15), while the TPR of Basic+Proposed only drops by 5.23% (from 0.439 at N = 30 to 0.416 at N = 15). More specifically, for B- M3D+Proposed on the 512×512 pixels images, the TPR atN = 15 is even slightly larger than that at N = 15. It implies that the proposed method is more reliable than the traditional averaging method when the number of the available reference images is limited.