CBIR Systems
Fabio Di Donna, Lucia Maddalena1, and Alfredo Petrosino2 1
National Research Council, ICAR Via P. Castellino 111, 80131 Naples, Italy
2 University of Naples Parthenope, Department of Applied Science
Via A. De Gasperi 5, 80133 Naples, Italy
Abstract. This paper focuses on the embedding of the uncertainty about color images, naturally arising from the quantization and the hu-man perception of colors, into histogram-type descriptors, adopted as indexing mechanism. In particular, our work has led to an extension of the GIFT platform for Content Based Image Retrieval based on fuzzy color indexing in the HSV color space. To quantify the performances of this basic system, we have investigated different indexing strategies, based on classical logics and fuzzy logics. Performance improvements are shown, in terms of effectiveness, perfect/good searches, number and posi-tion of relevant images returned, especially in the case of large databases containing images with noisy interferences.
Keywords:Content Based Image Retrieval, Image Indexing, HSV Color Space, Fuzzy Color Histogram.
1
Introduction
Content Based Image Retrieval (CBIR) has received increasing attention as a result of the availability of large scale image repositories in several domains, such as video surveillance, medical image management, multimedia libraries, art collections, geographical information systems, law enforcement agencies, and journalism. CBIR has been proposed to overcome the difficulties encountered in textual annotation for large image databases [5, 30]. Like a text-based search engine, a CBIR system aims to retrieve information that is relevant (or similar) to the users query, by addressing the problem of assisting a user to retrieve im-ages from un-annotated databases, based on features that can be automatically derived from the images. Today, there exist several CBIR systems based on dif-ferent methods, such as QBIC [11], Terraserver [36], VIR [37] or Excalibur [9], or a set of prototypes such as the Chabot and Galaxy’s projects from the UC Berke-ley [15, 26], MIT’s Photobook [27], CANDID [21], SCORE [3], VisualSEEK [32], or VORTEX [31].
Most of the research effort for CBIR systems has been focused on the search of powerful representation techniques for discriminating elements among the F. Masulli, S. Mitra, and G. Pasi (Eds.): WILF 2007, LNAI 4578, pp. 394–403, 2007.
c
global database. Although the nature of data is a crucial factor to be taken into consideration, most often the final representation is a feature vector extracted from the raw data, which reflects somehow its content. Most systems use color features in the form of color histograms to compare images [28, 34, 35, 39]. The ability to retrieve images when color features are similar across the database is achieved by using texture features [1, 16, 17, 24]. Other important attributes employed in comparing similarity of image regions are shape [6, 7, 10, 18, 19], spatial relationships [8, 20], or a combination of them [38].
The approach more frequently adopted for CBIR systems is based on the conventional color histogram (CCH), which contains occurrences of each color obtained counting all image pixels having that color. Each pixel is associated to a specific histogram bin only on the basis of its own color, and color similarity across different bins or color dissimilarity in the same bin are not taken into account. The consequence is that: a) CCH is sensitive to noisy interferences, such as illumination changes and quantization errors; b) large dimension of CCH involves large computation on indexing. These problems could be addressed by considering color similarity of each pixel’s color associated to all the histogram bins. If the color similarity is modeled through a fuzzy-set membership function, the representation leads to a fuzzy color histogram (FCH), like the one proposed in [13], although the real capabilities of such approach in the context of real CBIR systems are not completely clear, mainly when applied to large image databases. The usefulness of benchmarking is undeniable in the development of different algorithms, and recent attempts to benchmark CBIR systems in this respect have been made.
The paper is positioned in this context. We report the study made about the inclusion of uncertainty in color based indexing to augment the retrieval capa-bilities of the GIFT platform [12], an open source CBIR system. To quantify the benchmarking performances, we have investigated different indexing strate-gies, based on classical logics and fuzzy logics, on two different image databases. Performance improvements are shown, in terms of effectiveness, perfect/good searches, number and position of relevant images returned, mainly in the case of large databases containing images with noisy interferences.
The paper is organized as follows. The next Section reports the targeted CBIR and the operations involved within. Section 3 discusses the color histogram and fuzzy color histogram to make histogram based indexing mechanisms. In the last Section we present the experimental results and comparisons on two large image databases.
2
CBIR Reference Scheme
We adopt the method used inGIFT[12] for extracting local and global features and for retrieval as aprotocol. GIFT (GNU Image Finding Tool) is the outcome of theViperproject [25]. This open source software tool uses many well-known techniques for text retrieval and a large number of color and texture features, together with their frequency statistics. The feature sets used are:
1. global color features in the form of a color histogram using HSV (18 hues, 3 saturations, 3 values, plus 4 grey levels);
2. local color features at different scales obtained by partitioning the images (scaled to 256×256 pixels) successively into four equally sized regions (four times) and taking the mode color of each region as a descriptor;
3. local texture features by partitioning the images and applying Gabor filters in 3 scales and 4 directions. Gabor responses are quantized into 10 strengths; 4. global texture features represented as a simple histogram of responses of the
local Gabor filters in various directions and scales.
For the four feature groups two different weightings are used, depending on the term frequency tfij (frequency of occurrence of feature j in image i) and
thecollection frequency cfj (frequency of occurrence of feature j in the entire
database). Considering a query q containing N images with relevances Ri ∈
[−1,1], i = 1, . . . , N, the frequency of occurrence of feature j in the pseudo-image corresponding toqis tfqj= 1 N N i=1 (tfij·Ri).
The two global histogram features for each imagekare weighted according to a histogram intersection [35] as:
FeatureWeightkj = sign(tfqj)·min(|tfqj|,tfkj),
while the two block feature groups, that represent around 80% of the features, are weighted according to the inverse document frequency weighting:
FeatureWeightkj = tfqj·log2(
1 cfj
).
Then, ascoreis assigned to each possible result image k with featuresj, com-puted as:
Scorekq =
j
(FeatureWeightkj).
Scores are calculated for all four feature groups separately and then added in a normalized way.
3
Embedding of Uncertainty About Color in CBIR
Uncertainty about color similarity has been modeled through a fuzzy-set mem-bership function, so leading to a fuzzy representation of histogram and the in-dexing mechanism in the above described CBIR. Specifically, given a color space containing color bins, a fuzzy color histogram (FCH) of an image I containing N pixels can be expressed asF(I) = [f1, f2, . . . , fn], with
fi= N j=1 μijPj= 1 N N j=1 μij,
wherePj is the probability of a pixel selected from imageI being thejth pixel
(which is 1/N), andμij is the membership value of thejth pixel in theith color
bin. In contrast with CCH, FCH considers not only the similarity of different colors from different bins but also the dissimilarity of colors assigned to the same bin. Therefore, FCH effectively alleviates the sensitivity to the noisy interference. In order to quantify the perceptual color similarity, we consider Euclidean distance between colors represented in the HSV color space, which is perceptually uniform and therefore allows to obtain an accurate quantification of perceptual color similarity.
To compute the FCH of a color image, we adopt the method proposed in [13]. It consists in performing first a fine uniform quantization in RGB color space by mapping all pixel colors ton histogram bins, then transforming the colors from RGB to HSV color space. Finally, obtained colors in HSV color space are classified into nclusters (with nn) using fuzzy C-means (FCM) clustering algorithm [4], with each cluster representing an FCH bin. Through these steps, a pixels membership value to an FCH bin can be represented by the corresponding fine color bins membership value to the coarse color bin. Membership values need to be computed only once, and they are represented as a membership matrix M = [mij]n×n. Each elementmij inM is the membership value of thejth fine
color bin distributing to the ith coarse color bin. Thus, the FCH of an image can be directly computed from its CCH without computing membership values for each pixel. That is, given a CCHHn×1withn bins, the corresponding FCH Fn×1 withnbins can be computed asFn×1=Mn×nHn×1, where membership matrixM is pre-computed only once and can be used to generate FCH for each database image.
In order to insert FCH into GIFT, we needed to set quantization parameters n and n. As already mentioned in§2, the basic HSV color space quantization adopted in GIFT consists in 166 color bins. Experimental results showed that such quantization is too coarse for FCH; therefore we quantized the color space into n = 4100 color bins (16 for each HSV component, plus 4 gray levels). Moreover, in order to choose the value of the numbernof clusters, we conducted several tests on the considered databases, to obtain the optimal performance of the CBIR system (see§4).
4
Experimental Results
4.1 Performance Measures
The usualrecalland precision measures are quite inadequate in the context of CBIR, since they do not take into account all information returned by a database query, such as the position of returned images and their similarity degree with respect to the query image.
Among several performance measures, we choose the Effectiveness measure [14] normalized in [0, 1], defined as:
EF F =ef f−
R−1 2Er+R−1
1−2ERr+−R1−1 ef f =SumOptR/SumRAll where
– R is the number of relevant images in the database,
– Er is the number of images returned by the query,
– SumOptRis the optimal sum of positions of relevant images,
– SumRAll= (SumR+Er+ (Er+ 1) +. . .+ (Er+Mr−1))/R, – SumRis the sum of positions of relevant returned images,
– Rr is the number of relevant images returned by the query,
– Mr=R−Rr is the number of relevant images not returned by the query.
Such measure allows better than others to evaluate CBIR performance taking into account peculiarities of such systems, such as the position of images returned by a query. Moreover, as a measure of performance we considered also the number of:
– Perfect searches: searches which return all relevant images in first positions;
– Good searches: searches which return all relevant images in whatever posi-tion.
Even though a perfect search is the best attainable goal for a CBIR system, a good search is still an ambitious objective, since it means that all relevant images have been returned by the system.
4.2 Performance Evaluation
Performance of the various approaches has been tested on the public domain Stanford10K image database [33], consisting of about 10000 images, and on a database consisting of about 180 images from one of COREL’s CD-ROMs [2], in the following referred to as Alberta database. Both the databases have a predetermined set of images similar to some fixed images, so that it is possible to compare results and evaluate performance.
First experiments have been devoted to the choice of the numbern of clus-ters (see §3) in order to optimize performance. In Fig. 1 we report Effective-ness values obtained with FCH on the considered databases, varying n. In the case of Stanford10K database (Fig. 1-(a)), the maximum Effectiveness value EFF=0.31340 has been obtained forn=60, and it is better than the Effective-ness value EFF=0.30419 obtained using CCH. In the case of Alberta database (Fig. 1-(b)), the maximum Effectiveness value EFF=0.60431 has been obtained for n=50. However, in this case it is comparable (only slightly worse) to the Effectiveness value EFF=0.60518 obtained using CCH; this is mainly due to the fact that Alberta database has a small number of images, whose content is much simpler than that of the Stanford10K database, with no lighting intensity variations.
(a) (b)
Fig. 1.Effectiveness values obtained with FCH varying the numbern of clusters on: (a) Stanford10K database; (b) Alberta database
Moreover, using FCH strategy we achieved the same number of perfect and good searches obtained with CCH strategy (with optimal values forn) for both databases. Specifically, among the 32 queries predefined in Stanford10K database we obtained three perfect searches and five good searches, while among the 15 queries predefined in Alberta database we obtained one perfect search and five good searches.
The number of relevant images returned using CCH and FCH strategies was the same for almost all queries. In the case of Stanford10K database, query number 29 using FCH returned one more image than using CCH. Query image number 29 and relevant images returned with both the strategies are reported in Fig. 2. Here we can observe that the relevant image returned in position 17 by FCH was not at all returned by CCH. The content and the dominant color of such image is the same of all other relevant images; the only change, apart from shot position, is the lighting intensity. This observation confirms the superiority of FCH to CCH for image retrieval in terms of sensitivity to lighting intensity variations. In the case of Alberta database, query number 11 using FCH returned one more image than using CCH; analogous conclusions can be drawn analyzing results of such query (not reported here for space constraints).
Concerning exclusively the position of returned relevant images, we found that among the 32 queries for the Stanford10K database there were:
– 20 cases where positions are the same using CCH and FCH;
– 5 cases where positions returned with FCH are much better (more than 5 positions higher) than those returned with CCH;
– 4 cases where positions returned with FCH are better (more than 2 positions higher) than those returned with CCH;
– 3 cases where positions returned with FCH are worse (more than 2 positions lower) than those returned with CCH;
(a) (b)
Fig. 2.Query image number 29 for Stanford10K database and relevant images returned using: (a) CCH; (b) FCH
In particular, considering Query number 16 where both CCH and FCH strategies return 5 relevant images, FCH returned a perfect search, while one of the relevant images returned by CCH is in position 9.
As a final experiment, we compared best Effectiveness values obtained with FCH with those obtained using not only CCH, but also k-Means (KM) [23] and Deterministic Annealing (DA) [29]. In Fig. 3 we report Effectiveness values ob-tained with all the considered strategies on both the considered image databases. In the case of Stanford10K database (3-(a)) we can observe that FCH, KM and DA attain much higher Effectiveness than CCH, with FCH reaching the best Effectiveness value. In the case of Alberta database (3-(b)) we can observe that only FCH and CCH attain high Effectiveness values.
(a) (b)
Fig. 3.Effectiveness values obtained with different indexing strategies on: (a) Stan-ford10K database; (b) Alberta database
5
Conclusions and Ongoing Work
The paper concerned the inclusion of uncertainty about color in the HSV color space in the open source GIFT platform for CBIR and the benchmarking of different indexing strategies, based on classical logics and fuzzy logics, over two image databases, Stanford10K and Alberta. Specifically, the benchmarking has been made in terms of effectiveness, perfect/good searches, number and position of relevant images returned, also in presence of noisy interferences. The retrieval results are very encouraging in most cases and this proves that the use of un-certainty in CBIR is natural and desirable as long as human perception remains the key factor in judging and using the results.
References
1. Aksoy, S., Haralick, R.M.: Content-based image database retrieval using variances of gray level spatial dependencies. In: Ip, H.H.-S., Smeulders, A.W.M. (eds.) MI-NAR 1998. LNCS, vol. 1464, Springer, Heidelberg (1998)
2. Alberta database,http://db.cs.ualberta.ca/mn/CBIRone/
3. Aslandogan, Y.A., Thier, C., Yu, C., Liu, C., Nair, K.: Design, implementation and evaluation of SCORE (a System for COntent based REtrieval of pictures). In: Proc. of the 11thInt. Conference on Data Engineering, ICDE’95, pp. 280–287
(1995)
4. Bezdek, J.C.: Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum, New York (1981)
5. Del Bimbo, A.: Visual Information Retrieval, Morgan Kaufmann Publishers, San Francisco (1999)
6. Del Bimbo, A., Pala, P.: Visual Image Retrieval by Elastic Matching of User Sketches, IEEE Trans. Pattern Analysis and Machine Intelligence 19(2), 121–132 (1997)
7. Ciocca, G., Schettini, R.: Content-based similarity retrieval of trademarks using relevance feedback. Pattern Recognition 34, 1639–1655 (2001)
8. Deng, Y., Manjunath, B.S.: An efficient low-dimensional color indexing scheme for region-based image retrieval. In: Proc. on Intl. Conf. Acoustics, Speech, and Signal Proces. (ICASSP). 6, pp. 3017–3020. IEEE Computer Society Press, Los Alamitos (1999)
9. Excalibur Tech. Corp., Excalibur, Web (2001)
10. Fleck, M.M., Forsyth, D.A., Pregler, C.: Finding naked people. In: Proc. of the Europ. Conf. on CV, pp. 593–602. Springer, Heidelberg (1996)
11. Flickner, M., et al.: Query by Image and Video Content: the QBIC system. IEEE Computer 9(10), 23–32 (1995)
12. Gnu Fundation, The GNU Image-Finding Tool, http://www.gnu.org/ software/gift
13. Han, J., Ma, K.-K.: Fuzzy Color Histogram and Its Use in Color Image Retrieval. IEEE Trans. on Image Processing 11(8), 944–952 (2002)
14. Heczko, M., Keim, D., Weber, R.: Analysis of the effectiveness-efficiency depen-dance for image retrieval, DELOS Workshop, Zurich (2000)
15. University of California, UC Berkeley Digital Library Project, Web (2001) 16. Lin, H.-C., Wang, L.-L., Yang, S.-N.: Regular-texture image retrieval based on
17. Liu, F., Picard, R.W.: Periodicity, Directionality, and Randomness: Wold Features for Image Modeling and Retrieval. IEEE Trans. Pattern Analysis and Machine Intelligence 18(7), 722–733 (1996)
18. Mehrotra, R., Gary, J.E.: Similar-Shape Retrieval in Shape Data Management. Computer 28(9), 57–62 (1995)
19. Jain, A., Vailaya, A.: Image Retrieval Using Color and Shape. Pattern Recogni-tion 29(8), 1233–1244 (1996)
20. Kankanhalli, M.S., Mehtre, B.M., Huang, H.Y.: Color and spatial feature for content-based image retrieval. Pattern Rec. Letters 20, 109–118 (1999)
21. Kelly, P.M., Cannon, T.M., Hush, D.R.: Query by image example: the CANDID approach. In: Proc. of the SPIE, Storage and Retrieval for Image and Video Databases III 2420, SPIE, pp. 238–248 (1995)
22. Krishnapuram, R., Medasani, S., Jung, S.-H., Choi, Y.-S., Balasubramaniam, R.: Content-based image retrieval based on a fuzzy approach. IEEE Trans. on Knowl-edge and Data Engineering 16(10), 1185–1199 (2004)
23. MacQueen, J.B.: Some Methods for classification and Analysis of Multivariate Observations. In: Proc. of 5-th Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 281-297, University of California Press,Berkeley (1967) 24. Manjunath, B.S., Ma, W.Y.: Texture Features for Browsing and Retrieval of Image Data. IEEE Trans. Pattern Analysis and Machine Intelligence 18(8), 837–842 (1996)
25. Muller, H., Squire, D.McG., Muller, W., Pun, T.: Efficient access methods for content-based image retrieval with inverted files, in: Proc. Multimedia Storage and Archiving Systems IV (VV02), Boston, Massachusetts, USA, pp. 20–22 (1999) 26. Ogle, V., Stonebraker, M.: Chabot: Retrieval from a relational database of images.
IEEE Computer 28(9), 40–48 (1995)
27. Pentland, A., Picard, R.W., Sclaroff, S.: Photobook: Content-based manipulation of image databases, Tech. Rep. 255, MIT Media Laboratory Perceptual Comput-ing (November 1993)
28. Quddus, A., et al.: Content-based object retrieval using maximum curvature points in contour images. In: Proc. of the SPIE/EI’2000, Symp. on Stor. and Retr. for Media DB, SPIE, vol. 3972, pp. 98–105 (2000)
29. Rose, K.: Deterministic annealing for clustering, compression, classification, re-gression, and related optimization problems. In: Proc. of IEEE, vol. 86(11), pp.2210-2239 (1998)
30. Santini, S.: Exploratory Image Databases: Content-Based Retrieval, Communica-tions, Networking, and Multimedia. Academic Press, San Diego CA (2001) 31. Schonfeld, D., Lelescu, D.: VORTEX: Video retrieval and tracking from
com-pressed multimedia databases-visual search engine. In: Proc. of the 32nd Hawai Int. Conference on System Sciences, IEEE, pp. 1–12 (1999)
32. Smith, J.R., Chang, S.-F.: VisualSEEk: a fully automated content-based image query system. In: ACM Multimedia’96, Boston MA, USA, pp. 87–98 (1996) 33. Stanford10K database,
http://www-db.stanford.edu/∼wangz/image.vary.jpg.tar
34. Stricker, M., Orengo, M.: Similarity of Color Images. In: Niblack, W.R., Jain, R.C. (eds.) Proc. SPIE Conf. on Storage and Retrieval for Image and Video Databases III, pp. 381–392 (1995)
35. Swain, M.J., Ballard, D.H.: Color Indexing, Int. J. Computer Vision 7(1), 11–32 (1991)
36. Microsoft, Terraserver (2001)
37. Virage Inc., VIR image engine (2001), http://www.virage.com/products/ image vir.html
38. Zhong, Y., Jain, A.K.: Object localization using color, texture and shape. Pattern Recognition 33(4), 671–684 (2000)
39. Wang, J.Z., et al.: Content-based image indexing and searching using Daubechies’ wavelets. Int. Journal on Digital Libraries 1, 311–328 (1997)
