Content-Based Image Retrieval Using
Support Vector
Machine in digital image
processing techniques
Ramakrishna Reddy.Eamani
Anna University, Department Of Electronics And Computer Engineering Faculty Of Electronics And Computer Engineering
G.V.Hari Prasad
Usharama College Of Engineering & Technology , Department Of Electronics And Computer Engineering Faculty Of Electronics And Computer Engineering
Abstract
The rapid growth of computer technologies and the ad-vent of the World Wide Web have increased the amount and the complexity of multimedia information. A content-based image retrieval (CBIR) system has been developed as an ef-ficient image retrieval tool, whereby the user can provide their query to the system to allow it to retrieve the user’s desired image from the image database. However, the tradi-tional relevance feedback of CBIR has some limitations that will decrease the performance of the CBIR system, such as the imbalance of set problem, classification prob-lem, limited information from user problem, and insuffi-cient training-set problem. Therefore, in this study, we pro-posed an enhanced relevance-feedback method to support the user query based on the representative image selection and weight ranking of the images retrieved. The support vector machine (SVM) has been used to support the learn-ing process to reduce the semantic gap between the user and the CBIR system. From these experiments, the proposed learning method has enabled users to improve their search results based on the performance of CBIR system. In addi-tion, the experiments also proved that by solving the imbal-ance training set issue, the performance of CBIR could be improved.
1. Introduction
Images are considered as the prime media type for the use of retrieving hidden information within data [6]. In general, an image retrieval system is a computer system for browsing, searching, and retrieving images from a large
digital-images database [1]. In the early trend, image re-trieval utilized certain methods of adding metadata, such as captioning, keywords, or descriptions to the images [1], so that retrieval can be performed over the annotation words. Obviously, annotating images manually is a time-consuming, laborious, and expensive task for large image databases, and is often subjective, context-sensitive, and in-complete [1].
Thus, content-based image retrieval (CBIR), which is an-other method of image retrieval, attempts to overcome the disadvantage of the keyword-annotation method. The CBIR aims to retrieve images based on their visual similarity to a user-supplied query image or user-specified image features. The visual contents of an image, such as color, shape, tex-ture, and spatial layout, have been used to represent and index the image [1]. However, the performance of CBIR has been limited by several issues, such as subjectivity of human perception [6], similarity of visual feature [8], and semantic query-gap issues [3].
To solve these problems, interactive relevance feedback, which involves the interaction between human and system, was introduced. Relevance feedback is a supervised ac-tive learning technique, which uses the positive and nega-tive feedback examples from the users to improve system performance. For a given query, the system first retrieves a list of ranked images according to the predefined simi-larity metrics. Subsequently, the user marks the retrieved images as relevant (positive examples) or irrelevant (nega-tive examples) to the query. Thus, the system will refine the retrieval results based on the feedback and present a new list of images to the user. This process will go through several iterations until the user is satisfied with the retrieval result.
and re-weighting (discussed in ”Related Work” section). These problems include imbalance of training-set problem, classification problem, limited information from user problem, and in-sufficient training-set problem. Among these problems, im-balance training set issue will be the focus on this study.
In order to solve imbalance training set issue, we pro-posed a relevance feedback that use the representative im-age selection and weight ranking approach. This approach will adapt support vector machines (SVMs) for the purpose of supervised learning and retrieve the image according to the users’ preference. The related work on relevance feed-back based on CBIR is reviewed in Section 2 and the details of the proposed method is presented in Section 3. The ex-perimental setup and its results are given in Section 4, while in Section 5, we have discussed the results of the experi-ments. In the last section, the conclusion and the direction for future work are presented.
2. Related Work
The traditional CBIR relevance-feedback techniques in-clude query refinement [6] and re-weighting [9]. However, both techniques did not achieve a satisfactory performance for CBIR owing to several issues. One of the issue is how to incorporate positive and negative examples to refine the query or to adjust the similarity measure [1].
In fact, there has been a trend on CBIR researches focus-ing on the relevance-feedback issues. For instance, Cheng et al. [10] proposed a new relevance-feedback model suit-able for medical image retrieval; Qin et al. [11] proposed an active relevance-feedback framework to make the rel-evance feedback more informative, using unlabeled data in the training process; Das and Ray [5] provided a brief overview on feature re-weighting approach; Crucianu et al. [3] discussed the main issues related to relevance feedback for image retrieval as well as the recent developments in this domain, and Qi and Chang [4] introduced a compos-ite relevance-feedback approach for image retrieval using transaction-based and SVM-based learning.
According to earlier studies on SVM-based relevance feedback, the SVM is considered to normally treat the prob-lem as a strict binary-classification problem, without notic-ing an important issue of relevance feedback , such as the imbalanced training-set problem in which the negative in-stances significantly overwhelm the positive ones [7]. In this study, relevant or positive image is the image that is similar to the query image in user perception, meanwhile, irrelevant or negative image is the image that is dissimilar to the query image. Consequently, this problem will degrade the performance of the CBIR system when the number of labeled positive-feedback samples is small [2][8][11]. As a result, the information we gain from the user is very limited. In addition, the training samples (user’s labeled images) are insufficient because the users would only label a few im-ages, and unable to label each feedback sample accurately all the time [2][11]. This issue had been mentioned previ-ously as the limited information from user problem. Hence, a proper technique of relevance feedback that is adaptable with SVM is desired.
3. Proposed Method
In this study, the methodology consists of four main parts which are data collection, preprocessing, feature similarity measure and relevance feedback as shown in Fig. 1. How-ever, we focus on the relevance feedback part which has consists of five units. Among these units, we are highly emphasizing the use of representative image selection and weight ranking units. The benefits from these two units are:
1. Representative image selection unit is use to select a set of informative images from image database for the
use of labeling process. According to Qin et al. [11], it is an effective way for the user to do labeling work which it will keep the mass of labeling task very little but meaningful. Besides, if there are more relevant im-ages being retrieved, the possibility of imbalance train-ing set problem to occur will be minimized.
2. Weight ranking unit is use to provide a better way for user to do labeling. User will rank the retrieved
In CBIR, user will provide a query image and it will go through preprocessing process to extract its features content such as color, texture and shape. In this study, the query im-age will be segmented to several significant regions by using MAP segmentation technique [13]. In each image region, the regions features such as color and texture features will be extracted to describe the particular region. Generally, the images features selection is a very fundamental issue in designing a content-based image retrieval system. The combination of two or more features are best representing
❜₃₃₃Ž❜₃₃₃Ž❜₃₃₃₃
₃ ₃ Ž₃
Ž₃ ₃❜Ž₃₃₃₃
₃ ₃₃₃₃Ž!Ž₃ ₃ Ž₃
Ž₃₃₃ ŽŽ₃₃ ₃₃₃ ₃₃ Ž
₃Ž₃₃₃ Ž
₃₃₃₃₃₃Ž₃ĊŽŽ ₃₃#Ž!Ž₃₃₃₃Ž₃ŽŽ₃₃₃ŽŽ!₃₃₃ ₃₃ŽŽ₃ ₃Ž₃
Ž₃ ₃Ž ₃₃ŽŽ₃₃₃₃ ₃ ₃ ŽŽ Ž₃ ₃Ž ₃₃ ₃ ₃❜₃₃₃₃₃ ₃❜₃₃₃₃₃Ž₃Ž₃ ₃₃₃Ž₃
Ž₃₃₃ ₃₃Ž!Ž₃₃₃₃₃ ₃ŽŽ ❜₃₃₃₃₃ ₃Ž!Ž₃ ₃Ž₃
₃₃₃₃₃₃ ŽŽ₃ ₃ ŽŽ₃Ž₃ ₃₃Ž₃Ž!Ž
₃ ₃
Ž₃
❜₃₃Ž₃❜₃₃₃Ž₃❜₃₃₃ Ž₃₃₃
₃₃ ₃ ₃❜₃₃
₃₃❜₃ ₃₃❜Ž₃ ❜₃₃
₃Ž₃Ž❜₃Ž!ŽŽ₃ Ž₃Ž₃₃₃₃₃ ŽŽŽ
₃ ₃ Ž₃Ž
₃₃Ž Ž₃₃ Ž₃₃₃
₃₃ Ž₃Ž!Ž₃ ₃Ž₃Ž₃Ž$₃Ž₃❜Ž
Figure 1. Methodology of the relevance feed-back based CBIR by using SVM.
the images content and there are no single feature can rep-resent the whole content of an image perfectly. Out of all the available features, the combination of color and texture features is chosen to represent the image regions.
In this study, the haar wavelet filter and DWT techniques [12][15] and wavelet channel combination [16] will be used to extract the texture features. However,the average RGB values of all pixels which belongs to the region will be de-fined as the color feature [14]. The extracted features will be compared and ranked through feature similarity measure process.
In this study, Earth Mover Distance (EMD) will be used as the image similarity measure [17]. EMD is used to com-pute the dissimilarity between sets of regions and returns the correspondence between them [17].
After that, system will select a set of estimated possible positive image set (EPPIS), as an example, top 100 images from the ranked list will be classed as the EPPIS set. Its aim is to select a set of images that contain a subset of images from image database which are most likely relevant to the query [11]. In the representative image selection, system will select a set of representative images from the EPPIS set which have the minimum information loss when retrieve the images from EPPIS for user labeling process [11]. As a re-sult, the selected images for user labeling process will fulfill two behaviors which the image has the similar behavior in training the classifier and do not contain redundancy.
In the user labeling process, the selected representative images will be displayed and retrieved for the user to do labeling on it. In this study, user will give a sequence of images which is called the image ranking sequence and this sequence will be the feedback to the CBIR system. By us-ing another word, user will rank a sequence of relevant im-ages in order with respect to their similarity to the query image [10]. Thus, the retrieved images which are out of the image ranking sequence will be labeled as the irrelevant images. There are two types of user-defined preference re-lation which are > and = that will be used by the user to show the
image ranking sequence. As an example, if the similarity degree of IT1 and IT2 are same, then it will denote IT1
= IT2 . Meanwhile, if the similarity degree of IT1 is more than IT2, then it will denote IT1 > IT2 . Hence, the
leftmost to rightmost of image ranking sequence showing the most similar to less similar to the user desired image.
will use the following for-mula 1 [11]. For any given query images, their correspond-ing weight of region
features is F = {wc, wt } whereby wc is the weight of color feature and wt is the weight of texture feature.
1
wf =
γ(f )
(1) F
k₃=1 γ (1k)
where γ(f) = ₃x∈L,y∈L₃d(f) (x, y) 2 and d(f) is the ele-ments distance metric for region features among those rele-
vant labeled images.
As mentioned above, user is required to input the im-age ranking sequence according to the similarity degree among these retrieved relevant images. At the same time, each region also will be ranked in the ranking sequence by system according to their distance to the query image. In other words, the ranking sequence generated by a method is closer to the ranking sequence responded by the user if its feature is closer to user’s opinion.
The following formula 2 is the formula for re-weighting the weight of each region. It is used to evaluate how close the two sequence is [10].
₃ ₃
Rnorm (Δsystem , user ) = 1 1 + S+−S− (2)
2 Smax+
where system = rank ordering of labeled relevant images induced by the similarity values computed by an image
re-trieval system; user = rank ordering of labeled relevant
images by user; S+ = number of image pairs where a bet-ter image is ranked ahead of a worse one by system ; S−
= number of image pairs where a worse image is ranked
ahead of a better one by system ; Smax+ = maximum
possi-ble number of S+ from user .
According to Cheng et al [10], the Rnorm calculation
method is based on the ranking of images pairs in system
relative to ranking of corresponding images pairs in user .
The range of Rnorm values is from 0 to 1 and value 1 indicates that the rank ordering of system is same as the
user provided ranking [10]. As a result, Rnorm can rep-resents which part of region in the query image that user
pay attention to according to the value of R₃₃ i i ₃ norm. For a given query image IQ with m segmented regions as
IQ = RQ , wQ |i = 1, ..., m where each RQi re-gion has their corresponding weight, wQi . The system will estimate
Rnorm of each region, Rnorm = (r1 , r2 , ..., rm) after the user feedback the image ranking sequence to the system.
The new weight of each region will defined in for-mula 3.
wQi
ri
(3)
= ₃ jm==1 rj
In weight ranking process, the new weight of each re-gion and its region features has been calculated and ana-lyzed from users’ feedbacks. These weights will be updated and used for the next iteration for relevance feedback. Next, the CBIR system will classify the unlabeled images from image database into two classes which are relevant and ir-relevant by using the Support Vector Machine (SVM). In this study, two-class SVM will be used to do the classifica-tion purpose. The images which classified as the relevant images will be retrieved for the next iteration. In general, the relevance feedback mechanism as discussed above will be repeated iteratively until the retrieval result is been sat-isfied by the user. If the user is satisfied with the retrieved images, we can conclude that CBIR system has successfully formulated the user interest area and correctly re-weighting the weight of each region and its region features.
4. Experimental Results
4.1. Experiment Environment
In this study, the experiment used five categories of im-ages: animal, building, flower, fruits, and natural scene, as shown in table 1. There are 5 categories with a total num-ber of 1000 images in the image database and each category has 200 images. The results of our experiment were eval-uated by using the standard information retrieval measure-ment which are precision, recall and F1 [18].
Table 1. Categories of image database that used in our experiment
Category Category Name Amount
1 Animal 200
2 Building 200
3 Flower 200
4 Fruits 200
5 Natural Scene 200
Total 1000
5. Results And Discusion
The experiments were conducted by comparing the pro-posed method (labeled as proposed method) with two other conventional relevance feedback methods which are relevance-feedback based CBIR by Qin et al. [11] (labeled as conventional 1) and relevance-feedback based CBIR by Cheng et al. [10] (labeled as conventional 2). To conduct a fair experiment, the same techniques of preprocessing and feature similarity measurement will be applied at these three relevance feedback methods.
The performance of the SVM based relevance feedback became poor when the number of positive feedback is small [8]. According to Kim et al.[8], it is primarily due to two main reasons which are:
1. The SVM classifier is unstable when the size of train-ing set is small.
2. There are usually many more negative feedback sam-ples than the positive ones in the relevance feedback
process.
Besides, the poor performance of SVM classifier will in-directly cause the performance of relevance feedback based on CBIR to become poor. In order to reduce the imbal-ance training set problem, the proposed relevance feedback method is designed to retrieve more positive images for user labelling. In this project, the positive image means the im-age that is same category as the query image, otherwise, it is the negative image.
❜₃₃Ž❜₃₃Ž❜₃₃₃ ₃❜₃₃Ž₃₃₃ ₃Ž❜Ž ₃ ₃₃ ₃₃❜₃❜₃₃₃ Ž ₃❜ ₃Ž ₃₃Ž₃Ž₃Ž ₃Ž ₃₃₃❜ ŽŽ ₃ ₃₃Ž❜ ₃₃ ₃Ž❜ ❜₃❜ ₃❜❜ ₃₃₃ ₃₃₃₃₃₃Ž❜ ❜₃₃Ž₃ŽŽ₃
₃₃ ₃Ž ₃₃❜₃₃ ❜
# !Ċ$
₃Ċ& !Ċ''
₃Ċ% !Ċ'
₃Ċ$
₃₃❜₃❜₃₃₃
Ž
₃ ₃
₃₃ ₃
Ž
❜
Ž
₃ ₃Ċ# !Ċ#'
₃ ₃
Ž
❜₃
₃ !Ċ#
!Ċ& !Ċ₃'
!Ċ% !Ċ₃
!Ċ$
!Ċ# !Ċ!'
! !
₃ # ' $ ' %
❜₃₃Ž❜₃₃Ž ₃₃❜₃❜₃₃₃ Ž ₃ ❜Ž❜₃ ₃₃₃ ě₃Ž₃₃Ž❜₃₃ŽŽ₃ ₃ ₃₃❜₃❜₃₃₃ Ž ₃ ❜Ž❜₃ ₃₃₃ ě₃Ž₃₃Ž❜₃₃ŽŽ₃ # ₃₃❜₃❜₃₃₃ Ž ₃ ❜Ž❜₃ ₃₃₃ ₃₃₃₃₃❜₃ ₃₃❜₃₃ ₃ ❜Ž❜₃ ₃₃₃ ě₃Ž₃₃Ž❜₃₃ŽŽ₃ ₃ ₃ ❜Ž❜₃ ₃₃₃ ě₃Ž₃₃❜₃₃ŽŽ₃ # ₃ ❜Ž❜₃ ₃₃₃ ₃₃₃₃₃❜₃ ₃₃❜₃₃
Figure 2. Relationship between the posi-tive image percentage and F1 measurement among three different relevance feedback based on two-class SVM methods.
From figure 2, the trend of conventional 1 show that the performance of SVM classifier does not being affected by the imbalance training set. Although the SVM classifier of conventional 1 method does not being affected, the perfor-mance of conventional 1 relevance feedback based on CBIR is still poor. However, conventional 2 and proposed method shows that the performance of SVM classifier will be influ-enced by the percentage of positive images in the training set. The proposed method has a better F1 rate than the con-ventional 2 method. This shows that the proposed method able to retrieve more positive images for relevance feedback process. Hence, it can conclude that when more positive images appear in the training set, the SVM classifier perfor-mance will achieve a better result as shown by using F1 rate. Moreover, the improvement of SVM classifier performance will increase the performance of CBIR system. For infor-mation, the performance result of SVM classifier as shown in figure 2 is the SVM experiments results for fruit cate-gory. Therefore, the performance of CBIR results for this incremental of SVM classifier performance can be shown in figure 3. As a result, the increasing number of positive images in the training set will solve the imbalance training set issues. Then, the performance of SVM classifier will be increased while the imbalance training set has been solved.
In the end, the performance of relevance feedback based on CBIR will be increased while the SVM classifier per-formance is increase. Thus, the results demonstrate that a better relevance-feedback technique is necessary to improve the CBIR performance.
❜₃₃Ž❜₃₃Ž₃₃❜₃₃₃₃ ₃₃❜₃ ₃₃₃₃₃ ₃₃₃ ₃ ❜₃₃₃ ₃ ₃❜❜ ₃ ₃❜ ₃
₃ ₃Ž₃₃❜ ₃₃ ₃₃₃₃Ž₃❜❜₃ ₃₃₃
₃$ ₃#
₃₃
❜₃
₃Ċ ❜₃❜₃ ❜₃₃₃❜❜₃₃
₃! ❜₃❜₃ ❜₃₃₃❜❜₃₃
₃ &₃₃Ž₃Ž ₃ ₃ ₃
₃
! Ċ # $
Figure 3. Comparison of F1 rate for different relevance feedback method.
The experiment results also prove that the proposed rel-evance feedback based CBIR system is capable to select the positive images from the set database images. Figure 3 shows that the average F1 value for six iterations of rel-evance feedback are 0.30 for conventional 1, 0.32 for con-ventional 2 and 0.34 for proposed method. This shows that the proposed method is generally outperform from the other two conventional methods.
Other than that, the experiment results shows that the proposed and conventional 2 method has better perfor-mance in the first iteration of relevance feedback if com-pared to the conventional 1. This means that the representa-tive image selection can perform better than just retrieving the top N images for user labelling. The N images show the total number of retrieval in each iteration. As a result, the representative image selection unit is able to choose the sig-nificant and informative positive images from the database images rather than just follow the result of similarity mea-surement. Besides, the experiment results also shows that the weight ranking unit is capable to re-weight the weight of features more precisely in a more proper way. Hence, the proposed method is able to retrieve more relevant im-ages against the increasing of relevance feedback iterations. Lastly, the experiments also shows that the incorporation of representative image selection and weight ranking units is capable to increase the performance of CBIR system.
6. Conclusion
In this study, we proposed a relevance feedback based on SVM learning method to retrieve images according to the user preference. This proposed method has been used to support the learning process to reduce the semantic gap between the user and the CBIR system. Besides, it also aims to solve the imbalance training set problem in order to improve the performance of CBIR. Based on the experiment results, it shows that the proposed method achieved the best performance when it compare with two others conventional methods. In addition, the experiments also been proven that by solving the imbalance dataset issue, the performance of CBIR could be improved. Hence, the proposed method is capable to solve the CBIR problems.
7. Acknowledgements
This work was supported by the Ministry of Science & Technology and Innovation (MOSTI), Malaysia, and
the Research Management Center, Universiti Teknologi Malaysia (UTM), under the Vot 79227.
References
[1] F. Long, H. Zhang and D.F. David. ”Multimedia infor-mation retrieval and management - Technological fun-damentals and applications, chapter fundamentals of content-based image retrieval,” Springer, 2003.
[2] D.C. Tao, X.O. Tang, X.L. Li, X.D. Wu, ”Asym-metric bagging and random space for support vector machines-based relevance feedback in image retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol.28, n07, July 2006.
[3] M. Crucianu, M. Ferecatu, N. Boujemaa. ”Relevance feedback for image retrieval: a short survey,” Report of the DELOS2 European Network of Excellence (6th Framework Programme), October 10, 2004.
[4] X. Qi and R. Chang. ”Image retrieval using transaction-based and SVM-based learning in relevance feedback session,” M.Kamel and A.Campilho (Eds): ICIAR 2007, LNCS 4633, pp. 638-649, 2007 @ Springer-Verge Berlin Heidelberg 2007.
[5] G. Das and S. Ray. ”A comparison of relevance feed-back strategies in CBIR,” IEEE, 2007.
[6] Y. Rui, T.S. Huang, M. Ortega, and S. Methrotra. ”Rel-evance feedback: A power tool in interactive content-based image retrieval,” IEEE Trans. On Circuits and Systems for Video Technology, 8(5), 644-655, 1998.
[7] C.H. Hoi, C.H. Chan, K.Z. Huang, M.R. Lyu, I. King. ”Biased support vector machine for relevance feedback in image retrieval,” In Proceedings of Intl. Joint Conf. on Neural Networks (IJCNN’04), Budapest, Hungary, 2004.
[8] D.H. Kim, J.W. Song, J.H. Lee and B.G. Choi, ”Sup-port vector machine learning for region-based image re-trieval with relevance feedback,” ETRI Journal. Volume 29, Number 5, October 2007.
[9] Y. Rui and T.S. Huang. ”A novel relevance feedback techniques in image retrieval,” In: Proc. 7th ACM Conf. on Multimedia, pp. 67-70, 1999.
[10] P.C. Cheng, B.C. Chien, H.R. Ke, W.P. Yang. ”A two-level relevance feedback mechanism for image re-trieval”, Expert Systems with Applications, Volume 34, Issue 3, Pages 2193-2200, April 2008.
[11] T. Qin, X.D. Zhang, T.Y. Liu, D.S. Wang, W.Y. Ma, H.J. Zhang. ”An active feedback framework for image retrieval,” Pattern Recognition Letters, Volume 29, Is-sue 5, Pages 637-6461, April 2008.
[12] J.R. Smith and S.F. Chang. ”Automated binary texture feature sets for image retrieval,” Proc. ICASSP-96, 7-10 May.
[13] K. Blekas, A. Likas, N. Galatsanos and I. Lagaris. ”A Spatially-Constrained Mixture Model for Image Seg-mentation,” IEEE Transactions on Neural Networks, vol. 16(2), pp. 494-498, 2005.
Communication, 2001.
[16] J. Ohm, F. Bunjamin, W. Liebsch, B. Makai, K. Mller, A. Smolic, D. Zier. ”A Set of Visual Feature Descrip-tors and their Combination in a Low-Level Descrip-tion Scheme,” Signal Processing: Image Communica-tion 16, 157-179, 2000.
