Proceedings of Int’l Symposium on MM Information Processing (ISMIP’98). Taiwan, Dec 98. pp 182-192.
Towards Pseudo-object Models for Content-based Visual Information Retrieval
Tat-Seng Chua and Mohan Kankanhalli School of Computing
National University of Singapore Lower Kent Ridge Road, Singapore 119260
Tel: +65-874-2726 Fax:+65-779-4580 E-Mail: {chuats, mohan}@comp.nus.edu.sg Abstract
Current image/video retrieval systems rely mainly on the visual features and text annotations of images as the basis for retrieval. A typical feature such as the color histogram only attempts to capture the main characteristics of the overall image. Although it is able to retrieve a large number of relevant images, its overall retrieval effectiveness is limited. Recent enhancements look into the use of pseudo object models with relevance feedback. This paper examines a number of these models to support content-based image retrieval. It describes the development of a color-based pseudo-object model with relevance feedback. The paper also discusses the design of an object- based video-indexing scheme to model video contents.
Keywords: Content-based visual information retrieval; pseudo-object model; stratification model;
relevance feedback.
1 Introduction
The introduction of MPEG-1 and – 2 video standards have accelerated the widespread use of digital video in our daily life. It permits video to be integrated with computer and networking technologies. It brings us a step closer to realizing the dreams of integrating broadcasting, computer and telecommunication technologies [20] to support advance innovative applications that may have profound effects on our everyday life.
The impending introduction of fully digital consumer and broadcast video has raised expectation that video could be better managed and manipulated to serve the users. Video is a rich source of
“live”, on-line information. However, it is also a sequential medium, which means that the users must put up with the often laborious and time-consuming process of browsing video to look for information of interests. Being digital means that we are able to develop more effective techniques to analyze, index, manage and manipulate video contents to support the users in ways that are not possible with analog video. Modern users should have the ability to search and summarize video contents, and to fast-forward video based on semantics. These are the subject of current efforts on MPEG-4 and MPEG-7 video standardization. To realize these functionalities, we need to develop better ways to index video and identify objects in video.
Recent image retrieval systems have examined the use of combinations of visual and semantic features such as the color, texture, shape, and text annotation for retrieval [9, 13, 26]. Most of these
features such as the color histogram [29] only attempt to capture the main characteristics of the overall image. Although it is able to retrieve a large number of relevant images, its overall retrieval effectiveness is limited. Recent enhancements look into the judicious use of domain knowledge [9]
and the application of pseudo object model with relevance feedback [5]. In particular, the pseudo object models [3, 18, 22] attempt to capture local object information and thus permit more accurate modeling of images' contents. Together with relevance feedback, a powerful scheme can be developed to extract and index local object information within visual contents.
The modeling of video has followed the traditional segmentation approach of first breaking the video into shots, and model the contents of the shots and their higher level entities individually [8].
The stratification approach [1] was proposed in the early 90s to model video as multiple strata.
Each stratum corresponds to a simple concept (object), event or category. The overall content of the video is the sum of all strata. It is a powerful model, one that corresponds to modeling video contents in terms of individual occurrences of objects/events.
This paper examines the object-based modeling of video content, and the development of better model to extract and index local object information within the video content. It discusses a number of pseudo-object models and the use of relevance feedback to extract local object information. In particular, it describes the development of a color-based pseudo-object model with relevance feedback. The paper also discusses the design of an object-based video-indexing scheme (based on the stratification approach) to model video contents.
2. Review of Pseudo-Object Models and Relevance Feedback Techniques for Content-based Image Retrieval
Pseudo-Object Models
Most early image retrieval systems use the keywords or free-text descriptions of images supplied by the authors as the basis for retrieval [4, 23]. Recent image retrieval systems [13, 28] employ the visual content of images as the basis for retrieval. The most common visual content features used is the color which can be represented in a variety of forms. One popular representation is the color histogram [16, 22, 28, 29]. The histogram technique is a global method without any local information, so images with very different appearances can have similar histogram [31]. The use of global features to model images’ contents is crude, and could only model the overall content of images approximately. However, it has been demonstrated to be effective in retrieving a large proportion of relevant images.
To further enhance retrieval effectiveness, recent approaches attempt to model the spatial information of overall images [28, 31], and/or to extract local object information [3, 18, 22]. One of the early pseudo-object models is the color-pair model [18]. It is developed to search for appearances of objects in video. It selected a set of distinct color pairs from a query image frame to match against the contents of video in the database. The color pairs model the distinct boundaries and relationships between objects in an image frame and thus it is a pseudo-object model. In order to eliminate the contributions of image background and to perform object-level retrieval, [6]
considered only color-pairs within object segments in images. Coupled with the use of perceptually similar colors, substantial improvement in retrieval effectiveness was obtained.
The color coherent vector (CCV) [21, 22] was proposed as a refinement to the color histogram method by incorporating color-spatia l information. In CCV, the pixels within a given color bucket are split into two classes based on spatial coherence. A coherent pixel is part of a sizeable
contiguous region, while an incoherent pixel is not. In general, coherent regions tend to correspond to part of objects within an image while non-coherent pixels tend to come from image background.
Thus the coherent color histogram can be regarded as a pseudo object representation. By using CCV, we can prevent coherent pixels in one image from matching non-coherent pixels in another.
This allows fine distinctions that cannot be achieved by using color histogram alone.
[3] represented image contents as a set of “blobs”. Each blob represents a region of the image that is roughly homogeneous with respect to color and texture. A blob is represented as a combination of color, texture and shape descriptions within the region. The blobs recognize the images as combinations of objects, and thus querying in blobs is more meaningful than using global features.
Relevance Feedback Techniques
Another way to improve retrieval effectiveness is to employ relevance feedback (RF) techniques. In a typical RF process, user's relevance judgments of retrieved images are used to refine the query.
The RF techniques work on the premise that relevant images are similar to each other. Hence relevant judgment data can be used to derive new query representations that are able to retrieve more relevant images in subsequent retrievals.
Relevance Feedback techniques were introduced in free-text information retrieval systems over 20 years ago. It was first applied to enhance a text-based image retrieval system [23]. More recently, RF has been applied to content-based retrieval models based on color corelograms [14], modified color histogram [16], and multiple features [13, 15, 26]. In all cases, the RF process is designed to extract common features from relevant images and uses the features extracted to refine the queries.
Promising results were obtained in all cases.
Other RF models incorporated a learning component in the system by extracting knowledge from the positive and negative examples supplied by the users. [17] used the knowledge extracted to generalize and classify various parts of the scene. [30] employed the knowledge to identify clustering in the data to aid subsequent retrieval. While [19] integrated the knowledge in a density estimation to refine the query.
In [10], user's relevance judgments are used to make predictions of user’s behaviors using Bayesian learning based on a probabilistic model. The predicted user behaviors are combined with the selections made during a search to estimate the probability associated with each image. These probabilities are then used to select images for display.
Finally, [15, 16] used the relevance judgment data to generate a decision tree to help better classify images into relevant and non-relevant sets.
3 Color-based Pseudo Object Model with Relevance Feedback
This section describes the development of a color-based pseudo-object model with relevance feedback. The main idea here is to extract common pseudo object information from the relevant image set and use the information extracted to enhance subsequent retrievals via query refinement.
Color Coherent Vector Representation
The pseudo object model we employ is the Color Coherence Vector (CCV) [22]. In CCV, pixels are classified as either coherent or non-coherent. We define a color coherence vector as:
ccv = (Hc, Hnc) (1) where:
• Hc is the normalized coherent color histogram.
• Hnc is the normalized non-coherent color histogram.
In general, coherent regions tend to correspond to part of objects within an image while non- coherent pixels tend to come from the image’s background. Thus, matching of coherent histograms partly corresponds to matching of objects.
The similarities between the query image and all images in the database are computed based on the correlation of corresponding coherent and non-coherent histograms with contributions of perceptually similar colors. In order to find more images in the database having objects similar to that of query image, we give more weight to the contribution of the coherent part. The derivation of the similarity formula is given in [5].
Pseudo Object Based Relevance Feedback Method
The relevance feedback process aims to extract common dominant coherent colors from the relevant images. By definition, a dominant color is one that appears frequently in relevant images, but rarely in the rest of image collection. This information can be estimated using the probabilistic 0.5 formula, which has been widely used in free-text RF processes [27]. The 0.5 formula for estimating the weight of color i is given by:
) 5 . 0 )(
5 . 0 (
) 5 . 0 )(
5 . 0 log (
+
− +
−
+
−
−
−
= +
i i i
i i
i i
r n r
R
r R n N
w r (2)
where:
• N is the number of images presented to the user.
• R is the total number of relevant images indicated by the user.
• ni is the number of images among those returned whose H is above the noise thresholdic 1.
• ri is the number of relevant images with H above the noise threshold. ic
In order to model user's perception of similar colors, we incorporate perceptually similar colors into the above computation. We use wi as the basis to rank the importance of a color. From extensive tests, we found that it is only necessary to select up to top k (k is set to 10) dominant coherent colors whose wi is positive. These dominant colors are referred to as D , i = 1,…,k; where k ic ≤ 10.
We use Dc to modify the query CCV. As we are interested only in extracting common object information from the relevant image set, only the coherent component of query CCV is modified using Dc. The non-coherent part of query CCV remains unchanged. The query modification process is:
if Otherwise
) 0 (
) 0 ) (
1
( c
c i
D j
c c j
i i c
i i
Q
k H Q Q
c
∈D
+
=
∑
∀∈+ α β (3)
)
0 ( ) 1
( nc
i i nc
i Q
Q + = (4)
1 Here the noise threshold is set to eliminate color noise. In our experiments 2% is found to be a suitable threshold.
Here α and β are used to control the query modification process. We choose α = 0.15 and β = 0.85 to give more weight to the relevant images. The new query is normalized before a new retrieval is performed.
Color Interval Decision Tree
Instead of presenting the ranked list of images retrieved using the modified query to the user directly, it has been found to be more effective to modify the ranked list using the overall knowledge of the retrieved and relevant image sets [16]. This knowledge is captured in the form of a decision tree. A decision tree is a representation of the decision procedure for determining the classification of a given image. In this work, we employ Quinlan's ID3 algorithm [24] to build a decision-tree based on the coher ent part of CCV to classify images into relevant and non-relevant sets.
As ID3 requires a discrete feature space, the Multi-Interval Discretization Algorithm (MIDA) [15, 16] is first employed to extract the significant color intervals of the Hc part of the retrieved images.
This is done in such a way that will maximize the information values of the overall representation.
These significant color intervals are then used as branch points to construct a CCV Interval Decision Tree (CIDT). The CIDT is used to help in predicting the relevance of new images. Further details on how the CIDT can be constructed is given in [15].
Overall Procedure of Pseudo Object Based Relevance Feedback Process The overall relevance feedback procedure adopted in our work is as follows:
a. Initial retrieval: Using the query image supplied by the user, the CCV is used to search the image database to return a ranked list of images to the users.
b. Relevance feedback: By browsing through the retrieved images, users can select images deemed relevant to the query. This information is then submitted to the system through a graphical user interface.
c. Query refinement: From the set of relevant images, the system first selects a set of dominant coherent colors using the probabilistic 0.5 formula (Equation 2) to represent the relevant image set.
The appropriate coherent colors in the query vector are modified using Equation 3 to make the query more similar to the relevant images.
d. Color Interval Decision Tree: At the same time, we build the CIDT using the coherent colors and relevant judgment information.
e. Another round of retrieval: A new round of retrieval is initiated by using the modified query to compute the similarity of all images with the query. Every image on the result list is also passed through the CIDT to obtain a weight that gives the degree of relevance of these images with the query. This weight is used to revise the similarity value of the image. The final similarity value is used to rank the images.
Implementation and Testing
The system is implemented in C++ on a Sun workstation. It is tested on a large image collection of over 12,000 images. These images has been acquired commercially from Kagema Corporation. The images in the collection are general and cover a wide variety of categories. A set of 8 queries was
designed to test the effectiveness of the system. The details of the image collection and the queries used can be found in [5].
We adopt a color resolution of 316 L*u*v colors. This color resolution has been found to be most effective in [5]. For comparison purpose, we compare the retrieval results of CCV method with RF against those obtained using the normal histogram method with RF. These two methods are referred to as CCV316 and NH316 respectively. For CCV316, the following tests were conducted:
(a) M1: The normal CCV retrieval without RF. This provides a basis for comparing the effectiveness of various stages of RF operations. (b) M2: M1 with one RF iteration using only the pseudo object-based query modification described previously. (c) M3: M1 with one RF iteration using only the CIDT. (d) M4: M1 with one RF iteration using the combined operation of pseudo object query modification followed by CIDT.
Figure 1 summarizes the results of the tests. By using the results of initial CCV (M1) as the base, it can be seen that the various stages of RF have resulted in various degrees of improvement in retrieval effectiveness. The use of the query modification stage alone (M2) has resulted in an improvement of 24.64% in average normalized precision and 4.9% in average normalized recall.
The use of CIDT alone (M3) has resulted in further improvement of 27.14% in average normalized precision and 4.9% in average normalized recall. The best performance is achieved using a combination of the query modification and CIDT to perform RF (M4), which has resulted in a significant improvement of 35.36% in average normalized recall and 6.7% in average normalized precision. The results are far superior to those that can be achieved using the Normal Histogram model (NH316).
Figure 1: Comparison of results
Figure 2 shows the top 18 images retrieved using the CCV method with RF based on Query Q2 to retrieve “images of horses in outdoor environment”. Figure 2(a) shows the query image used; 2(b) presents the result of the initial retrieval; and 2(c) shows the output of method M4. The relevant images are marked with red dot at the bottom of the image. From the Figure, it can be seen that 2(c) contains a lot more relevant images with object “horse” than 2(b), thus further demonstrating the effectiveness of the RF process.
4. An Object-based Video Indexing Scheme
The retrieval of video can be viewed as similar to that of images in that it is possible to model its contents using a combination of visual and text features. Thus, technique developed for images can be used to perform content-based retrieval of videos. However, video is much more complex than image. It is a temporal medium. Hence it is necessary to consider motion, the segmentation of
0.2 0.22 0.24 0.26 0.28 0.3 0.32 0.34 0.36 0.38 0.4
M1 M2 M3 M4
Normalized average precision
CCV316 NH316
0.5 0.6 0.7 0.8 0.9 1
M1 M2 M3 M4
Normalized average recall
video into meaningful shots, and the sequencing of shots for effective presentation. The sequencing and presentation of video shots are unique and fundamental to video retrieval as different sequencing of the same set of shots can give rise to totally different meanings. This fact has been exploited in movie making in creating rich variety of scenes and emotional feelings [12].
(a) The query image
(b) Retrieval output of CCV316 without RF
(c) Retrieval output of CCV316 with RF based on M4 Figure 2: Output of Query Q2
Most video retrieval systems adopt the traditional segmentation model of dividing the video sequence into meaningful shots, and modeling the contents of the shots and their higher level entities individually for retrieval. One such system is the video retrieval and sequencing system developed in [8]. The video is first divided into shots using semi-automated tool. Each video shot is then logged using text descriptions, audio dialogue, and cinematic attributes. A concept-layer is then built to model the higher level entities and the relationships among the shots. A free-text concept-based retrieval model is used to retrieve the video shots based on free-text query. A cinematic, rule-based, virtual editing tool is developed to sequence the video shots retrieved.
During query, the users specify a free-text query, optionally with the cinematic rule for presentation, and a time limit. The system will return a video sequence within the time limit. An application in the domain of animals has been developed [32]. Snapshots of video sequences retrieved using the query “koala sleeping” are shown in Figure 3. The figure also shows different automated sequencing of shots in response to applying different cinematic rules [11] for presentation.
(a) Cinematic rule: concentration rule; Tine duration: 25 seconds
(b) Cinematic rule: enlarge rule; Tine duration: 25 seconds
(c) Cinematic rule: general rule; Tine duration: 25 seconds
Figure 3: Sequencing of retrieved video shots using the query “Koala Sleeping”
Although the segmentation approach has been widely used, it has serious limitations. In such a model, the granularity of the information is a shot, which typically contains a self-contained but high-level concept. First of all, it is not possible for users to access or present the video contents within the shot boundaries. Secondly, the creation of shots is a non-trivial task and it imposes the intentionality of the authors early during shot creation. It thus does not naturally support other users who might need to access the video materials for different purposes. Finally, it is hard to describe fully the contents of the shots. Automated description of video contents at shot level is also not possible.
To overcome these problems, we employ an object-based video indexing scheme to model the contents of video based on the occurrences of simple objects and events (known as entities). This is similar to the stratification approach proposed in [1]. In this scheme, vide o is modeled as multiple occurrences of simple entities. The entity can be a concept, object, event, category, or dialogue that are of interests to the users. The entities may occur over many, possibly overlapping time periods.
A strand of entity and its occurrences over the video stream is known as a stratum. Figure 4 shows the modeling of a News video with strata such as the object “Anchor-Person-A”; categories “home news”, “international news”, “live reporting”, “finance”, “weather” and “sports”; and dialogues.
The meaning of the video at any time instance is simply the union of the entities occurring during that time, together with the available dialogue.
Figure 4: Object-based modeling of a News video
The object-based modeling of video has several advantages. (a) As strata tend to contain only simple entities, the pseudo-object models together with RF and other learning methods may be adopted to identify and track entities automatically. (b) The meaning of video at any instance is simply the union of strata occurring at that time. Thus video sequence may be flexibly retrieved by returning those portions of video whose meaning matches the query. (c) The strata information provides the meta-information for the video. Such information can be used to support innovative functions such as the content-based fast-forwarding, and summarization [2] of video etc.
Using the object-based video indexing scheme as the base, we are developing several applications in the domain of News and Sports. We are also developing tools to automate the process of creating most strata, and advanced functionalities to perform interaction, fast-forwarding, and summarization [2] of video.
5. Conclusion
This paper discusses the pseudo-object based models to model images and videos. It describes the development of a color-based pseudo-object model for image retrieval with relevance feedback.
Test results on a large image collection indicate that the use of pseudo-object model with RF can lead to substantial improvement in retrieval effectiveness.
The paper also describes an object-based video retrieval model. It suggests that pseudo-object model with RF and other learning capabilities may be adopted to create strata of simple entities automatically. This will make stratification approach feasible. Current works are centered on creating advanced functionalities for the interaction, fast-forwarding, and summarization of video.
Acknowledgements:
The authors would like to acknowledge the support of National University of Singapore for the provision of a research grant #RP3960687 (Video Classification Based on Contents and Motions) under which this research is carried out.
Raw Video Sequence
Sentences A Sentences B Sentences C
Anchor-Person-A:
Home-News:
International News:
Finance:
References
[1] T.G. Aguierre-Smith & N.C. Pincever. Parsing movie in context. USENIX. Nashville. 157-167, Summer 1991.
[2] R.P. Carlos & K. Uehara. Video summarization based on semantic representation. Proceedings of Conference on Advanced Multimedia Content Processing. 1-16, Nov 1998.
[3] C. Carson, S. Belongie, H. Greenspan & J. Malik. Region-based image querying. CVPR’97 Workshop on Content-based Access to Image and Video Libraries. 109-116, 1997.
[4] A.E. Cawkell. Imaging systems and picture collection management: a review. Information Services & Use, 12:301-325, 1992.
[5] T.S. Chua & C.X. Chu. Color-based pseudo object model for image retrieval with relevance feedback. In 1st International Conference on Advanced Multimedia Content Processing. 148-162, Nov 1998.
[6] T.S. Chua, S.K. Lim & H.K. Pung. Content-based retrieval of segmented images. Proceedings of ACM Multimedia ’94, 211-218, 1994.
[7] T.S. Chua & W.C. Low. Image retrieval using multiple features and domain knowledge. In Proceedings of International Symposium on Multimedia Information Processing, 543-548, Dec.
1997.
[8] T.S. Chua & L.Q. Ruan. A video retrieval and sequencing system. ACM Trans. on Information Systems, 13(4): 373-407, Oct 1995.
9] T.S. Chua, K.C. Teo, B.C. Ooi & K.L. Tan. using domain knowledge in querying image databases. Third International Conference on Multimedia Modeling, Toulouse, France, 339-353, Nov. 1996.
[10] I.J. Cox, M.L. Miller, S.M. Omohundro & P.N. Yianilos. Pichunter: Bayesian relevance feedback for image retrieval. International Conference on Pattern Recognition, 361-369, 1996.
[11] G. Davenport, T.A. Smith & N. Pincever. Cinematic primitives for multimedia. IEEE Computer Graphics & Applications. 11(4): 67-74. 1991.
[12] M.R. Dyer. Film: a montage of theories. E.P. Dutton, New York. 1966.
[13] C. Faloutsos, R. Barber, M. Flicker, J. Hafner, W. Niblack, D. Petkovic & W. Equitz. Efficient and effective querying by image content. Journal of Intelligent Information Systems, 231-262, 1994.
[14] J. Huang, S.R. Kumar & M. Mitra. Combing supervised learning with color correlograms for content-based image retrieval. Proceedings of ACM Multimedia’ 97, 325-334, 1997.
[15] W.C. Low. Relevance feedback techniques for content-based image retrieval. Technical report, Department of Information Systems and Computer Science, National University of Singapore, 1998.
[16] W.C. Low & T.S. Chua. Color-based relevance feedback for image retrieval. International workshop on MM DBMS, Dayton, USA, 116-123. IEEE Computer Society, 1998.
[17] T.P. Minka & R.W. Picard. Interactive learning using a society of models. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 447-452, 1996.
[18] A. Nagasaki & Y. Tanaka. Automatic video indexing and full-video search of object appearances. Visual Database Systems, II, IFIP, Elsevier Science Publishers B.V., 113-127, Oct 1992.
[19] C. Nastar, M. Mitschke, C. Meihac & N. Boujemaa. Surfimage: a flexible content-based image retrieval system. Proceedings of ACM Multimedia '98, 339-344, 1998.
[20] N. Negroponte. Being digital. Vintage Books. New York, 1996.
[21] G. Pass & R. Zabih. Histogram refinement for content-based image retrieval. In IEEE Workshop on Application of Computer Vision, 96-102, 1996.
[22] G. Pass, R. Zabih & J. Miller. Comparing images using color coherence vectors. Proceedings of ACM Multimedia’ 96, 65-73, 1996.
[23] R. Price, T.S Chua & S Al-Hawamdeh. Applying relevance feedback on a photo archival system. Journal of Information Science, 18:203-215, 1992.
[24] J. R. Quinlan. Induction of decision trees. Machine Learning, 81-106, 1986.
[25] Y. Rui, T.S. Huang & S. Mehrotra. Content-based image retrieval with relevance feedback in mars. In Proceedings of IEEE International Conference on Image, 1997.
[26] Y. Rui, T.S. Huang & S. Mehrotra. Relevance feedback techniques in interactive content- based image retrieval. In Proceedings of IS&T and SPIE Storage and Retrieval of Image and Video Databases VI, 1998.
[27] Gerard Salton. Automatic Text Processing. Addison-Wesley Publishing Company, Cornell University, 1989.
[28] J.R. Smith & S.F. Chang. Image indexing and retrieval based on color histograms. Proceedings of ACM Multimedia '96, 87-98, 1996.
[29] M.J. Swain and D.H. Ballard. Color indexing. International Journal of Computer Vision, 7(1):11-32, 1991.
[30] M.E.J. Wood, N.W. Campbell & B.T. Thomas. Iterative refinement by relevance feedback in content-based digital image retrieval. Proceedings of ACM Multimedia '98, 13-20, 1998.
[31] H. Wynne, T. S. Chua & H. K. Pung. Integrated color-spatial approach to content-based image retrieval. Proceedings of ACM Multimedia’ 95, 305-313, 1995.
[32] Y. Zhang. A digital video retrieval system. Honours year project report. School of Computing, National University of Singapore.
