Open issues and research trends in Content-based Image
Retrieval
Raimondo Schettini
DISCo
Universita’ di Milano Bicocca [email protected] www.disco.unimib.it/Schettini/
IEEE Signal Processing Society Transactions on Image Processing EDICS (editors’ information classification scheme)
Amended: 2 July 1998 1 IMAGE/VIDEO CODING AND TRANSMISSION 2 IMAGE/VIDEO PROCESSING
3 IMAGE FORMATION
4 IMAGE SCANNING, DISPLAY, AND PRINTING
5 IMAGE/VIDEO STORAGE, RETRIEVAL, AND MULTIMEDIA
5 IMAGE/VIDEO STORAGE, RETRIEVAL, AND MULTIMEDIA 5-DTBS Image and Video Databases
5-SRCH Image Search and Sorting 5-INDX Video Indexing and Editing
5-INTG Integration of Images and Video and Other Media 5-CONT Content-based Multimedia
5-MAPP Multimedia Applications 5-AUTH Authentication and Watermarking
5-OTHE Other Image/Video Storage, Retrieval, and Multimedia IEEE Signal Processing Society
Transactions on Image Processing EDICS
Why content-based image retrieval is so popular ?
-Large collection of image have been -digitized in the hope of -Conservation -Distribution -Better access
Applications
• Art gallery, museum.
• Photographic archives (journalism).
• Textile databases.
• Trademarks databases.
• E-commerce catalogues.
• Web-searches.
• Medical image databases.
• Facial databases (security, law enforcement).
• Engineering design databases.
• Geographic databases.
• Satellite image databases.
• Home-photo collections
• Education and training
• …
Image domains
• Narrow domains: limited and predictable variability of relevant aspects of its appearance
Image domains
• Broad domains: unlimited and unpredictable variability of relevant aspects of its appearance even for the same semantic meaning
Possible queries in content-based image retrieval
•Target search
•Does a given image exist as is in the database?
•Does a scaled and/or rotated version of this image exist in the database?
•Does a region of this image exist in the database?
•Does an object exist in the database?
Search by association
•Show similar images to a query image Category search
•Show images belonging to the same category
Impossible queries in content-based image retrieval
High level semantic queries (in broad domains):
- A girl dancing on a sunny beach - the pope kissing a baby - a beautiful girl/boy - ....
Schematic description of the activities of a CBIR system
Feature selection
Features will ideally present the following basic properties:
• perceptual similarity: the feature distance between two images is large only if the images are not
"similar";
• efficiency: they can be rapidly computed;
• economy: their dimensions are small in order not to affect retrieval efficiency;
• scalability: the performance of the system is not influenced by the size of the database;
• robustness: changes in the imaging conditions of the database images do not affect retrieval.
How to describe visual content?
General features
• Color
– Color Histogram – Color Coherence Vectors – Correlograms
– Spatial Chromatic Histogram – ...
• Texture – Wavelets
– Grey-Levels Distribution – Gabor Filters – Tamura descriptors – ...
• Shape
– Deformable Template – Invariant Moments – Wavelets
– Description of boundaries – ...
• Other
– Discrete Cosine Transform – Regions Layout/Relationship – ...
How to describe visual content?
The role of segmentation
http://www.uk.research.att.com/dart/image_analysis.html
Evaluating similarity between segmented images
•Unsupervised segmentation is still very difficult.
•Quantitative measures for evaluating segmentation results are needed.
Evaluating similarity between segmented images
(Pala, 2000)
•Representation of segmented images:
•Region adjacency graphs
•Multidimensional indices
C. Carson et al., 1999
Evaluating similarity between segmented images
Multidimensional indicesEvaluating similarity between segmented images Evaluating similarity between segmented images
How to describe visual content?
Weak segmentation
Weak segmentation is possible (and very often useful)
THESAURUS DEFINITION
MATCHING /DECISION RULE
Vegetation, Water, Sky
+
Snow Water Ground Sky Vegetation
...
How to describe visual content?
Domain specific features
How to describe visual content?
Domain specific features
subsam pling
mask
extract window (20x21 pixel)
input
hidden layer
output
Matching
preprocessing Input image pyramid
Hidden annotation: face detection and recognition
Face detector
How to describe visual content?
Features definition
• Color distribution
• Texture pattern
• Shape
• Real content
Degree of difficulty
• Objects
• Partial semantic
Image processing in content-based image retrieval
Image processing in content-based image retrieval
Original scene
?
Image processing in content-based image retrieval
Image processing in content-based image retrieval Image processing in content-based image retrieval
Image processing in content-based image retrieval Image processing in content-based image retrieval
Image processing in content-based image retrieval
Image normalization using Retinex
Original image
Illuminant change Retinex filtering result
3
Image processing in content-based image retrieval
RE TI NE XE D OR GI NI AL
NO MR AL IZ ED
Illuminant invariance color indexing
Experimental Setup
Experimental Setup
HSVM HIST CCV SCH
HSVM HIST CCV SCH
Success of Target
Search
Illuminant invariance color indexing
image Enhancement
Image processing in content-based image retrieval
Feature selection and similarity evaluation
Gap between low-level features and image semantics
•heavy involvement of the user
•lack of flexibility
Are these similar ?
Feature selection and similarity evaluation
•A common idea is that similarity may be modeled as a weighted sum of the feature distances.
•An emerging idea is to see similarity as a probabilistic concept.
Are these similar ?
Feature selection and similarity evaluation
Rubner, 1999
Scalability in CBIR
Scalability and image semantics
Possible solutions:
•Apply relevance feedback
•Reduce the DB size by image classification
•Design and Exploit image annotation tools
•Exploit all the available information
Relevance feedback
Positive Examples Negative Examples
Agreement between positive examples
Agreement between positive and negative examples Weight Estimation
Wh+- Wh*
Wh+ Wh*
Wh
The idea User interaction
Relevance Feedback
Example
Relevance feedback: Query Reformulation
Query Relevant Objects
Shape Color
The idea
User interaction
Relevance feedback:
Exploit all the available information Relevance Feedback
“Human-in-the-Loop”
Database SIMILARITY EVALUATION
Ranked Results
pianta con lunghi steli sinu pianta con lunghi steli sinuosi e larghe foglie ("albero della vita");
sul collo, ampia fascia con motivo a treccia;
tipologia"arcaica".Dipintoinbruno di manganese e verde rame. osi e larghe foglie ("albe pianta con lunghi steli sinuosi e larghe foglie ("albero della vita"); sul collo, ampia fascia con motivo a treccia; tipologia "arcaica". Dipinto in bruno di manganese e verde rame. ro della vita"); sul collo, ampia fascia con motivo a treccia; tipologia "arcaica". Dipinto in bruno di manganese e verde rame.v pianta con lunghi steli sinuosi e larghe foglie ("albero della vita");
Query Relevant Objects
Shape Color
Query Reformulation
Positive Examples Negative Examples
Agreement between positive examples
Agreement between positive and negative examples Weight Estimation
Wh+ - Wh*
Wh+ Wh*
Wh
Features Weights
Pictorial Textual
pianta con lunghi steli sinu pianta con lunghi steli sinuosi e larghe foglie("piantacon lunghi steli sinu pianta con lunghi steli sinuosi e larghe foglie("albero della vita"); sul collo albero della vita"); sul collo, ampia fascia con motivoatreccia;
tipologia ("albero della vita");
pianta con lunghi steli sinu pianta con lunghi steli sinuosi e larghe foglie("albero della vita"); sul collo, ampia fascia conmotivoa treccia; tipologia ("albero della vita");
pianta con lunghi steli sinu pianta con lunghi steli sinuosi e larghe foglie("albero della vita"); sul collo, ampia
piantacon lunghi steli sinu pianta con lunghi steli
sinuosi e
larghe foglie ("albero della vita");sul collo,ampia fascia con
motivoa
treccia;
tipologia ("albero della vita");
Browsing and visualization
Rubner, 1999
Database Organization
Clustering
• Can be used to:
– Improve retrieval efficiency
– Provide an overview of database content
Relevance feedback, Database Filtering and Semantic Associations
Combination of - unsupervised learning;
- supervised learning;
- short term learning (relevance feedback);
- long term learning (user profile)
Impossible queries in content-based image retrieval
Clutter scene...
Impossible queries in content-based image retrieval
Reality vs image reality
Hidden Annotation
High level categorization
IMAGE CLASSIFICATION
Close-up Text Graphic Outdoor Indoor
...
A few applications...
General Purpose Content-based retrieval
Design a retrieval system able to
• use both low level and high level descriptions of visual contents automatically extracted.
• cover a wide range of users requirements.
• adaptively select the retrieval strategy that is most suited for the users needs.
• All these with a simple interface !!!
The ‘Quicklook 2 ’ system
Overview
Web-Demo at http://quicklook.itc.cnr.it
The ‘Quicklook
2’ system
Queries available
• Pictorial queries
– Provide an example of ready-made image – Sketch an image with the available tool
– Browse the database to find one or more relevant images with which to begin
• Textual queries
– Filter the database with textual keywords – Search images by textual annotation similarity – Search images by text similarity
• Advanced queries
– Combination of one or more of the above queries – Relevance feedback
Journal indexing
Graphics Digital documents
Photographs Texts Compound
documents
Segmented documents
Not compound documents
Photo indexing
Indoor Outdoor Close-up Pornographic
Images
Photos to classify
Non-Pornographic Images
Rejected Images
Biometrics Pattern recognition in cultural heritage archives
Pattern recognition in cultural heritage archives Video Indexing
Key-Frames Cuts / Fades Detection
...
...
Video Indexing
Abstraction: summarization
Key Frame Levels
Structure Levels
Classification of videos
• Scenes classification
• Video surveillance
Classification of web pages
• Improving the performance of search engines
3D Objects
• Challenges – 3D Pre-processing
– Features to use in indexing 3D objects
• 2D Features
• 3D Features
• Invariants
– Interface to use to submit a query
• 2D Projections View
• 3D Examples – Matching 2D and 3D data
Adult image filtering 3D Object indexing
• 3D face recognition
Cross-media 3D face recognition
Photographs
Videos Tridimensional objects
Videos
High level Low level
Media
Features Matching
strategies 3D objects
Web pages
Images
Middle level
Open issues and future researches in Content-based Image Retrieval
Target search Similarity search
Semantic retrieval
