Image Information Mining (IIM or I2M) is an interdisciplinary approach to automatize remote sensing analysis that draws upon expertise in computer vision, image under- standing, data mining, machine learning, databases, artificial intelligence, and software design (Burl et al., 1999). The main goals of this field are to seek solutions to automatize
the extraction of information from Earth Observation (EO) archivesthat can lead to im- age understanding and knowledge discovery (Datcu et al., 2007). An IIM system offers a user the opportunity to deal with a large collection of images by accessing in a large image database and allows the user to extract and infer knowledge about patterns hid- den in the images, so that the set of relevant images is dynamic, subjective and unknown (Hsu et al., 2002).
In section 3.2.1 we present a description of the main components of IIM systems. All of these components are grouped into a generic architecture depicted in Figure 3.1. Section 3.2.2 introduces examples of IIM systems.
3.2.1 Generic Architecture
Figure 3.1 shows the typical architecture of IIM systems. Generally, an IIM system is composed of two fundamental parts: off-line part and online part. The tasks of storing the image into the database, to extract the main features, to perform the clustering al- gorithms, to create the index catalogue, and to save the catalogue in the database are executed in the off-line part, whilst the user interaction by given queries as well as the interactive training and the probabilistic search are performed in the online part.
A generic architecture of an IIM system has the following components: image database, feature extraction, data reduction and content index generation, user iteration or human- machine communication, which are described in detail in the next sections.
38 3. IMAGEINFORMATIONMINING SYSTEMS
User
Feature Extraction
Image Index Database
Generation Semantic Catalog Probabilistic Search Interactive Training Query Image Processing Off-line process Online process Human - Machine Communication
Figure 3.1: Main components of Image Information Mining systems. The off-line pro- cesses involve feature extraction, clustering and catalog generation, while the online pro- cesses enable the human-machine communication.
3.2.1.1 Image database
Today, Earth images from space are acquired continuously and have become powerful scientific tools to enable better understanding and improved management of the Earth and its environment. An EO image is an invaluable source of information since it pro- vides the human being with perception and understanding about the scene that it repre- sents.
Unlike image processing which focuses on understanding and extracting specific fea- tures from a single image, the focus of image mining is the automatic extraction of pat- terns from a large collection of images. The image database is formed by EO data pro- vided by multiple sensor both optical and SAR as for example IKONOS, Landsat, TerraSAR- X, etc. Compared to another database, an image database presents some special charac- teristics (Hsu et al., 2002) like:
• The implicit spatial information is critical for the interpretation of the image con- tent.
• Image characteristics could have multiple interpretation for the same visual pat- terns.
3.2. IMAGEINFORMATIONMININGSYSTEMS 39
3.2.1.2 Feature extraction
An image is characterized by its primitive featuressuch as color, shape (Jain and Vailaya, 1996), texture (Tuceryan and Jain, 1998). Thus an image will be represented as a multi- dimensional feature vector acting as a signature. The amount of details in an image is increasing with the resolution. As a consequence more classes can be extracted from the image and more features/signatures are needed to characterize accurately an image content.
The primitive feature extraction methods aim at obtaining the best features for char- acterizing the image content, either using optical or radar images. When the IIM system uses radar images, the optical human capabilities to distinguish the image content be- come more complicated, therefore the feature extraction methods and their evaluation turn into an important issue.
3.2.1.3 Data reduction and content index generation
In IIM, after the multidimensional feature vector is obtained, images are assigned with a suitable content based description extracted from these features. The processes involved in this step are:
• Clustering: Similar extracted features are grouped together using a classification algorithm. Thus, pixels containing similar features belong to the same class. An example of a clustering algorithm is the k-means (MacQueen, 1967) method, which assigns each data point to the cluster whose center is nearest.
• Catalogue Generation: Once a clustering algorithm is performed, a data structure for indexing is created and stored in a database.
3.2.1.4 User interaction or human-machine communication
The interaction between the user and IIM systems is performed through the graphical interface, which enables the human-machine communication. The user defines the se-
mantic labelusing words from natural language and associates it with a generic class from the catalogue. Thus, an image is represented by its objects and a semantic label is assigned to each of them allowing the system to retrieve the image content using prede- fined labels. The semantic labels correspond to a high level of abstraction of the image.
However, since the user creates the association between the semantic labels and sig- nal classes there is always a semantic gap, which limits the use of IIM systems. Many methods have been presented trying to fill this gap as for instance the work of (Li and Bretschneider, 2007) which proposed a Bayesian network to infer the semantic concept of segmented scenes. A common technique to provide regions with semantic mean- ing consists in manual annotation as it was published in (Comaniciu and Meer, 2002). Schr ¨oder et al. (2000) presented a concept of interactive learning and probabilistic re- trieval, where the user gives positive and negative examples redefining the query and probabilistic map.
A common technique used to human-machine communication is the relevance feed-
back, which is an automatic process used to improve the retrieval effectiveness. In the work of (Cox et al., 2000) the implementation of relevance feedback is based on Bayesian networks. This method models the user reaction to a certain target image and infers the probability of the target image based on the previously performed actions.
40 3. IMAGEINFORMATIONMINING SYSTEMS
3.2.2 Examples of IIM systems
Several systems have been developed following the IIM approach for remote sensing imagery such as (Agouris et al., 1999), (Datcu et al., 2003), (Li and Narayanan, 2004), (Shyu et al., 2007). Theses systems have made significant contributions in the design of system frameworks for geo-spatial CBIR techniques with large-scale spatial databases.
• Agouris et al. (1999) presented an environment for CBIR from large spatial databases. This system enabled queries on shapes and topology. The authors presented the re- sults obtained with the system but did not present an evaluation of the system. • The integration of spectral and spatial information mining in remote sensing im-
agery was presented in (Li and Narayanan, 2004). The system is used in a specific application for land cover and land use classifications. The features are extracted using Gabor wavelets and the classification is performed through Super Vector Ma- chine (SVM). The authors posed the necessity of clustering and retrieval perfor- mance analysis from a large image database. A set of quantitative experiments were performed in order to test the accuracy of the retrieval results provided by the system. Their experiments did not consider the user interaction and the subjective factor in defining the semantic labels.
• Datcu et al. (2003) introduced the Knowledge-driven content based image infor- mation mining system called KIM. It is based on a Bayesian approach, and its architecture is defined in terms of stochastic problem modelling of probabilities. The system is composed of two main parts: off-line process and on-line process. The features extraction, data reduction and catalog generation and ingestion in the image database are performed in the off-line process whilst in the on-line section the system present a Graphical User Interface (GUI), which allows the interaction between the user and the system. The user is a main actor since he is the provider of the semantic interpretation of the image contents. This system has an application- free orientation using remote sensing imagery with optical as well as radar data. The evaluation of KIM was presented in (Daschiel and Datcu, 2005) using optical images. A complete description of KIM is provided in chapter 6 since it has been used during the subjective evaluation of this dissertation.
• The Geospatial Information Retrieval and Indexing (GeoIRIS) system (Shyu et al., 2007) was introduced in 2007. It enables scalable processing and retrieval of a large volume of data by automatically preprocessing and indexing satellite images. This system incorporates automatic feature extraction methods, visual content mining from a large-scale image database, and high dimensional indexing for fast retrieval. In addition, GeoIRIS includes techniques for complex queries that merge the in- formation from heterogeneous geospatial databases, retrieves the objects based on shape and visual characteristics and a semantic model to link low level image fea- tures with high-level visual descriptors. However, the performance evaluation of this system has not yet been carried out.