KNOWLEDGE DISCOVERY IN DATABASES

This section focuses on describing and explaining the process that leads to discovering new knowledge. It defines a sequence of steps (with eventual feedback loops) that should be followed to discover knowledge (e.g. patterns) in data. Each step is usually realized with the help of available commercial or open-source software tools as will be shown in the application chapters.

Since the 1990s, several KDD processes have been developed. The initial efforts were led by academic research and were quickly followed by industry. The basic structure of the model proposed by Fayyad et al (Fayyad, Piatetsky-Shapiro, & Smyth , 1996) is the one proposed in this thesis. The process consists of multiple steps that are executed in a sequence. Each subsequent step is initiated upon successful completion of the previous step and requires the result generated by the previous step as its input.

KDD is defined (Fayyad, Piatetsky-Shapiro, & Smyth , 1996) as the nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data.Here, data are a set of facts (for example, cases in a database) and pattern is an expression in some language describing a subset of the data or a model applicable to the subset. Hence, in our usage here, extracting a pattern also designates fitting a model to data;

finding structure from data or, in general, making any high-level description of a set of data.

The Fayyad et al. KDD model consists of nine steps, which are outlined as follows:

1. Developing and understanding the application domain. This step includes learning the relevant prior knowledge and the goals of the end user of the discovered knowledge.

2. Creating a target data set. Here the data miner selects a subset of variables (attributes) and data points (examples) that will be used to perform discovery tasks. This step usually includes querying the existing data to select the desired subset.

3. Data cleaning and preprocessing. This step consists of removing outliers, dealing with noise and missing values in the data, and accounting for time sequence information and known changes.

4. Data reduction and projection. This step consists of finding useful attributes by applying dimension reduction and transformation methods, and finding invariant representation of the data.

5. Choosing the data mining task. Here the data miner matches the goals defined in Step 1 with a particular DM method, such as classification, regression, clustering, etc.

6. Choosing the data mining algorithm. The data miner selects methods to search for patterns in the data and decides which models and parameters of the methods used may be appropriate.

7. Data mining. This step generates patterns in a particular representational form, such as classification rules, decision trees, regression models, etc.

8. Interpreting mined patterns. Here the analyst performs visualization of the extracted patterns and models, and visualization of the data based on the extracted models.

9. Consolidating discovered knowledge. The final step consists of incorporating the discovered knowledge into the performance system, and documenting and reporting it to

the interested parties. This step may also include checking and resolving potential conflicts with previously believed knowledge.

A very important consideration in the Knowledge Discovery Process is the relative time required to complete each step. It includes reviews of partial results, possibly several iterations, and interactions with the data owners. In general, we acknowledge that the data preparation step is by far the most time-consuming part of it.

Given these notions, we can consider a pattern to be knowledge if it exceeds some interestingness threshold, which is by no means an attempt to define knowledge in the philosophical or even the popular view. As a matter of fact, knowledge in this definition is purely user-oriented and domain-specific and is determined by whatever functions and thresholds the user chooses.

3.1.1 Data Mining

While KDD refers to the overall process of discovering useful knowledge from data, data mining refers to a particular step in this process. Data mining is the application of specific algorithms for extracting patterns from data. The term Data Mining was introduced relatively recently, in the mid-1990s, although data mining concepts have an extensive history. Data mining covers areas of statistics, machine learning, data management and databases, pattern recognition, artificial intelligence, and other areas. All of these are concerned with certain aspects of data analysis. As a result, they have much in common but each also has its own distinct problems and types of solution. The fundamental motivation behind data mining is autonomously extracting useful information or knowledge from data stores or sets. The goal of building computer systems that can adapt to special situations and learn from their experience has attracted researchers from many fields, including computer science, engineering, mathematics, physics, neuroscience, and cognitive science.

Unlike the majority of statistics, data mining typically deals with data that have already been collected for some purpose other than the data mining analysis. The majority of the applications presented in this chapter use data formerly collected for any other purposes. Data mining research has led to a wide variety of learning techniques that have the potential to renew many scientific and industrial fields.

The knowledge discovery goals are defined by the intended use of the system. We can distinguish two types of goal: (1) verification and (2) discovery. With verification, the system is limited to verifying the user’s hypothesis (Piatetsky-Shapiro , Brachman , Khabaza , Kloesgen , & Simoudis , 1996). With discovery, the system finds new patterns autonomously. We further subdivide the discovery goal into prediction, where the system finds patterns for predicting the future behavior of some entities, and description, where the system finds patterns for presentation to a user in a human-understandable form. In this thesis, we are primarily concerned with discovery-oriented data mining.