Predictive Model Using Characteristic Rule

One challenging issue with the classification rule based scoring prediction model is how to handle unbalanced class sizes or the situation where there are too many classes present in the dataset. This is substantial because, in many datasets, the class sizes are very different. For example, it is common to find the target class comprising less than 10% of the total instances. In a marketing database of a promotion campaign a customer positive response rate of 1% or less is common. This situation also arises in the fraud detection problem with an adverse effect [Bolton and Hand 2002]. In the credit card transaction database the probability of fraud may be very low and is estimated as 0.2% in [Brause et al 1999]. It becomes much lower when the pre-processing step takes place in the fraud detection prediction system by taking a sample from the entire database. According to Hassabi “out of some 12 billion transactions made annually, approximately 10 million – or one out of every 1200 transactions – turn out to be fraudulent. Also, 0.04% (4 out of every 10,000) of all monthly active accounts are fraudulent [Hassibi 2000]”. Both classifications rule mining and scoring predictive model building might be difficult to work with this type of real dataset. Some of difficulties faced are

(1) If the class support is set to high and class error is set to low, then there is a good chance of finding a small number of classification rules which are not good enough

Figure 3.16 Lift curves of different models constructed with churn data

0 10 20 30 40 50 60 70 80 90 100 0 7.49 9 17.4 6 27.4 8 37.4 9 47.4 5 57.4 7 67.4 9 77.4 5 87.4 6 100 % examples % c h u rn e d = y e s Random Model Model-1 Model-2 Model-3 Model-4 Model-5

to build a scoring predictive model. A few classification rules of a class cover a small number of training instances within the class and this will also find a small number of the same type of training instances to be used in predicting future instances.

(2) If both the class support and error are set to be very low, it will result in a large number of classification rules. This causes an increased computer execution time. Moreover, since class support is low, this kind of rule also has a very low predictive power and hence scoring predictive models built with these rules will suffer from poor performance.

(3) The last difficulty is the cost of the class support calculation and the errors in classification rules for all classes. Despite the fact that we are not interested in all the classes, but only interested in a target class. Not only that, we need to apply both the class support and error of all classification rules for all classes in the calculation of scores for future instances. This makes the scoring calculation more time consuming.

In this section, characteristics rules are proposed for scoring model building, to address the above difficulties of building a scoring prediction using classification rules and to make the prediction easier and faster. The overall proposed algorithm for the characteristics rule based scoring prediction is presented in Function (3.4).

In Chapter 2, rule metrics are defined and explained. Among them, rule support and rule confidence are two pieces of information about characteristics rules which will be used in the calculation of the score for future instances. Let the relational dataset D be used for building the predictive model with nnumber of instances and l number of

attributes or fields a₁,a₂,_L,a_l . The fields could be either discrete or numerical. The numerical attributes need to be discretized. Instances are pre-classified and belong to

one of the mclasses of CaC2,L,Cm . We are interested in a target class sayCt. These

instances are submitted to a GA characteristics rule mining method as discussed in Section 3.4.2 and the result is a characteristics rule set R .

The objective is to build a predictive scoring model using R , which will generate extra field score to D . Since we have a set of characteristics rules

R r r

r₁, ₂,_L, _p ∈ and each characteristics rule r_j has two pieces of information e.g. support denoted bySupp(r_j)and confidence denoted byConf(r_j), we can use them to

calculate the value of the score field for all instances in D . The following function is proposed to calculate the score for an instance e in D ,

CHAR-RULE-SCORING

INPUT:

example_list {ntest examples} char_rule_list {Characteristic rules}

n′ {Number of target examples}

STEPS:

for each example e ∈ example_list do {initialization} ; 0 : .num = e e.denum:=0; endfor

for each rule r char_rule_list do

select (*) as rule_examples from example_list where r;

for each example e ∈ rule_examples do ;

. . :

.num rs rc

e + = ∗ {multiplication of rule support and confidence} ; . : .num rs e + = endfor endfor

for each example e ∈ example_list do {initialization} if e.denum!=0 then e.score:=e.num/e,denum; else e.score:=0

endif enddo

scored_example_list :=SortAndExtract(example_list, n′ );{extracted n′ sorted examples}

RETURN: Examples sorted on prediction score

) ( ) ( * ) ( . 1 1

∑

∑

The dataset D is then sorted on the value of the score from highest to lowest. This means that the instance at the top of D has the higher probability to be the target class than the instances at the bottom of D . The end-user needs to choose a cut-off point to select a number of records from the top to be declared as the target class.

3.7 Summary

The rule mining method plays a major role in knowledge discovery and DM. There are several methods to mine rules from data. One of the most popular methods for rule mining is to extract rules from a decision tree. The decision tree method starts rule mining with a data set of many instances belonging to known classes. The decision tree is also known as a classification model which allows the classes to be reliably discriminated. The decision tree has been applied in classification problems extensively. The limitation of using the decision tree in rule mining includes its complex architecture, which can result in many unwanted rules. Apart from this problem, the decision tree may fail to mine rules that are important to users since the decision tree provides one solution of many possible solutions and rules that are extracted from that solution, which may not contain the user targeted rules. The parameters used in constructing the decision tree are not directly related to the rule constraints such as the support and confidence from the DM field and hence the quality of the rules from the decision tree may not be useful to end-users.

This chapter has proposed an innovative rule mining method based on GA rule mining which is constraint-based rule mining and incorporates both the ideas of GAs and Apriori. This method is powered by the features of using the support and confidence measures from the DM discipline to provide end-users control over the quality of the rules. The robust feature of the method is that it mines three major kinds of rules i.e. association, characteristic and classification rules. The introduction of a GA has made

the proposed method simpler than the decision tree method. The method has been tested with a benchmark data set. The results show that rules mined by the proposed method are better than the decision tree method. This chapter has also studied the scoring predictive model and its performance measures. Several algorithms are proposed to build predictive scoring models with the mined classification and characteristics rules for use in business. These proposals have also been tested with a benchmark dataset from the telecommunication industry.

Chapter 4

Rule Mining with Supervised Neural Networks

4.1 Introduction

The first task in the process of DM of a huge data set, is to construct a model of the data. This model can be used to quickly reveal interesting information in the form of rules. In Chapter 3 decision tree and GA based DM models and rule mining techniques were introduced. This chapter discusses and proposes methods to mine rules using a NN with supervised learning and shows its usefulness in DM. As we know NN hide information in the distributed weights of its links. NNs are limited due to being unable of providing an accurate, comprehensible interpretation of these weights. Neural network researchers have been attempting to convert these weight values into understandable information. One possible interpretation of this information is in the form of rules. The present chapter addresses this issue and solves this problem.

There are two types of NNs based on their learning ability, supervised and unsupervised. Both supervised learning and unsupervised learning are discussed in chapter 2. This chapter investigates rule mining techniques by constructing a DM model with a single-layered neural network with supervised learning which is referred by SSNN in this thesis. The supervised learning of SSNN is where inputs are given along with the desired response and weights are adjusted to minimize the difference in output. SSNN is the earliest neural network devised by AI researchers. The salient feature of the single-layered neural network is that it is simple in design and learns from data more quickly than other types of NNs, such as multi-layered NNs with supervised learning. However the inherent weakness of the single-layered neural network is its inability to model non-linear data. This is the motivation behind incorporating SSNN in complex

systems like knowledge discovery or DM systems where substantial non-linear data is used. This chapter proposes an alternative way of modelling non-linear data using SSNNs. This alternative approach has also been used for the proposed rule mining method used in this chapter. The proposed SSNN rule mining method can attain significantly improved learning and convergence speed as well as the capability of dealing with real data. A set of SSNN has been used in the proposed rule mining method each of which models a local linear relationship existing in the entire non-linear relationship of the data. The idea here comes from the local linearization approach for a non-linear function approximation: a set of linear functions is established to model the non-linear function in a piece-wise manner.

This chapter is organized as follows. The chapter starts with a survey of past applications of neural networks for rule mining Section 2. Section 3 describes the SSNN model and the learning algorithm used in the proposed rule mining method. Section 4 explains the non-linear relationship approximation by a set of SSNNs. Section 5 discuses the method of reducing the number of SSNNs required in approximating a non- linear relationship. Section 6 derives three methods for three types of rules, e.g.,

association, characteristics, and classification rule mining. Subsequently, experiments

are illustrated to show the effectiveness of these methods in section 7. Section 8 presents guided rule mining methods using a SSNN. This method is evaluated with a benchmark dataset and the results are also presented in this section. Section 9 is the summary of the chapter.