An Application of ESSNN to Phishing Websites
6.4. ESSNN vs Other Data Mining Algorithms 1 Experimental Strategy
6.4.4. Results from the Third Data set
The set of features achieved from the IG in this data set when threshold value is set to 1% are shown in Table 6.6.
Chapter 6 Page 134 AN APPLICATION OF ESSNN TO PHISHING WEBSITES
From this table, we can see that a cyclical concept drift occurs because some features that are considered significant in the first data set, but drop from the second data set become significant again in the third data set. These features are, age of domain and Google index. In addition, some features become insignificant and dropped from the list achieved from the second data set. Those are, .htaccess redirect, PageRank and statistical-reports based feature. We will consider the third data set as a testing data set and pass it to the ESSNN in order to assess the effect of these changes on the ESSNN performance. The results show that the overall accuracy of the ESSNN drops to 92.02%. Although this accuracy rate is considered acceptable, we will create a new classifier and add it to the ESSNN in order to assess if it will improve the ESSNN performance. At the end of this experiment the ESSNN will include three classifiers. Table 6.7 shows the performance of the ESSNN as well as the considered DM algorithms on different threshold values.
The testing data set used for assessing the performance of the ESSNN as well as all considered DM algorithms is a combination of the testing data sets from the first, second, and third data sets. However, we notice that the same set of features has been selected when threshold value is set to 30% in all data sets (first, second, and third data sets) these are HTTPS and URL of Anchor. Therefore, no classifiers will be created at this threshold value because that will result in deriving a redundant classifier as confirmed from the results in the second data set.
Chapter 6 Page 135 AN APPLICATION OF ESSNN TO PHISHING WEBSITES
Table 6.7 Confusion matrices (Third Data set)
From Table 6.7 and Figure 6.4, we find that the ESSNN produces the best results when we add the classifier created at threshold=1% to the ESSNN. Not only that, but also the ESSNN achieves the best results in this experiment compared to the results achieved earlier, i.e. experiments on the first and the second data sets in terms of specificity (TNR), sensitivity (TPR), F1-Score, precision, and accuracy.
Figure 6.4 Performance of ESSNN and other DM algorithms (Third Data set)
In terms of F1-score, the ESSNN outperforms BN, C4.5, SVM, LR, and RIPPER by 3.31%, 3.06%, 7.61%, 4.06% and 4.81% respectively. In addition, the ESSNN
80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% BN C4 .5 SV M LR RIPPE R ESSNN BN C4 .5 SV M LR RIPPE R ESSNN BN C4 .5 SV M LR RIPPE R ESSNN
Threshold = 1% Threshold = 10% Threshold = 30%
Chapter 6 Page 136 AN APPLICATION OF ESSNN TO PHISHING WEBSITES
produces 3.77%, 3.57%, 8.46%, 4.59%, and 5.40% higher prediction accuracy than BN, C4.5, SVM, LR, and RIPPER respectively. That can be due to the fact that the ESSNN becomes more talented in dealing with different phishing websites that are designed using different features. In other words, the ESSNN reaches a stage of having several classifiers (three classifiers) where each of which is considered an expert in a particular part of feature space or concept. However, at the same threshold value we notice a slight improvement on the performance of the considered DM algorithms compared to the results achieved from the second data set. That is because of the emergence of a cyclical concept drift. RIPPER produces the worst results in this experiment among all other experiments when the threshold value is set to 10%. That is because RIPPER learns the rules greedily and prunes rules using incremental reduced error pruning method (Witten et al., 2011), which results in removing some rules that might be useful in determining the website class which in turn limits its ability in handling the virtual concept drift.
Overall, some features have shown to be more significant in predicting the phishing websites because they have been selected in all sub experiments when the threshold value is set to 1%. Those features are HTTPS, URL of anchor, adding prefix and suffix to domain, sub domain and multi sub domains, Using Free domain registration, request URL, links in <Meta>, <Script> and <Link> tags and web traffic.
In general, this set of experiments shows that the traditional DM algorithms that are commonly used for classifying phishing are not the appropriate choice for the application of phishing websites. On the other hand, the ESSNN is able to handle the dynamic nature of phishing.
Chapter 6 Page 137 AN APPLICATION OF ESSNN TO PHISHING WEBSITES
The results achieved from the ESSNN in experiments 1, 2 and 3 when the threshold value is set to 1% (the classifier that produced at this threshold value is the classifier that has been added to the ESSNN in all cases, i.e. in experiments 1, 2 and 3) have been compared against other considered DM algorithms using the paired t-test in order to observe the significance in performance. The results show that the ESSNN algorithm produced a significant improvement over other considered DM algorithms (3 wins and 0 losses) with 95% confidence level when the epoch size is set to 500. This result confirms that the ESSNN is able to handle the virtual concept drift that occurring in phishing websites.