Integration of Bagging and Greedy Forward Selection on Image Pap Smear Classification using
Naïve Bayes
Dwiza Riana1, Achmad Nizar Hidayanto2 and Fitriyani3
1STMIK Nusa Mandiri Jakarta, 2Universitas Indonesia, 3Universitas BSI [email protected], [email protected], [email protected]
Abstract-Herlev dataset consists of 7 cervical cell classes, i.e.
superficial squamous, intermediate squamous, columnar, mild dysplasia, moderate dysplasia, severe dysplasia, and carcinoma in situ is considered. The dataset will be tested to classify two classes, consisting of normal and abnormal cells. Seven different cell types will be classified to separate the cells into 7 classes which are 3 normal cell classes and 4 abnormal cell classes. There are still some difficulties to classify the dataset into seven classes. This Pap smear image dataset has a class with a number of different and unbalanced classes. Another condition is that the data has features that are suspected to be irrelevant, so it is still difficult to classify especially abnormal classes. To handle the class imbalance, this study used ensemble method (Bagging). For handling data that had features and had no contribution, we made feature selection of Greedy Forward Selection. Furthermore, Naïve Bayes was used as learning algorithms. The results of this study obtained the highest accuracy value for the classification of two classes that are normal and abnormal using Naïve Bayes model with Greedy Forward Selection of 92.15%. As the classification of seven classes is good enough for Naïve Bayes model and Greedy Forward Selection with Bagging of 63.25% although it still needs to improve.
Keywords-Pap smear images, classification, feature selection, Bagging, Naïve Bayes.
I. INTRODUCTION
In the worldwide, cancer shows that cervical cancer is still considered to be the fourth most common cause of cancer deaths among women[1]. To improve the quality of life, we need to be aware of prevention in the form of early checkup known as Pap smear. Through a Pap test the precancerous condition and normal to abnormal cell changes that may develop into cancer can be known as early as possible. Therefore, the cells which develop into cervical cancer can be prevented.
Researchers generally classify cells in two classes.
Classification into seven classes is still difficult to do.
Nevertheless, there has been an effort to do so. The Pap smear system was first introduced in 1989 and updated in 2001 as a uniform terminology system that would provide clear guidance for clinical management. In this system, cells were divided into two categories, normal and abnormal. Furthermore, the research in [2] introduced a Pap smear result database. The database consisted of 917samples that were distributed unevenly in seven
classes by a thorough inspection conducted by an expert. Pap smear image classification in Herlev data has automatically become an area of interest to researchers, and many methods have been proposed, involving feature extraction techniques and machine learning algorithms, to recognize abnormalities in image. The condition of the nucleus and cytoplasm of each cell provides significant information about the normal and abnormal conditions of the cell. In Herlev [2], there are 20 features that can be processed to recognize the pathological cases of cervical cancer. For this reason, the researches of classification method on normal [3][4][5] and abnormal class classification [6] have been done. Some common approaches used for this purpose are feature analysis [7][8] and texture [9][10][5].
Many researches about single image classification of Pap smear have been done, but studies that examine irrelevant features and imbalanced classes have not been widely practiced.
Research using the same dataset but with the features, method and algorithm that are different use algorithm of Artificial Neural Network (ANN), Ecludian Distance (ED) and Support Vector Machine (SVM) [11]. The other researches [12] use ConvNet, but this research only classified 2 classes which are normal and abnormal. Up to now, classification of 7 classes is still difficult to do. The other research using the different dataset and using Naïve Bayes without feature selection and ensemble [13]. Research [14] used Naïve Bayes classification algorithms in Weka, Nearest Neighbor filters and Naïve Bayes transfers.
The results of the research show that the use of Naïve Bayes method have the significant results.
The use of Bagging model on previous researches can overcome imbalance class case. In [15] [16] noisy and imbalance classes are more effectively handled by Bagging technique. In [17] used integration of Bagging and Greedy Forward Selection using Naïve Bayes in prediction software defect show that Naïve Bayes method, Bagging and Greedy Forward Selection can have classification accuracy for 2 classes which are good.
This study proposes to do classification comparation of cervical cancer to classify 2 classes of normal and abnormal class and to classify 7 data classes using the Naïve Bayes model (NvB), Naïve Bayes and Bagging (NvB+BG), the feature of Greedy Forward Selection (NvB+GFS). The next proposed
model is the Naïve Bayes model with Greedy Forward Selection and Bagging (NvB+GFS+BG), Bagging model to handle the problem of imbalance class and the Greedy Forward Selection model for the selection of relevant features in the Pap smear single image classification.
This paper is divided into several sections. Section 2 discusses about materials and methods used in the study. Section 3 describes the results and discussion about the comparison of the four models, t-test and Friedman test, which will be enclosed with conclusions and further research plans.
II. MATERIAL AND METHOD A. Datasets
In this study the dataset used is Pap smear dataset with 917 records and 20 features or attributes. Dataset on single cell Pap smear images is called Herlev data bank [2]. Each cell was classified into seven classes by cyto-technicians and doctors on manual measurement and clinical confirmation.
TABLE I. THE HERLEV DATA OF CELL IMAGE PAP SMEAR
Name of Classes Image Amount of data
Normal Superficial 74
Normal Intermediate 70
Normal Columnar 98
Mild (Light) Dysplasia 182
Severe Dysplasia 146
Moderate Dysplasia 197
Carcinoma In Situ
150
Total of Data 917
The database is available in open source for research purpose and is used for analysis and validation. In table.1, the distribution of the data was given. Normal Superficial (NS), Normal Intermediate (NI), and Normal Columnar (NC) are the normal classes, whereas the other four classes were categories of abnormal cells that include: Mild (Light) Dysplasia (MLD), Severe Dysplasia (SD), Moderate Dysplasia (MD), and Carcinoma in Situ (CIS) [2]. In this study the feature type was given a naming that refered to the naming of the previous
feature [2]. Description of the 20 features of the nucleus area, cytoplasm area, N/C ratio, nucleus brightness, cytoplasm brightness, nucleus shortest, diameter, nucleus longest diameter, nucleus a elongation, nucleus a roundness, cytoplasm shortest diameter, longest cytoplasm diameter, cytoplasm elongation, cytoplasm roundness, perimeter nucleus, perimeter cytoplasm, nucleus relative position, nucleus maximum, minimum nucleus, maximum cytoplasm, and minimum cytoplasm.
B. Proposed Method
Figure 1 is a proposed research design. The research design consists of several stages involving a learning scheme consisting of feature selection, learning algorithm and meta learning.
Fig. 1. Research Design
The use of 10-fold cross validation on the dataset divided the dataset into 10 sections. The first data became testing data, and the second data up to the tenth data became training data.
The next step was to perform feature selection using Greedy Forward Selection (GFS). This step was intended to produce features from a single image of relevant Pap smear to improve the performance of a single Pap smear image classification in 7 classes. After the feature selection process then the training data that had been formed with cross validation would be created into new training data randomly by Bagging model based on Naïve Bayes as many iterations which were entered. If more records enter in a particular class category, then the record is predicted to include a certain class. For example: if more records enter the normal superficial class category then the record will be predicted as a superficial normal class.
1. Greedy Forward Selection Feature
This study used feature selection concept to determine the best and worst attributes using Information Gain [18]. Two steps taken on feature selection for the Greedy Algorithm with attribute subset selection [18] were Stepwise Forward Selection and Stepwise Backward Elimination.
The best and worst attributes can be determined by using statistical significance tests which assume that attributes are not related to each other (independent) [18]. The steps in determining the evaluation in the selection of attributes can use the Information Gain used in the Decision Tree [18].
The selection of attributes using Information Gain is to select the highest gain, and the formula used is as follows where the first step is to find the entropy value as follows:
( ) = − ( )
( ) is to know the value of entropy. If an attribute has different values, then use formula as follows:
( ) = | |
| | ( )
Then to get the gain result where the formula becomes:
( ) = ( ) − ( )
The results of the feature selection in this study show for the Pap smear single image classification for two class classification, selected three most relevant features are N/C ratio, cytoplasm brightness, and nucleus perimeter. While the feature selection for image classification in seven classes there are six most relevant features, namely nucleus area, N/C ratio, cytoplasm brightness, nucleus a roundness, shortest diameter cytoplasm, longest diameter cytoplasm, and nucleus relative position.
2. Meta Learning Bagging
Meta learning Bagging is done as a continuation of the training process. Bagging technique is one of these ensemble techniques, and this technique on single image classification of Pap smears separates training data into new training data with random sampling and builds new training-based data model.
The ensemble technique is a successful technique for dealing with unbalanced datasets though not specifically designed for unbalanced data problems [19]. Bagging algorithm for classification [20]:
Algorithm Bagging input
Data set D={( , )}
Base learning algorithm The number of iterations T
1. for t=1 to T do
2. ℎ = ( , ) /* is the bootstrap distribution */
end for
output H(x)=max ∑ (ℎ ( ) = ) /* I(x)=1 if x is true, and 0 otherwise
*/
3. Learning Algorithm Naïve Bayes
The Naïve Bayes equation using the gaussian distribution because the Pap smear dataset is a numeric type data. In the
Gaussian distribution, calculated mean and standard deviation on all attributes, here are the equations used:
( , , ) = 1
√2
( )
According to [18] Naïve Bayes in classification can produce high and fast accuracy when applied to large data. The value of the attribute in a particular class does not depend on the value of other attributes; this argument can be called class- conditional independence so that the calculation can be made simpler and is called "naive". Model performance is measured by using confusion matrix, so it can be analyzed on how well the performance of the classifier can recognize the tuples of different classes. The Confusion matrix provides performance assessment of object classification correctly or falsely [21].
Confusion matrix is two dimensional matrices that describe the comparison between prediction results and reality. Here is the equation of confusion matrix model:
= +
+ + +
= = =
+
= =
+
= +
= +
− = (1 + )
( + )
− =
AUC =
F-measure combines recall or sensitivity and precision to produce effective metrics for information retrieval in sets that contain imbalance issues. Model analysis step was done by using t-test and Friedman test.
The paired-samples t-test is a procedure used to compare the mean of two variables, previous and subsequent variables using a model. The t-test can also be called z-test in the statistics, and this test was performed to determine whether there was a significant difference in the two variables tested.
The algorithm for comparison of z-test performance [21] as follows:
Algorithm z-test input:
1. Section (classification accuracy) for the first sample (model );
2. The section (classification accuracy) for the second sample (model );
3. The sample size on for the first sample;
4. The sample size on for the second sample;
Then − is calculated based on t value for comparison with the following equation:
| | = .
. .| − | . Where:
= . + .
+ , = 1 − , + − 2 output:
Significance level for the different accuracy of the two models.
In statistics if p > 0.05, it means that there is no significant difference, but if p-value < 0.05, then there is a significant difference between two models.
III. RESULT AND DISCUSSION
The series of research designs was evaluated by using a conventional dataset of Pap smear consisting of a single cell that has 20 features. Experiments on this research used Rapidminer application and application development using IDE (Integrated Development Environment) NetBeans using Javanese. Measurements on the classification of 2 classes on single Pap smear image data consisted of accuracy result, AUC, sensitivity, f-measure, and G-mean. Comparison of measurements was made to the results of the four models that were the Naïve Bayes model (NvB), the Naïve Bayes and Bagging model (NvB+BG), the Naïve Bayes and Greedy Forward Selection (NvB+GFS) models, the Naïve Bayes model with Greedy Forward Selection and Bagging (NvB+GFS+BG).
Table II contains a comparison of all four models.
TABLE II. RESULT COMPARISON OF ALL FOUR MODELS
Classification Result Accuracy Model
NvB NvB+BG NvB+GFS NvB+GFS+BG
2 Kelas
Accuracy 91.71% 91.71% 92.15% 92.14%
AUC 0.956 0.957 0.961 0.959 sensitivity 98.37% 98.37% 97.33% 97.33%
f-measure 0.95 0.95 0.95 0.95 G-mean 0.85 0.85 0.87 0.87 7 Kelas Accuracy 55.62% 55.49% 62.59% 63.25%
Figure 2 is an accuracy comparison graph of NvB model, NvB+BG model, NvB+GFS model, and NvB+GFS+BG model.
Fig. 2. Graph of Model Comparison Accuracy
The result of performance measurement was furthermore analyzed by using t-test to know the best model. The t-test was done by comparing two models and measuring p-value. If p- value < alpha value (0.05), then there is a significant difference
between the two models compared. On the contrary if p-value
> alpha value, then there is no significant difference.
TABLE III. T-TEST MODEL OF NVB WITH NVB+GFS
NvB NvB+GFS
Mean 0.73665 0.7737
Variance 0.065124405 0.04368968
Observations 2 2
Pearson Correlation 1 Hypothesized Mean Difference 0
df 1
t Stat -1.134762634
P(T<=t) one-tail 0.229932461 t Critical one-tail 6.313751515 P(T<=t) two-tail 0.459864922 t Critical two-tail 12.70620474
The t-test was performed on accuracy by using statistical methods to test the hypotheses on Naïve Bayes (NvB) with Naïve Bayes and Greedy Forward Selection (NvB+GFS).
H0: There is no difference in the accuracy average value of NvB and NvB+GFS.
H1: There is a difference between the accuracy average value of NvB and NvB+GFS.
The average value of accuracy of the NvB+GFS model is higher than the NvB model of 0.7737. In the statistical test the alpha value is determined at 0.05. If the p value is smaller than alpha (p < 0.05), then H0 is rejected and H1 is accepted, so there is a significant difference between the compared models, but if the p value is greater than the alpha value > 0.05), then H0 is accepted and H1 is rejected, so there is no significant difference between the comparison model (Table III)
TABLE IV. P-VALUE FOR EACH MODEL
Model P-value
NvB vs NvB+BG 0.5
NvB vs NvB+GFS 0.459
NvB vs NvB+GFS+BG 0.464 NvB+BG vs NvB+GFS 0.460 NvB+BG vs NvB+GFS+BG 0.464 NvB+GFS vs NvB+GFS+BG 0.509
It can be seen that the value of P (T <= t) is at 0.459 and this shows that the p-value is bigger than the alpha value (0.459
> 0.05) so the hypothesis of H0 is accepted and H1 is rejected.
That the hypothesis of H1 is rejected means that there is no significant difference between the NvB model and the NvB+GFS model. The complete p-value for each model is given in Table IV. All the results of p-value can be concluded that there is no significant difference between NvB model with other model.
91.71% 91.71% 92.15% 92.14%
55.62% 55.49% 62.59% 63.25%
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
NvB NvB+Bag NvB+GFS NvB+GFS+Bag Accuracy
2 Classes 7 Classes
Furthermore, Friedman test was done to find out the best model, Table IV shows the p-value of Friedman test. The results of the Friedman test support the analysis that the integration of Bagging and Greedy Forward Selection in Naïve Bayes shows no significant difference for the classification results.
Tables V and VI show the same analysis results that for this dataset the condition of data imbalance and feature selection in this study does not make a significant difference.
TABLE V. P-VALUE FRIEDMAN TEST
Model NvB NvB+BG NvB+GFS NvB+GFS+BG
NvB 1 0.980 0.529 0.529
NvB+BG 0.980 1 0.304 0.304
NvB+GFS 0.529 0.304 1 1.000
NvB+GFS+BG 0.529 0.304 1.000 1 TABLE VI. MODEL SIGNIFICANCE OF FRIEDMAN TEST FOR THE
CLASSIFICATION
Model NvB NvB+BG NvB+GFS NvB+GFS+BG
NvB No No No No
NvB+BG No No No No
NvB+GFS No No No No
NvB+GFS+BG No No No No
The rank of the average was derived from the model comparison and it is known that the average rating value was divided by the number of samples used. The rank difference of the mean of each model was compared with Critical Difference which can be seen in Table VII. Critical Difference can be calculated by using =
, , ( ) and in this experiment the value of Critical Difference is 3.3399. If the value in Table VII is greater than that of Critical Difference, then the model has a significant difference compared to other models.
TABLE VII. PAIRWISE OF DIFFERENCE
Model NvB NvB+BG NvB+GFS NvB+GFS+BG
NvB 0 0.500 -1.750 -1.750
NvB+BG -0.500 0 -2.250 -2.250
NvB+GFS 1.750 2.250 0 0.000
NvB+GFS+BG 1.750 2.250 0.000 0
Critical difference: 3.3399
Figure 3 is the average rating of the friedman test. It is known that NvB+GFS and NvB+GFS+BG model have the highest average rating, followed by NvB model, and NvB+BG model have the lowest rating average.
Fig. 3. Rank of Friedman Test
IV. CONCLUSION
Pap smear image classification is required in order to facilitate the early detection process of cervical cancer.
Accurate classification is a crucial procedure for detecting pre- cancerous conditions and abnormal cell changes. Pap smear image datasets generally have class imbalance and irrelevant features. The results of this study indicate that these conditions cannot be handled with Bagging and Greedy Forward Selection integration although in previous studies the integration has resulted in significant improvements in the performance of classification accuracy. In this research, Naïve Bayes algorithm was used to classify Pap smear image for the classification of two classes that are normal and abnormal class. Classification was also performed for seven classes, consisting of three normal classes and four abnormal classes. This study proposed Naïve Bayes model and the feature selection of Greedy Forward Selection (NvB+GFS) to overcome the issues of irrelevant features while to overcome the problem of class imbalance used Naïve Bayes model with Greedy Forward Selection and Bagging (NvB+GFS+BG). The experiment results in this study got the highest accuracy value on NvB+GFS model of 92.15%
for the classification of 2 normal and abnormal classes.
Accuracy for the classification of 7 classes was just at 63.25%
in NvB+GFS+BG model and was still possible to improve.
The accuracy result of this research was compared to previous research [11[ and [12] using the same dataset which is Herlev Data although the features, methods and algorithm that are different. The result of classification of 2 classes with the method of ANN at 83.39% [11] and ED at 85.24% [11] while the result of this research with the method of NvB+GFS is higher at 92.15% although this result is lower than method accuracy of SVM at 93.72% [11] and lower than ConvNet method at 98.3% [12]. In this study, classification of 7 classes has also been done with the model of NvB+GFS+BG with the accuracy of 63.25% whereas in [11] and [12] classification of 7 classes has not been done.
To determine the best models in the Pap smear image classification in this research, a friedman test was performed.
The result of friedman test is known that the rating average with the highest grade for all the classification are NvB+GFS model and NvB+GFS+BG model. This research is a preliminary study of research on datasets which have class imbalance and
1.25
1.75
3.50 3.50
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00
NvB+BG NvB NvB+GFS NvB+GFS+BG Rank
irrelevant features in Pap smear images. Further research will apply some alternative method to classify single image Pap smear into seven classes so that the better accuracy can be obtained.
ACKNOWLEDGMENT
Dwiza Riana would like to thank RISTEKDIKTI. This research was supported by The Ministry of Research, Technology, and Higher Education. This work is using the data from: Pap smear Benchmark Data for Pattern Classification J.
Jantzen, J. Norup, G. Dounias, and B. Bjerregaard, University Dept. of Pathology Herlev Ringvej 75, DK-2730 Herlev, Denmark.
REFERENCES
[1] E. Somers, “International Agency for Research on Cancer.,” C. Can. Med. Assoc. J. = J. l"Association medicale Can., vol. 133, no. 9, pp. 845–846, 1985.
[2] E. Martin and J. Jantzen, Pap-Smear Classification.
2003.
[3] D. Riana, D. E. O. Dewi, D. H. Widyantoro, and T. L.
R. Mengko, “Color canals modification with canny edge detection and morphological reconstruction for cell nucleus segmentation and area measurement in normal Pap smear images,” AIP Conf. Proc., vol. 1589, no. Icmns 2012, pp. 414–417, 2014.
[4] D. Riana, M. E. Plissiti, C. Nikou, D. H. Widyantoro, and T. L. R. Mengko, “Inflammatory cell extraction and nuclei detection in Pap smear images,” Int. J. e-Health Med. Commun., vol. 6, no. 2, pp. 27–43, 2015.
[5] D. Riana, D. H. Widyantoro, and T. L. Mengko,
“Extraction and classification texture of inflammatory cells and nuclei in normal pap smear images,” Proc. - 2015 4th Int. Conf. Instrumentation, Commun. Inf.
Technol. Biomed. Eng. ICICI-BME 2015, pp. 65–69, 2016.
[6] D. Riana, D. Ekashanti, O. Dewi, D. H. Widyantoro, and T. L. R. Mengko, “Segmentation and Area Measurement in Abnormal Pap Smear Images Using Color Canals Modification with Canny Edge Detection,” in International Conference on Women’s Health in Science & Engineering, 2012, pp. 1–4.
[7] J. Hyeon, C. Ho-Jin, B. D. Lee, and K. N. Lee,
“Diagnosing Cervical Cell Images Using Pre-trained Convolutional Neural Network as Feature Extractor,”
in Big Data and Smart Computing (BigComp), 2017 IEEE International Conference on, 2017, pp. 390–393.
[8] Y. Ramdani, D. Riana, and A. Mubarok, “Analisa Size Citra Sel Tunggal Nukleus Menggunakan Global Threshold dan Operasi Kanal Warna.”
[9] A. Herliana and D. Riana, “KLASIFIKASI SEL TUNGGAL PAP SMEAR TEKSTUR TERSELEKSI
MENGGUNAKAN CORRELATION-BASED FEATURES SELECTION BERBASIS DECISION TREE J48,” pp. 144–147, 2014.
[10] D. Kashyap, A. Somani, and J. Shekhar, “Cervical Cancer Detection And Classification Using Independent Level Sets And Multi SVMs,” 39th Int.
Conf. Telecommun. Signal Process., pp. 523–528, 2016.
[11] M. S. A and S. Jereesh,
“AutomatedCervicalCancerDetectionthroughRGVFse gmentationandSVM Classification.pdf,” pp. 663–669, 2015.
[12] L. Zhang, L. Lu, I. Nogues, R. Summers, S. Liu, and J.
Yao, “DeepPap: Deep Convolutional Networks for Cervical Cell Classification,” IEEE J. Biomed. Heal.
Informatics, vol. XX, no. c, pp. 1–1, 2017.
[13] Y. E. Kurniawati and A. E. Permanasari, “Comparative Study on Data Mining Classification Methods for Cervical Cancer Prediction Using Pap Smear Results,”
2016.
[14] Y. Ma, G. Luo, X. Zeng, and A. Chen, “Transfer learning for cross-company software defect prediction,” Inf. Softw. Technol., vol. 54, no. 3, pp.
248–256, 2012.
[15] T. M. Khoshgoftaar, J. Van Hulse, and A. Napolitano,
“Comparing boosting and bagging techniques with noisy and imbalanced data,” IEEE Trans. Syst. Man, Cybern. Part ASystems Humans, vol. 41, no. 3, pp.
552–568, 2011.
[16] R. S. Wahono, N. S. Herman, and S. Ahmad, “Neural Network Parameter Optimization Based on Genetic Algorithm for Software Defect Prediction,” vol. 20, no.
10, pp. 1951–1955, 2014.
[17] Fitriyani and R. S. Wahono, “Integrasi Bagging dan Greedy Forward Selection pada Prediksi Cacat Software dengan Menggunakan Naïve Bayes,” J.
Softw. Eng., vol. 1, no. 2, pp. 101–108, 2015.
[18] J. Han, M. Kamber, and J. Pei, Data Mining: Concepts and Techniques. 2012.
[19] I. H. Laradji, M. Alshayeb, and L. Ghouti, “Software defect prediction using ensemble learning on selected features,” Inf. Softw. Technol., vol. 58, pp. 388–402, 2015.
[20] X.-Y. Liu, J. Wu, and Z.-H. Zhou, “Exploratory Undersampling for Class Imbalance Learning,” IEEE Trans. Syst. Man Cybern., vol. 39, no. 2, pp. 539–550, 2009.
[21] F. Gorunescu, Data mining: concepts and techniques.
2011.
