Classifiers Evaluation: Comparison of Performance Classifiers Based on Tuples Amount

Classifiers Evaluation: Comparison of Performance

Classifiers Based on Tuples Amount

Mochammad Yusa

[email protected]

Ema Utami

Department of Computer Science, Magister Teknik Informatika

Universitas AMIKOM Yogyakarta, Indonesia [email protected]

Abstract— The aim of this study is to compare some classifiers’ performance related to the tuples amount. The different metrics of performance has been considered, such as: Accuracy, Mean Absolute Error (MAE), and Kappa Statistic. In this research, the different numbers of tuples are considered as well. The readmission process dataset of Diabetic patients, which has been experimented, consists of 47 features and 49.736 tuples. The methodology of this research starts from preprocessing phase. After that, the clean dataset is divided into 5 subsets which represent every multiple of 10.000 tuples randomly. Each particular subset will be validated by three traditional classifiers i.e. Naive Bayes, K-Nearest Neighbor (k-NN), and Decision Tree. We also implement some setting parameters of each classifier except Naïve Bayes. Validation method used in this research is 10-Fold Cross-Validation. As the final conclusion, we compare the performance of classifiers based on the number of tuples. Our study indicates that the more the number of tuples, the lower and weaker the MAE and Accuracy performances whereas the kappa statistic performance tend to be fluctuated. Our study also found that Naïve Bayes outperforms k-NN and Decision Tree in overall. The top classifiers performances were reached in a 20.000-tuple evaluation.

Keywords—classifiers; classification; Naïve Bayes; k-NN, Decision Tree; Accuracy; MAE, Kappa Statistic.

I. INTRODUCTION

Statistically, the measurement of quality towards diabetic disease medication can be seen from some factors i.e. medication process, ethnic, race, life style, age, and etc. In the diabetic disease, features discrepancy related to the patients has significant impacts for medication quality to glycemic serum in human body. Diabetic disease was linked to the quality of hospital medication by observing rates of Readmission Process [1, 2] because of hospital medication toward diabetes treatment tends to be expensive [3]. By using classification technique of data mining, the prediction of readmission process especially for diabetic patients can assist the sufferers to save their energy and money.

Classification technique is one of data mining methods that has been growing significantly. There are so many classifications implemented to solve many problems such as manufacturing [4], agriculture [5], economic [6], education [7], and health [8]. In classification techniques of data mining, there are algorithm models with different performances.

Reference[9] states that an algorithm model with good performances has uncertainly good performances compared with different dataset and evaluated with other classifiers.

Reference [10] showed that classification is so dependent on the applied algorithm. But in that research, there is no explanation about relationship between performance and the number of tuple. Other researches about the classifiers’ performance differences and finding the best classifiers importance will be implemented as following:

Reference [11] compared three classifiers. They chose Naïve Bayes, Lazy Classifier (k-NN), and Decision Tree on Leukemia Cancer Dataset which consisted of 7130 features and 72 tuples. Accuracy and time execution were the targets of performance measurement in this study. The result showed that Naïve Bayes outperformed Decision Tree and Lazy Classifier on either accuracy or time execution.

Reference [12] studied the similar measurement of classification performance, but dataset they chose was different. They evaluated performance of classifiers from Heart Disease Dataset consisting of 14 features and 294 tuples. They conclude that k-NN had best performance on accuracy than Naïve Bayes and Decision Tree. Based on their study, k-NN had an impressive accuracy where it was implemented on less features of dataset.

Mittal and Gill [13] analyzed effectiveness of classifiers in three different datasets based on accuracy, Statistical Kappa, and Mean Absolute Error (MAE). Result showed the dependence from the number of tuples and features as well to the classifiers performances.

Performance evaluation is also used to predict whether client on a bank will deposit or not. Karim and Rahman [14] evaluated two different classifiers i.e. Decision Tree based on Gain Ratio Method (C4.5) and Naïve Bayes. This study concluded that Decision Tree based on Gini Index method has better accuracy than Naïve Bayes at 17 features and 4521 tuples.

Based on the studies explained above, performance of classifiers refers to number of features or attributes and tuples or records.

Because of significant dissemination of database in many fields, choosing classifier method becomes a risk. In addition to

choose the right method, it is also important to find the performance classifiers to handle the big data including number of tuples and features. This research paper focuses on dimensionality of tuples amount. We analyze one dataset with the same feature amount but we simultaneously test with different tuples amount to find out relationship between classifiers performance and tuples amount in one dataset dimensionality. The considered performances are accuracy, MAE, and Kappa Statistic while the considered classifiers are Decision Tree, k-NN, and Naïve Bayes. We also explore classifiers by implementing some methods called setting parameters. The proposed work will detail the way how we experiment.

This study is organized in such a way that the section 2 and 3 discussed about proposed work and material theory. Section 4 discusses about training set consisting data preparation and dataset. Result experiment is detailed on section 5. The analysis of comparison performance between classifier and tuples amount will be explained on section 6. Conclusion is given on section 7.

II. PROPOSED WORK

Fig 1. Shows diagram blogs of proposed methods. There are two main processes in our research; data preparation and evaluation. Data preparation process provides the way to reduce incomplete data, missing values, and duplicated data. It will be explained entirely in section 5.

Data Preparation Final dataset Categorized to Subset A, B, C, D, E evaluated by implementing considered classification algorithms Applying 10 Fold Cross Validation Performance measurement Comparing the performance with tuples amount conclusion

Figure 1. Proposed method

After getting the final dataset, we divide it into 5 subsets or portions with multiple of 10.000 tuples. Then, every subset is validated by using chosen classifiers.

The present work has been validated with Rapid Miner 7.0.0 tool to analyze the performance of three different classifiers. They are Decision tree, Naive Bayes, and K-Nearest Neighbor. We evaluated the algorithms with some setting parameters. Information Gain, Gain Ratio, and Gini Index are the methods we consider to evaluate. The distances of k-NN evaluated are k = {1, 11, 101, 1001}. There is no exploration on Naïve Bayes Method. On evaluation process, we validated the training sets by using 10-fold cross validation.

After validating, the relationship of three portion of performances will be discussed to get the final conclusion.

III. MATERIALS

A. Decision Tree

The most popular classification technique to use is decision tree [15]. Decision tree represents tree structure where every node represents attribute, branch represents value of each attribute, and leaf represents classes. There are three setting parameters of Decision Tree Classifier. This research provides the explorations: Information Gain, Gain Ratio, and Gini Index.

1) Information Gain

Information gain forms a measurement of correlation in the parametric model that describes dependences between two random variables X and Y. This method of Optimal Splitting Point is typically used to ID3 algorithm [16].

2) Gini Index

Based on [16], one of criteria as best optimal splitting point of Decision Tree is Gini Index. This method is usually used by CART (C&RT) and SPRINT algorithm. Gini Index represents inequality to acquisition distribution and has the value of 0 to 1. The lower the value of Gini Index, the higher the measurement of equivalent.

3) Gain Ratio

Gain Ration is an improvement method of information gain which can reduce the attribute bias from branches of Decision Tree [16]. This method is usually used in C4.5 Algorithm.

B. Naive Bayes

Naïve Bayes classifier is simple and has great accuracy and rapid time execution when applied to a large database [15]. This algorithm works on assumption that an impact of given feature value is independent toward the other feature values. This assumption is called class conditional independent [17]. Naïve Bayes classification also can predict member class probability toward tuples which is given to other certain classes [18].

C. K-Nearest Neighbor

K-Nearest Neighbor Classification is a supervised method [15]. The aim of this classifier is to clarify object based on features and training samples [19].

D. Accuracy

The aim of measurement of accuracy performance is to clarify total tuples which is predicted properly.

Measurement of accuracy is a comprehensive valid classification from iteration as much as amount of k divided to total of early tuples [15].

E. Mean Absolute Error (MAE)

Mean Absolute Error (MAE) is a quantity used to measure the accuracy of the prediction toward encountered results. MAE draws value of an algorithm capability [13]. The lower the MAE performance, the stronger the algorithm capability. Measurement of MAE is relatively simple. MAE sums absolute values from Error rates to reach the comprehensive error values and then divide them to total number of tuples [20].

F. Kappa Statistic

Kappa Statistic is a statistical analytical measurement based on interpreting compatibility or degree of agreement for qualitative data [13]. Basically, Kappa Statistic performance draws analysis among different classes. The higher the Kappa Statistic performance, the more considerable the performance of classifiers.

IV. TRAINING DATA

A. Dataset

This research uses the dataset originated from UCI Machine learning which can be freely downloaded to Data Mining researchers. The dataset has been collected from The Health Facts Database (Cerner Corporation) and emanates from diabetic patients treated on that institution.

The original database consists of 41 tables with 117 features or attributes. There are about 74.036.643 data. Stract, et. all have preprocessed the database based on what attributes affect the diabetic disease [21]. They produce 55 attributes and 101.776 records after preprocessing process. The description of dataset can be seen in https://archive.ics.uci.edu/ml/-datasets/Diabetes+130-US+hospitals+for+years+1999-2008. But the available dataset is still redundant, noisy, and missing value. So in this research, we reprocess data preparation to minimize the inaccurate dataset. The process of preprocessing will be explained in next chapter.

B. Data Preparation

In this chapter, we analyze the final dataset which will be trained to get performances of classifiers. Generally, preprocessing data includes three main steps. These steps are considered important because of many incomplete data prepared by References [21]. The steps will be represented in following Fig 2:

Dataset Feature selection

Missing Data Analysis Duplicated data

analysis

Final Dataset 5 subsets of

multiple 10.000

Figure 2. Data preparation process

Figure 2 shows the phase of preprocessing to find out the final candidates of dataset in this study. First phase is feature selection. This phase is to choose or prune back attributes or features. In available dataset, we analyze the function of the attributes considering the ineffectual aspects of them to the readmission process of diabetic patients. Payer code and patient code are attributes we considered as useless attributes because they are ineffectual and multiple data inclusive. After feature selection process is done, we analyzed tuples amount which have many missing values. According to data analysis, we have found four attributes with considerably missing values. The attributes are weight 97 %, race 2 %, diagnosis 1 %, and medical specialty 47 %. We removed all tuples with missing values. In third phase, we identified duplicated data to prevent unclean tuples. After we have analyzed, we find no duplicated tuples. After three steps has done, we determine the final dataset which will be evaluated. The dataset consists of 47 attributes and 49.735 records.

V. EXPERIMENTAL RESUTS

After getting the final dataset, we split the dataset into 5 subsets spatially based on tuples amount. In first experiment, we evaluated classifiers by training 10.000 tuples and using 10-Fold Cross-Validation. We called the dataset as Subset A. Then we divide into each multiple of 10.000 tuples. Hence, there are the experimental results of 5 evaluation subsets (Fig. 2). The part of spatial subset will be evaluated by implementing classifiers methods. The result of validation will be detailed on following:

A. Subset A

Subset A is a part of subset which represents 10.000 data of training set. After we evaluate the classifiers by implementing the classifiers, we found some results. For Decision Tree classifier, implementation with Gain Ratio Setting Parameter has the best performance related to the accuracy and Kappa Statistic. It has 55,60 % accuracy and 0,135 of Kappa Statistic, But the MAE produced has the same result with Gini Index

setting parameter. Overall, in case of Decision tree evaluated to Subset A, the best performance belongs to Gain Ratio Setting Parameter.

TABLE I. TABLE I.EVALUATION RESULT OF SUBSET A Classifier _Parameter Setting Accuracy MAE _Statistic Kappa

Decision Tree Information Gain 49,85 0,506 0,083 Gini Index 49,89 0,504 0,083 Gain Ratio 55,60 0,504 0,135 k-NN K=1 48,91 0,511 0,068 K=11 57,82 0,514 0,113 K=101 59,84 0,520 0,057 K=1001 59,18 0,530 0,001 Naïve Bayes - 60,37 0,491 0,195

In evaluating k-NN to Subset A, k-NN with k=101 performs well in accuracy than any other k setting parameters. It has 59,84% of accuracy, but for the resulted MAE and Kappa Statistic, they tend to obtain more insignificant results than k=1 and k=11. K=1 setting parameter performs best of MAE than others whereas k=11 performs well in Kappa Statistic performance. So, it can be concluded that every single setting parameter has got the preeminence itself.

There is no setting parameter implemented in Naïve Bayes Classifier. In this case, Naïve Bayes is listed as the best performer. Naïve Bayes has 60,37% accuracy, 0,491 MAE, and 0,195 Kappa Statistic. In the nut shell, Naïve Bayes performs better than any other classifiers either in Accuracy performance or in MAE and Kappa Statistic of this experiment.

B. Subset B

The evaluated training set in second experiment is 20.000 tuples. The results are shown in Table II. The best performance of Decision Tree classifier is the method with Gain Ratio setting parameter. Resulted performance is 56,52%, 0,490, and 0,128 of accuracy, MAE, and Kappa Statistic respectively. Then, k-NN with k=101 obtains the best accuracy with 61,22% while the best MAE and Kappa statistic are produced by k=1 with 0,504 and k=11 with 0,111. Last, Naïve Bayes only produces good performance in Kappa Statistic and MAE than any other classifiers but it has lower accuracy than k-NN with k=101. Naïve Bayes produces 60,91% accuracy, 0,479 MAE, and 0,181 Kappa Statistic.

TABLE II. EVALUATION RESULTS OF SUBSET B Classifier Setting Parameter Accuracy MAE _Statistic Kappa

Decision Tree Information Gain 51,51 0,490 0,095 Gini Index 51,23 0,491 0,096 Gain Ratio 56,52 0,490 0,128 k-NN K=1 49,63 0,504 0,072 K=11 58,80 0,506 0,111 K=101 61,23 0,509 0,066 K=1001 60,46 0,516 0,002 Naïve Bayes - 60,91 0,479 0,181 C. Subset C

Experiment results for Decision Tree classifier show that setting parameter of Gain Ratio ensures the best of accuracy and Kappa Statistic than any other setting parameters of Decision Tree (Table III). Accuracy was 57,26 % and Kappa Statistic was 0,133. Whereas, MAE performance verified no significant difference.

TABLE III. EVALUATION RESULT OF SUBSET C

Classification Setting Parameter Accuracy MAE _Statistic Kappa

Decision Tree Information Gain 50,46 0,499 0,089 Gini Index 50,70 0,497 0,095 Gain Ratio 57,26 0,496 0,133 k-NN K=1 49,52 0,505 0,073 K=11 58,69 0,508 0,111 K=101 60,53 0,512 0,056 K=1001 59,70 0,520 0,002 Naïve Bayes - 60,48 0,489 0,170

In k-NN evaluation to Subset C, k-NN with k=101 indicates the best performance of accuracy with 60,53 % while the best MAE was obtained by k=1 of setting parameter and best Kappa Statistic is k=11 with 0,111. The third was Naïve Bayes Classifier. We consider that Naïve Bayes has best performance at all. It results 60,48 % of accuracy, 0,489, and 0,170 Kappa Statistic. Although the accuracy has lower performance than k-NN with k=101 but the gap is not too significant. So, for third experiment, we conclude that Naïve Bayes still outperforms than any other classifiers.

D. Subset D

Table IV refers to result of classifiers evaluation to Subset D (40.000 tuples). The result will be analyzed by classifiers. First is Decision Tree with any setting parameters. The best performance at all of Decision Tree setting parameter is Gain Ratio. It outcomes 55,11 % of accuracy, 0,514 of MAE, and 0,133 of Kappa Statistic. The second is k-NN evaluation to Subset D. in k-NN evaluation, the best accuracy is resulted from k=101 with 57,85 % while the best MAE is obtained from k=1 and best Kappa Statistic is k=11 with 0,112. Based on data, best on all performance indicators is k-NN classifiers with k=101. The third is Naïve Bayes classifier. In this experiment, Naïve Bayes achieves best performances of tested indicators. Naïve Bayes results 58,35 % of accuracy, 0,506 of MAE, and 0,182 of Kappa Statistic. So, in case of evaluating Subset D, Naïve Bayes outcomes best results than any other classifiers.

TABLE IV. EVALUATION RESULT OF SUBSET D

Classifier Setting Parameter Accuracy MAE _Statistic Kappa

Decision Tree Information Gain 48,81 0,516 0,092 Gini Index 48,9 0,516 0,095 Gain Ratio 55,14 0,514 0,146 k-NN K=1 48,09 0,519 0,07 K=11 56,25 0,522 0,112 K=101 57,85 0,528 0,072 K=1001 56,58 0,537 0,014

Naïve Bayes - 58,35 0,506 0,182

E. Subset E

Table V indicates results of Subset E experiment with 49.736 tuples. In case of Decision Tree experiment, Optimal Splitting Point of Gain Ratio method shows best performances in accuracy, MAE, and Kappa Statistic. Its Measurement performances are 54,13% in accuracy, 0,524% in MAE and 0,138% in Kappa Statistic. In k-NN evaluation, we found the similar result as experiment in the last subset. The best performance of accuracy is reached by using setting parameter of k=101 with 57,06 %, MAE is k=1 with 0,523, and Kappa Statistic is k=11 with 0,109. Although the score of performance tend to be deceasing. Naïve Bayes still has the performance of all with 57,52% of accuracy, 0,512 of MAE, and 0,182 of Kappa Statistic.

TABLE V. EVALUATION RESULT OF SUBSET E Classifier _Parameter Setting Accuracy MAE _Statistic Kappa

Decision Tree Information Gain 47,82 0,525 0,087 Gini Index 47,84 0,525 0,086 Gain Ratio 54,13 0,524 0,138 k-NN K=1 47,66 0,523 0,072 K=11 55,09 0,528 0,109 K=101 57,06 0,534 0,085 K=1001 55,98 0,543 0,032 Naïve Bayes - 57,52 0,512 0,182 VI. DISCUSSION

In this chapter, we will discuss about relationship between tuples amount and classifiers performance. We divide into 3 divisions; accuracy, MAE, and Kappa Statistic.

A. Accuracy

Based on the data which are displayed in Fig. 3, Decision Tree classifier accomplish the peak performance in Subset B. It has increased from Subset A to Subset B. But, results of Evaluating Subset C to E shows that it is decreasing. This similar phenomenon also occurs in k-NN with any setting parameters and Naïve Bayes as well. So, it can be inferred that productivity of accuracy performance will decrease if the numbers of tuples are increasing, but the ideal accuracy achieve the best performance if it evaluates with 20.000 tuples.

Figure 3. Accuracy performance

B. MAE

Performance of Decision Tree classifiers with any setting parameters achieves top performance in subset B consisting 20.000 tuples on average. The raise of performance has happened from subset A to Subset B in Decision Tree, k-NN, and Naïve Bayes classifiers (Fig. 4). Then the decrement occurs from Subset B to Subset E. It transpires to all classifiers with all setting parameters.

MAE performance is also affected by the number of data. This can be proved with the score increment of MAE performance from Subset B to E. This indicates that the more the number of data, the lower the MAE performance. The number of tuples also shows that while the number of tuples increases, Error level of classification also increases.

Figure 4. MAE performance

C. KAPPA STATISTIC

Figure 5 shows that Kappa Statistic performance results are not depending on tuples amount. The outcomes of kappa statistic performance tend not to be patterned and are always fluctuating. Decision Tree with Information Gain setting parameter shows that the increment of performance is displayed when evaluated on Subset A to Subset B then it decreases from Subset B to Subset C. Subset C to Subset D, it increases again and the last it decreases from Subset D to Subset E.

In k-NN evaluation results, they also tend to be fluctuated. Kappa Statistic of k-NN evaluation with k=1 shows that the highest performance is in Subset A. The decrement occurs in Subset B and then Subset C to Subset D result is increasing. Another example is k-NN with k=11. It figures out the fluctuated tendency of Kappa Performance as well. From Subset A to Subset B, kappa Statistic get increased. Then B to C, they are in the same result. After that, it decreases until the best performance appears in Subset E.

Fluctuated Kappa Statistic performance is also displayed from evaluation towards Naïve Bayes Classifier. The result shows that decrement of kappa statistic happens from Subset A to B and C, where subset C evaluation was the lowest performance. Then, there is Increment from Subs C to Subset D. but, Subset D and Subset E has the same result.

VII. CONCLUSION

This study has evaluated and validated some classifiers with any setting parameters toward a dataset consisting 47 features and 49.735 tuples. We also validated the dataset after dividing into 5 dimensions i.e. 10.000 (Subset A), 20.000(Subset B), 30.000 (Subset C), 40.000 (Subset D), and 49.735 (Subset E) tuples. Overall, Naïve Bayes outperforms any other classifiers with any other setting parameters. Peak Performance is found on evaluation toward Subset B (20.000 tuples) while the Naïve Bayes outperforms any other classifiers with setting parameters. Our study also finds that tuples amount has influenced the accuracy and MAE performances even though the Kappa statistic has not. Study shows that Kappa Statistic Performance tend to fluctuate. Based on that phenomenon, we also found that the more the number of the tuples, the lower and weaker the MAE and accuracy performances.

REFERENCES

[1] Ramirez, S., et.all. “Hemoglobin A1c Levels and Mortality in the Diabetic Hemodialysis Population, Findings from the Dialysis Outcomes and Practice Patterns Study (DOPPS)”. Journal of Diabetes Care, Volume 35. 2012. pp. 2527-2532.

[2] Dungan, K. “The effect of diabetes on hospital readmissions”. Jurnal of Diabetes & Science Technology. Volume 6, No. 5. 2012. pp. 1045-1052. [3] Rubin, D. J. “Hospital readmission of patients with diabetes”. Current

Diabetes Reports Volume. 15 Number. 4. 2015. Pp.1-9.

[4] Son, H., Kim, C., Hwang, N., Kim, C., & Kang, Y. “Classification of major construction materials in construction environments using ensemble classifiers”. Elsevier - Advanced Engineering Informatics Vol. 28. 2014. pp.1-10.

[5] Taghizadeh-Mehrjardi, R., Nabiollahi, K., Minasny, B., & Triantafilis, J. “Comparing data mining classifiers to predict spatial distribution of. Geoderma” 253-254, Elsavier. 2015. Pp.67-77.

[6] Koutanaei, F. N., Sajedi, H., & Khanbabaei, M. “A hybrid data mining model of feature selection algorithms and ensemble learning classifiers for credit scoring”. Journal of Retailing and Consumer Services 27. 2015. pp.11-23.

[7] Ragab, A. H., Noaman, A. Y., Al-Ghamdi, A. S., & Madbouly, A. I. “A comparative analysis of classification algorithms for students’ college enrollment approval using data mining. In Proceedings of the 2014 Workshop on Interaction Design in Educational Environments”. ACM. 2014. pp.106.

[8] Upadhyaya, S., Baker-Demaray, K., & Farahmand, T. “Comparison of NN and LR classifiers in the context of screening Native American elders with diabetes”. Elsevier Expert Systems with Applications Vol 40. 2013. pp.5830-5838.

[9] Rahman, R. M., & Afroz, F. “Comparison of Various Classification Techniques Using Different Data Mining Tools for Diabetes Diagnosis”. Journal of Software Engineering and Applications Vol.6. 2013. pp. 85-97. [10] Temurtas, H., Yumusak, N., & Temurtas, F. “A comparative study on

diabetes disease diagnosis using neural networks”. Expert Systems with Applications Volume 36 Elsevier,. 2009. pp.8610–8615.

[11] Durairaj, M., & Deepika, R. "Comparative Analysis of Classification Algorithms for the Prediction of Leukemia Cancer". International Journal of Advanced Research in Computer Science and Software Engineering; Volume 5, Issue 8. 2015. pp.787-791.

[12] Shaikh, A., Mahoto, N., Khuhawar, F., & Memon, M. "Performance Evaluation of Classification Methods for Heart Disease Dataset". Sindh Univ. Res. Jour. (Sci. Ser.) Vol. 47(3), 2015. pp. 389-394.

[13] Mittal, P., & Gill, N. S. "A Comparative Analysis of Classification Techniques on Medical Data Sets". IJRET: International Journal of Research in Engineering and Technology, Volume: 03 Issue: 06. 2014. 454-460.

[14] Karim, M., & Rahman, R. M. "Decision Tree and Naïve Bayes Algorithm for Classification and Generation of Actionable Knowledge for Direct Marketing". Journal of Software Engineering and Applications Volume. 6. 2013, pp.196-206.

[15] Kamber, M., & Han, J. "Data Mining: Concepts and Techniques Second Edition". San Francisco: Morgan Kaufmann Publishers. 2006.

[16] Gorunescu, F. "Data Mining Concept Model and Techniques". Berlin: Springer.2011.

[17] Babu, P., Ramana, B. V., & Kumar, B. R. "Automatic Diagnosis of Liver Status using Bayesian Classification". IEEE International Conference on Intelligent Networking and Computing. Kuala Lumpur: IEEE. 2010. [18] Domingos, P., & Pazzani, M. "On the optimality of the simple Bayesian

classifier under zero-one loss". Machine Learning 29. 1997. pp.103-130. [19] Larose, D., & Larose, C. "Discovering Knowledge in Data: An

Introduction to Data Mining". New York: John Wiley’s and Son.2014. [20] Willmott, C. J., & Matsuura, K. "Advantages of the mean absolute error

(MAE) over the root mean square error (RMSE) in assessing average model performance". Clim Res: Vol 30. 2005. pp.79-82.

[21] Strack, B., DeShazo, J. P., Gennings, C., Olmo, J. L., & Ventura, S. "Impact of HbA1c Measurement on Hospital Readmission Rates: Analysis of 70,000 Clinical Database Patient Records". Hindawi. 2014.