PERFORMANCE EVALUATION OF KNOWLEDGE BASED COLLABORATIVE FILTERING MODEL
P.Prabhu
1, N.Anbazhagan
21Assistant Professor, Research Scholar, Directorate of Distance Education,
2Professor & Head, Department of Mathematics, Alagappa University, Karaikudi, (India)
ABSTRACT
Collaborative filtering Recommender systems apply data mining techniques to produce personalized recommender system during the online interaction of active users. These systems use variety of techniques for achieving high success on business, banking, finance and other domains. The fast increase in users and products in recent years produces some of the key issues and challenges for recommender systems. These include scalability, sparsity and producing high quality of recommendations. Hence there is a necessity to produce new recommender systems to provide high accurate and fast recommenders systems to handle large mass of data. The issues and challenges are improved by introducing knowledge based proximity model using clustering technique. The performance of the model is experimentally evaluated with different statistical and decision making metrics using real-world datasets.
Keywords: Business Intelligence, Collaborative Filtering, Clustering, Data Mining, E-Commerce, Recommender System.
I. INTRODUCTION
The business, banking, finance and other applications offers millions of user and items. Finding intelligence from these large mass of data is challenging to the users and business. Recommender systems have produced number of solutions to this problem. A recommender system for the on-line business produces the suggestions to the on-line users that are fits to their needs. These systems have been implemented on various on-line business web sites for serving plenty of users. Collaborative Filtering (CF) recommender systems are one of the earliest and most successful recommender systems. Collaborative filtering systems recommend items based on the user‟s navigation patterns of items and the opinions of other similar minded users. A new active user is matched with existing user‟s navigation patterns that who have similar interest and tastes. Items that are liked by the similar users are having the probability to like the same items. Hence those similar users group history of items are produced as recommendations. The focus of this paper is to propose and experimentally evaluate the performance of the new knowledge based collaborative filtering model.
II. RELATED WORKS
This section briefly presents some of the research literature related to collaborative filtering recommender systems. Many algorithms have been proposed to generate recommendations.
In the early 1990s, collaborative filtering began to arise as a solution for dealing with large volume of online information. The first recommender system Tapestry, was designed to recommend documents from newsgroups manually [1]. The authors also introduced the term collaborative filtering as they used social collaboration to help users with large volume of documents.
Sarwar et al (2000) [2], used Singular Value Decomposition (SVD), to capture latent relationships between users and items that allow computing the predicted likeliness of a certain item by a user. Also it is used to produce a low-dimensional representation of the original user-item rating matrix and then compute neighbourhood in the reduced matrix.
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl ,(2001)[3], in their study, proposed the item- based approach looks into the set of items the target user has rated and computes how similar they are to the target item „i‟ and then selects „k‟ most similar items ). At the same time their corresponding similarities are also computed. Once the most similar items are found, the prediction is then computed by taking a weighted average of the target user's ratings on these similar items.
Przemyslaw Kazienko and Pawel Kolodziejski (2006)[4], the hybrid system of personalized product recommendation in e-commerce, by integrating various methods, was presented in the paper. Each e-commerce user has assigned their own weights corresponding to particular method. Due to the permanent and personalized adaptation of these weights, the system can adjust the influence of individual methods separately for each user.
Testing the implementation and evaluation of recommendation efficiency were also described.
P.Bedi,R.Sharma,H.Kaur (2009)[5], in their study uses collaborative filtering approach and proposed an Ant Recommender System (ARS) based on collaborative behavior of ants for generating Top-N recommendations.
The performance of ARS is evaluated using Jester dataset available on the website of University of California, Berkeley and compared with traditional collaborative filtering based recommender system.
C. Hwang (2010)[6], in their study, presents an implementation of GA for optimal feature weighting in the multi-criteria scenario. Their application of GA consists in selecting features that represent users‟ interest in a collaborative filtering context, in contrast to our method, which focuses on assigning weights to different recommendation algorithms in order to improve the overall performance in terms of accuracy, novelty and diversity.
III. SYSTEM ARCHITECTURE
This section introduces knowledge based model for collaborative filtering recommendation systems. The knowledge based model architecture is shown in Fig 1.
Figure 1 Knowledge Based Collaborative Filtering Model Architecture.
3.1 Methodology
This proposed algorithm is divided into sub phases. The First phase is pre-processing works in offline. In this phase data is pre-processed using various pre-processing techniques such as feature selection, normalization and data reduction.. In the Second phase modeling is constructed in offline. In this phase, user‟s knowledge about the data such as profile and preferences are defined and applied in terms of rules on utility matrix. Clustering is applied to identify the user navigation patterns or to cluster similar users. Once the user navigation patterns are identified using clustering the active users matching cluster is predicted using similarity matching to generate recommendations. Final phase is obtaining business intelligence i.e. recommendation generation process for the active user is performed online process. The list of recommendations for the active user is generated by finding the list of items from the selected cluster which are not purchased by the active user.
3.1.1 Pre-processing
The data required by the process may have incorrect, irrelevant, incomplete or missing data. Erroneous data leads to the loss of accuracy. It must be corrected or removed and missing data must be predicted. Also to overcome the scalability problem by selection of relevant features/attributes/columns the pre-processing is required. Feature selection and dimension reduction techniques are used to reduce the dataset from high to low.
3.1.2 Model Construction
This model mainly based on the knowledge information about users profile and preferences. This knowledge information about the user is applied in the form of rules. Users or items in the dataset are filtered based on knowledge defined in terms of rules to identify the navigation pattern. These defined rules are applied on pre- processed dataset to obtain filtered relevant dataset. The users profile information such as age, gender, occupation, zip code is used to form knowledge based rules. The user preferences such as genre information like Movie title , release date , video release date , IMDb URL , unknown , Action ,| Adventure , Animation , Children's , Comedy , Crime ,Documentary , Drama , Fantasy , Film-Noir , Horror , Musical , Mystery,
Romance, Sci-Fi ,Thriller , War and Western are also used define rules. This forms a utility matrix in terms of utility matrix (user-item) matrix.
3.1.2.1 Clustering of Utility Matrix
The important step in recommender system is to find the similarity between customers. It is used form proximity based similarity between active users and like minded existing users. The data set in the form of utility matrix is partitioned using clustering algorithm to identify similar users and their navigation patterns. This clustering process is the important model construction in recommender systems. By partitioning the item space into numerous smaller clusters each individual prediction computation will take less time to complete [7]. Since each partition is independent of others, prediction computation on each partition could be done in parallel, further increasing the rate at which predictions can be computed. The most widely used convergence criteria for the k- means algorithm is minimizing the SSE. The k-means algorithm always converges to a local minimum. K- means clustering is very simple and easy to implement. Hence in this research k-means clustering is used to cluster the users in terms of user item matrix to find their navigation patterns.
3.1.2.2 Prediction of Best Matching Cluster (BMC)
Best matching cluster of user‟s navigation patterns of the active user can be predicted by finding the proximity between the active user and cluster centres. The similarity of the active user i.e. matching-score is calculated with other clusters. The matching-score which have minimum value is identified as matching cluster. The proximity between two users is usually measured using distance measures like Euclidean distance, Jaccard, Pearson correlation and cosine. The Jaccard distance only takes into account the number of rated items but not the actual rating which is discarded, therefore, it loses accuracy in case that detailed recommendations were needed.
3.1.3 Intelligence
The final phase in recommender system is to generate recommendations based on the matching cluster of active user. Recommendation is the list of N items that the active user is interested. This list usually consists of list of items that are not already purchased by the active users. In this last step, the N recommendations are generated and listed. The following types of techniques are adapted for generating recommendations.
Most frequent items recommendation
o In this method, it looks into the best matching cluster users items and finds the frequency count of items. Then the model sort the items according to the frequency count and calculate the top-N most frequent items as recommendation list that have not been yet purchased by the active user.
Recommend a few of the highest rated items
o In this method, the best matching cluster users few of the highest rated items that have not been yet purchased by the active user is selected as recommendation.
In this paper, according to the frequency count of items from the identified cluster users, this returns N items as recommendation that have not yet been purchased by the active user. Also high rated items method is also used and average of these two methods is calculated.
3.2 Algorithm KBM
Input:Train Dataset ‟D‟; Test Dataset „TD‟;
The number of clusters „k‟; The number of recommendations „N‟.
Output: Recommendation List of „N’ items.
Begin
//Phase I: Pre-Processing Feature Selection;
Normalization;
Dimension Reduction using PCA;
/ / Phase II: Model Construction
// Filtering of attributes by formulating knowledge based rules
// Filtering of samples by formulating knowledge based rules 1 to nsr Do k-Means Clustering
// Find the Best Matching Cluster (BMC) index ci using Proximity For each Active user do
Calculate matching score using proximity measure Select as Best Match Cluster (BMC) index End
TMAE ← evaluate Matching Cluster
//Phase III: Intelligence - Generate recommendation list For each Active user do
Identify items of best matching cluster (BMC) of users Select „N‟ items
Present the‟ N’ items to the active user as the recommendation list End
End
IV. EXPERIMENTAL STUDY
The knowledge based model (KBM) has been evaluated using three measures precision, recall, and F-measure.
While precision measure the degree to which the recommendation model produces accurate recommendations (the ratio of number of products recommended correctly to number of products in recommendation set), recall measure the quality of the recommendation. As precision and recall are inversely related, a combination measure called the F-Measure is used to give equal weight to both precision and recall. The F-Measure value becomes maximum when both precision and recall are maximized. This algorithm has been experimentally simulated and evaluated with real-world datasets with different active users profile and preferences. MovieLens
100k data set collected by the GroupLens Research Project is used to test the performance of this proposed algorithm [8] and Jester dataset [9].
4.1 Clustering Quality Analysis
The are various methods for finding the optimal number of clusters k such as rule of thumb, the elbow method, information criterion approach, an Information Theoretic approach, choosing k using the Silhouette, Cross- validation, analyzing the kernel matrix, external links and bibliography.The optimum number of clusters can be experimentally evaluated. The experiment is repeated with different number of clusters and calculated using Silhouette measure. The Fig 2 shows Silhouette Index value obtained for various number of clusters k=2 to 25.
Figure 2 Sample Silhouette Index (SI) on MovieLens dataset
The pictorial representation of the above chart clearly shows that the optimum number of clusters k obtained using the knee value of silhouette value is 5.
4.2 Dimension Sensitivity Analysis
Unfortunately, there is no direct analytical method to determine the optimal number of dimension [10] so the optimal value has to be experimentally evaluated. The experiment is repeated with different number of dimensions and calculated the precision, recall and F1 measure. The above mentioned algorithm is tested by taking range of dimensions in dataset from 25 to 300 in the step of 25. Fig 3 shows the plot for Recall, Precision and F1 Measure values calculated for range of dimensions using MovieLens data set.
Figure 3 Recall, Precision and F1 Measure on MovieLens Dataset
The pictorial representation of the above chart clearly shows the recall, precision and F1-Measure knee value is obtained in dimension 200. Hence the optimum number of dimension of the selected dataset is 200.
4.3 Prediction Accuracy Analysis
The Mean Absolute Error (MAE) between the predicted ratings and the actual ratings of users within the test set is calculated. Several experiments were conducted using our proposed method for performance benchmark. The Total MAE can be calculated by averaging the Mean Absolute Errors of all active users. TABLE 1 shows prediction accuracy of various similarity metrics experimented using this proposed method.
Table 1 Prediction Accuracy TMAE of all Active users
Dataset Proximity
Euclidean Correlation Cosine
MovieLens 0.1214 0.147 0.1523
Jester 0.1788 0.19 0.1929
Fig 4 shows the plot for TMAE values of all active users calculated using various proximity measures on MovieLens and Jester data set.
Figure 4 TMAE Values Using Proximity Measures
The above pictorial representation shows that the TMAE of Euclidean is less than the other two similarity measure on MovieLens and Jester datasets. Since the number of test active users in MovieLens dataset are less than that of Jester dataset the accuracy is increased in MovieLens dataset.
4.4 Recommendation Quality Analysis
The above mentioned algorithm is tested by taking range of recommendations in dataset from 5 to 50 in the step of 5. Fig 5 shows recall values of the sensitivity of N recommendations quality calculated using this proposed method. The x-axis show the number of recommendations and y-axis shows the precision values calculated using proposed method.
Figure 5 Recall of top-N Recommendations on MovieLens dataset.
The above pictorial representation clearly shows that the maximum mean recall value is obtained in MovieLens dataset-5 dataset for top-N=10.Fig.6 shows precision values of the sensitivity of N recommendations quality calculated using this proposed method.
Figure 6 Precision of top-N Recommendations on Movie Lens Dataset.
The above pictorial representation clearly shows that the maximum mean Precision value is obtained in MovieLens dataset-1 dataset for top-N=10.Fig.7 shows F1 Measure of various number N of recommendations calculated using this proposed method.
Figure 7 F1 Measure of top-N recommendations on MovieLens Dataset.
The above pictorial representation clearly shows that the maximum mean F1 Measure value is obtained in MovieLens dataset-1 for top-N=10.Fig.8 shows recall values of the sensitivity of N recommendations quality calculated using this proposed method.
Figure 8 Recall Values of top-N Recommendations on Jester Dataset
The above pictorial representation clearly shows that the maximum mean recall Measure value is obtained in Jester dataset-3 for top-N=10.Fig.9 shows precision values of the sensitivity of N recommendations quality calculated using this proposed method.
Figure 9 Precision of Top-N Recommendations on Jester Dataset
The above pictorial representation clearly shows that the maximum mean Precision value is obtained on Jester dataset-2 for top-N=10.Fig.10 shows F1-Measure of various number N of recommendations calculated using this proposed method.
Figure 10 F1 Measure of Top-N recommendations on Jester Dataset.
The above pictorial representation clearly shows that the maximum mean F1 Measure value is obtained on Jester dataset-3 for top-N=10.
4.5 Evaluation with Conventional Systems
This section discusses about the comparison of proposed model with conventional k-Nearest Neighbour (k- NNBM) system .The TABLE 2 shows the comparison of different evaluation measures recall, precision and F1 values of k-NNBM and KBM models using MovieLens dataset for top-N=10.
Table 2 Recall, Precision, and F1 Measures of k-NNBM and KBM on MovieLens dataset
Algorithm
Evaluation Measure Recall Precision F1Measure
k-NNBM 0.30 0.65 0.44
KBM 0.64 0.78 0.66
The table values clearly show that the accuracy of KBM model is better than k-NNBM. We obtained more than 34% increase in recall and 13% in precision and 22% in F1-measure when compared with widely-used CF algorithms on MovieLens dataset. The overall performance of KBM is better than ARS method. The Recall, Precision and F1 Measure of k-NNBM and KBM are compared in Fig 11. The Recall, Precision and F1- Measure are shown along the x-axis and value of these measures is shown in the y-axis.
Figure 11 Recall, Precision, F1 Measures of NNBM and KBM on MovieLens Dataset
The pictorial representation of the above chart clearly shows that the accuracy of KBM model is better than k- NNBM. Therefore the performance of the KBNNM is more significant than the existing k-NNBM method since it gives more accuracy in terms of recall, precision and f1 measure. Thus the performance of proposed algorithm was tested with real-world datasets. The performance of the algorithm was found significant and there is a lot of scope for further improvements.
This proposed model is compared with conventional recommender system based on collaborative behaviour of Ants (ARS) on Jester dataset [5]. The TABLE 3 shows the comparison of different evaluation measures recall, precision and F1 values of ARS and KBM models using Jester dataset for top-N=10.
Table 3 Recall, Precision, F1 Measures of ARS and KBM on Jester dataset
Algorithm Evaluation MeasureRecall Precision F1 Measure
ARS 0.38 0.67 0.49
KBM 0.66 0.96 0.79
It is observed from the above table that more than 28% increase in recall and 29% in precision and 30% in F1- measure when compared with widely-used CF algorithms using Jester dataset. The overall performance of KBM is better than ARS method. The Recall, Precision and F1 Measure of ARS and KBM approaches compared in Fig 12. The Recall, Precision and F1-Measure are shown along the x-axis and value of these measures is shown in the y-axis.
Figure 12 Recall, Precision, F1 Measures of ARS and KBM using Jester dataset
The pictorial representation of the above chart clearly shows that the accuracy of KBM model is better than ARS. Therefore the performance of the KB is more significant than the existing ARS method since it gives more accuracy in terms of recall, precision and F1 measure. Thus the performance of proposed algorithm was tested with real-world datasets. The performance of the algorithm was found significant and there is a lot of scope for further improvements.
V. CONCLUSION
In this paper we presented and experimentally evaluated the performance of the new knowledge based collaborative filtering model using real-world datasets with different statistical and decision making metrics.
Our experiment suggests that knowledge based CF model provides improved prediction quality and accuracy.
The capability of handling sparse and scalable dataset is also improved. This model can also be applied to other applications like Crime detection, Bank Loan approval and finance. By introducing soft computing and optimization techniques there is a possibility of further improvement in quality.
REFERENCES
[1]. D. Goldberg, D. Nichols, B. M. Oki, and D. Terry, “Using collaborative filtering to weave an information tapestry,” Communications of the ACM, vol. 35, no. 12, pp. 61–70, 1992.
[2]. Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl, Item-Based Collaborative Filtering Recommendation Algorithms,WWW10, May 1-5, 2001, Hong Kong.ACM 1-58113-348-0/01/0005.
[3]. Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl, Analysis of Recommendation Algorithms for ECommerce,EC‟00, October 1720, 2000, Minneapolis, Minnesota. Copyright 2000 ACM 1581132727/00/0010.
[4]. Przemyslaw Kazienko and Pawel Kolodziejski, Personalized Integration of Recommendation Methods for E-commerce, International Journal of Computer Science & Applications ã 2006 Techno mathematics Research Foundation Vol. 3 Issue 3, pp 12-26.
[5]. P.Bedi,R.Sharma,H.Kaur, Recommender system Based on collaborative behaviour of Ants, Journal of Artificial Intelligence 2 (2):40-55,2009,ISSN 1994-5450,Asian Network for Scientific Information.
[6]. C. Hwang. Genetic algorithms for feature weighting in multi-criteria recommender systems. JCIT, 5(8), 2010.
[7]. Mark O'Connor, and Jon Herlocker. In Proceedings of the ACM SIGIR Workshop on Recommender Systems: Algorithms and Evaluation, 22nd International Conference on Research and Development in Information Retrieval (August 1999).
[8]. www.movielens.umn.edu
[9]. http://goldberg.berkeley.edu/jester-data/
[10]. Deerwester,S., Dumais,S.T., Furnas,G.W., Landauer,T.K., and Harshman,R. (1990). Indexing by Latent Semantic Analysis. Journal of the American Society for Information Science.41 (6), pp.391-407.
