IMPROVING MULTI-VIEW POINT CLUSTERING ON BIG DATA
M.kamarunisha
1, K.Praveena
2, R.Ragavi
31Associatet Professor, Dept of Computer Applications,
Dhanalakshmi Srinivasan College of Arts and Science for Women, Perambalur
2,3PG Scholar, Dept of Computer Applications,
Dhanalakshmi Srinivasan College of Arts and Science for Women, Perambalur
ABSTRACT
The entire data object is related to some another data objects in large set of database. It measure similarity between the objects through the clusters methods. The novel multi-viewpoint is used to define the similarity of two related clustering methods. Multi-view point is very useful to achieve the related object is grouped in cluster. Thus in all the documents their meta-tags are identified, starting from the base document. The features of the Big Data revolution, and proposes a Big Data processing model, from the data mining perspective. This data-driven model involves demand-driven aggregation of information sources, mining and analysis, user interest modeling, and security and privacy considerations. It solves the challenging issues in the data-driven model and also in the Big Data revolution.
Keywords: Introduction, Data Mining,Types of Clustering, Concept of Clustering, Partitional Clustering, Advantages, Disadvantages.
I. INTRODUCTION
Data mining (sometimes called data or knowledge discovery) is the process of analyzing data from different perspectives and summarizing it into useful information - information that can be used to increase revenue, cuts costs, or both. Data mining software is one of a number of analytical tools for analyzing data. It allows users to analyze data from many different dimensions or angles, categorize it, and summarize the relationships identified.
Technically, data mining is the process of finding correlations or patterns among dozens of fields in large relational databases. While large-scale information technology has been evolving separate transaction and analytical systems, data mining provides the link between the two. Data mining software analyzes relationships and patterns in stored transaction data based on open-ended user queries. Several types of analytical software are available: statistical, machine learning, and neural networks.
Big Data concern large-volume, complex, growing data sets with multiple, autonomous sources. With the fast development of networking, data storage, and the data collection capacity, Big Data are now rapidly expanding in all science and engineering domains, including physical, biological and biomedical sciences.
II.CLUSTERING
Clustering is a division of data into groups of similar objects.
Representing the data by fewer clusters necessarily loses certain fine details, but achieves simplification.
2.1 Types of Clustering
Data clustering algorithms can be hierarchical. Hierarchical algorithms find successive clusters using previously established clusters. Hierarchical algorithms can be agglomerative ("bottom-up") or divisive ("top-down").
Agglomerative algorithms begin with each element as a separate cluster and merge them into successively larger clusters. Divisive algorithms begin with the whole set and proceed to divide it into successively smaller clusters.
Partition algorithms typically determine all clusters at once, but can also be used as divisive algorithms in the hierarchical clustering. Two-way clustering, co-clustering or bi clustering are clustering methods where not only the objects are clustered but also the features of the objects, i.e., if the data is represented in a data matrix, the rows and columns are clustered simultaneously.
2.2.1 Hierarchical Clustering
Hierarchical clustering builds (agglomerative), or breaks up (divisive), a hierarchy of clusters. The traditional representation of this hierarchy is a tree (called a dendrogram), with individual elements at one end and a single cluster containing every element at the other. Agglomerative algorithms begin at the leaves of the tree, whereas divisive algorithms begin at the root.
Cutting the tree at a given height will give a clustering at a selected precision. In the following example, cutting after the second row will yield clusters {a} {b c} {d e} {f}. Cutting after the third row will yield clusters {a} {b c} {d e f}, which is a coarser clustering, with a smaller number of larger clusters.
2.2.2 Agglomerative Hierarchical Clustering
For example, suppose this data is to be clustered, and the euclidean distance is the distance metric.
Fig 2.1 Raw data The hierarchical clustering Dendogram would be as such:
Fig 2.2 Traditional Representation
This method builds the hierarchy from the individual elements by progressively merging clusters. In the example, six elements {a} {b} {c} {d} {e} and {f} are represented. The first step is to determine which elements to merge in a cluster. Usually, the major focus is to take the two closest elements, according to the chosen distance.
Optionally, one can also construct a distance matrix at this stage, where the number in the i-th row j-th column is the distance between the i-th and j-th elements. Then, as clustering progresses, rows and columns are merged as the clusters are merged and the distances updated. This is a common way to implement this type of clustering, and has the benefit of caching distances between clusters. A simple agglomerative clustering algorithm is described in the single-linkage clustering page; it can easily be adapted to different types of linkage.
Usually the distance between two clusters and is one of the following:
The maximum distance between elements of each cluster (also called complete linkage clustering):
The minimum distance between elements of each cluster (also called single-linkage clustering):
The mean distance between elements of each cluster (also called average linkage clustering, used e.g. in UPGMA):
The sum of all intra-cluster variance
The increase in variance for the cluster being merged (Ward's criterion)
The probability that candidate clusters spawn from the same distribution function (V-linkage)
Each agglomeration occurs at a greater distance between clusters than the previous agglomeration, and one can decide to stop clustering either when the clusters are too far apart to be merged (distance criterion) or when there is a sufficiently small number of clusters (number criterion).
III.CONCEPT CLUSTERING
Another variation of the agglomerative clustering approach is conceptual clustering.
IV. PARTITIONAL CLUSTERING 4.1 K-Means Clustering
The K-means algorithm assigns each point to the cluster whose center (also called centroid) is nearest. The center is the average of all the points in the cluster — that is, its coordinates are the arithmetic mean for each dimension separately over all the points in the cluster.
The main advantages of this algorithm are its simplicity and speed which allows it to run on large datasets. Its disadvantage is that it does not yield the same result with each run, since the resulting clusters depend on the initial random assignments. It minimizes intra-cluster variance, but does not ensure that the result has a global minimum of variance.
4.2 Fuzzy C-Means Clustering
In fuzzy clustering, each point has a degree of belonging to clusters, as in fuzzy logic, rather than belonging completely to just one cluster. Thus, points on the edge of a cluster, may be in the cluster to a lesser degree than points in the center of cluster. For each point x we have a coefficient giving the degree of being in the kth cluster uk(x). Usually, the sum of those coefficients is defined to be 1:
With fuzzy c-means, the centroid of a cluster is the mean of all points, weighted by their degree of belonging to the cluster:
The degree of belonging is related to the inverse of the distance to the cluster center:
then the coefficients are normalized and fuzzyfied with a real parameter m> 1 so that their sum is 1. So
For m equal to 2, this is equivalent to normalizing the coefficient linearly to make their sum 1. When m is close to 1, then cluster center closest to the point is given much more weight than the others, and the algorithm is similar to k-means.
The fuzzy c-means algorithm is very similar to the k-means algorithm:
Choose a number of clusters.
Assign randomly to each point coefficients for being in the clusters.
Repeat until the algorithm has converged (that is, the coefficients' change between two iterations is no more than ε, the given sensitivity threshold) :
o Compute the centroid for each cluster, using the formula above.
o For each point, compute its coefficients of being in the clusters, using the formula.
The algorithm minimizes intra-cluster variance as well, but has the same problems as k-means, the minimum is a local minimum, and the results depend on the initial choice of weights. The Expectation-maximization algorithm is a more statistically formalized method which includes some of these ideas: partial membership in classes. It has better convergence properties and is in general preferred to fuzzy-c-means.
V.QT CLUSTERING ALGORITHM
QT (quality threshold) clustering is an alternative method of partitioning data, invented for gene clustering. It requires more computing power than k-means, but does not require specifying the number of clusters a priori, and always returns the same result when run several times.
The user chooses a maximum diameter for clusters.
Build a candidate cluster for each point by including the closest point, the next closest, and so on, until the diameter of the cluster surpasses the threshold.
Save the candidate cluster with the most points as the first true cluster, and remove all points in the cluster from further consideration. Must clarify what happens if more than 1 cluster has the maximum number of points?
Recurs with the reduced set of points
VI. .COMPARISONS BETWEEN DATA CLUSTERINGS
There have been several suggestions for a measure of similarity between two clustering’s. Such a measure can be used to compare how well different data clustering algorithms perform on a set of data. Many of these measures are derived from the matching matrix (aka confusion matrix), e.g., the Rand measure and the Fawkes- Mallows Bk measures.
Several different clustering systems based on mutual information have been proposed. One is Marina Meila's 'Variation of Information' metric and another provides hierarchical clustering.
VII.HIERARCHICAL DOCUMENT CLUSTERING USING FREQUENT ITEMSETS
Document clustering has been studied intensively because of its wide applicability in areas such as web mining, search engines, information retrieval, and topological analysis. Unlike in document classification, in document clustering no labeled documents are provided. Although standard clustering techniques such as k-means can be applied to document clustering, they usually do not satisfy the special requirements for clustering documents:
high dimensionality, high volume of data, ease for browsing, and meaningful cluster labels. In addition, many existing document clustering algorithms require the user to specify the number of clusters as an input parameter and are not robust enough to handle different types of document sets in a real-world environment. For example, in some document sets the cluster size varies from few to thousands of documents. This variation tremendously reduces the clustering accuracy for some of the state-of-the art algorithms.
Frequent Itemset-based Hierarchical Clustering (FIHC), for document clustering based on the idea of frequent itemsetsproposed by Agrawal et. al. The intuition of our clustering criterion is that there are some frequent itemsets for each cluster (topic) in the document set, and different clusters share few frequent itemsets. A frequent itemset is a set of words that occur together in some minimum fraction of documents in a cluster. Therefore, a frequent itemset describes something common to many documents in a cluster. In this technique use frequent itemsets to construct clusters and to organize clusters into a topic hierarchy.
Here are the features of this approach.
Reduced dimensionality. This approach use only the frequent items that occur in some minimum fraction of documents in document vectors, which drastically reduces the dimensionality of the document set.
Experiments show that clustering with reduced dimensionality is significantly more efficient and scalable.
This decision is consistent with the study from linguistics (Longman Lancaster Corpus) that only 3000 words are required to cover 80% of the written text in English and the result is coherent with the Zipf’s law and the findings in Mladenic et al. and Yang et al.
High clustering accuracy. Experimental results show that the proposed approach FIHC outperforms best documents clustering algorithms in terms of accuracy. It is robust even when applied to large and complicated document sets.
Number of clusters as an optional input parameter. Many existing clustering algorithms require the user to specify the desired number of clusters as an input parameter. FIHC treats it only as an optional input parameter. Close to optimal clustering quality can be achieved even when this value is unknown.
VIII.ADVANTAGES
Provide most relevant and most accurate social sensing feedback to better understand our society at real time
Clustering and Searching mechanisms are more efficient than the existing system
Improve the efficiency of clustering
Increase in the speed and performance
Save computing time
Metadata filtering allows a user to modify the set of properties attributes and events exposed by a component
IX. DISADVANTAGES
HACE make it an extreme challenge for discovering useful knowledge from the Big Data.
For big data mining high performance algorithms are required
Because Big Data are often stored at different locations
Semantics and domain knowledge adds technical barriers to the big data access
The challenges at Tier I focus on data accessing and arithmetic computing procedures. Because Big Data are often stored at different locations and data volumes may continuously grow, an effective computing platform will have to take distributed large-scale data storage into consideration for computing.
The challenges at Tier II center around semantics and domain knowledge for different Big Data applications. Such information can provide additional benefits to the mining process, as well as add technical barriers to the Big Data access (Tier I) and mining algorithms (Tier III).
At Tier III, the data mining challenges concentrate on algorithm designs in tackling the difficulties raised by the Big Data volumes, distributed data distributions, and by complex and dynamic data characteristics.
X. CONCLUSION
A cloud storage system consists of storage servers and key servers. The threshold proxy re encryption scheme supports encoding, forwarding, and partial decryption operations in a distributed way. To decrypt a message of k blocks that are encrypted and encoded to n code word symbols, each key server only has to partially decrypt two codeword symbols in our system. By using the threshold proxy reencryption scheme, a secure cloud storage system that provides secure data storage and secure data forwarding functionality in a decentralized structure.
XI. FUTURE ENHANCEMENT
Provide robustness against server failures, a simple method is to make replicas of each message and store them in different servers. However, this method is expensive as z replicas result in z times of expansion.
One way to reduce the expansion rate is to use erasure codes to encode messages. A message is encoded as a code word, which is a vector of symbols, and each storage server stores a code word symbol. A storage server failure is modeled as an erasure error of the stored code word symbol. Random linear codes support distributed encoding, that is, each code word symbol is independently computed. To store a message of k blocks, each storage serve linearly combines the blocks with randomly chosen coefficients and stores the codeword symbol and coefficients.
REFERENCES
[1]. J. Shao and Z. Cao, “Providing Database as a Service” Proc. 12th Int’l Conf. Practice and Theory in Public Key Cryptography (PKC), pp. 357-376, 2009.
[2]. G. Ateniese, R. Burns, R. Curtmola, J. Herring, L. Kissner, Z. Peterson, and D. Song, “Order-Preserving Encryption Revisited: Improved Security Analysis and Alternative Solutions,” Proc. 14th ACM Conf.
Computer and Comm. Security (CCS), pp. 598-609, 2007.
[3]. G. Ateniese, R.D. Pietro, L.V. Mancini, and G. Tsudik, “Security and Confidentiality Solutions for Public Cloud Database Services” Proc. Fourth Int’l Conf. Security and Privacy in Comm. Netowrks (SecureComm), pp. 1-10, 2008.
[4]. A.G. Dimakis, V. Prabhakaran, and K. Ramchandran, “Ubiquitous Access to Distributed Data in Large- Scale Sensor Networks through Decentralized Erasure Codes”, 2005.
[5]. A.G. Dimakis, V. Prabhakaran, and K. Ramchandran, “Decentralized Erasure Codes for Distributed Networked Storage”, June 2006.
[6]. M. Mambo and E. Okamoto, “Proxy Cryptosystems: Delegation of The Power to Decrypt Ciphertexts”,1997.
[7]. M. Blaze, G. Bleumer, and M. Strauss, “Divertible Protocols and Atomic Proxy Cryptography”, 1998.
[8]. G. Ateniese, K. Fu, M. Green, and S. Hohenberger, “Improved Encryption Schemes with Applications to Secure Distributed Storage”, 2006.
Mrs. Kamarunisha M - Received M.C.A., M.Phil., Degree in Computer Science.
She has 14 years of Teaching Experience. She is guided 4 students in M.Phil. She had presented 6 papers in International Conference and also she presented 7 papers in National Conference. She published one paper in IJSTM International journal. She is currently working as Associate Professor in Department ofComputer Applications in Dhanalakshmi Srinivasan College of Arts and Science for Women, Perambalur, Tamil Nadu, India.
Ms.K.Praveena., PG scholar, [Department of CA] ,Pursuing M.C.A Computer Applications in DhanalakshmiSrinivasan College of Arts & science for women, Perambalur-621212, TamilNadu, (India)
Ms.K.Praveena., PG scholar, [Department of CA] ,Pursuing M.C.A Computer Applications in DhanalakshmiSrinivasan College of Arts & science for women, Perambalur-621212, TamilNadu, (India)
