Fuzzy Supervised Clustering Algorithm with the Particle Swarm Optimization

2018 International Conference on Computer, Communication and Network Technology (CCNT 2018) ISBN: 978-1-60595-561-2

Fuzzy Supervised Clustering Algorithm with the Particle

Swarm Optimization

Yuan-horng LIN

, Jeng-ming YIH

and Shin-hua WU

1

No. 140, Minsheng Rd., West Dist., Taichung City 403, National Taichung University of Education, Taiwan

2

No.1116, Sec. 2, Zhongshan E. Rd., Liuying Dist., Tainan City 736, Min-Hwei Junior College of Health Care Management, Taiwan

3

No. 300, Sec.2, Kungyi Rd., Nantun Dist., Taichung City 408, Hui-Wei Elementary School, Taiwan

*

Corresponding author

Keywords: Particle swarm optimization, Fuzzy clustering algorithm, Mahalanobis distance, Fuzzy c-means algorithm.

Abstract. In GK-algorithm, fuzzy clustering algorithm with preserved volume was used. However,

the added fuzzy covariance matrices in their distance measure were not directly derived from the objective function. A Fuzzy C-Means algorithm based on Mahalanobis distance (FCM-HM) was proposed to improve those limitations of GG and GK algorithms, but it is not stable enough when some of its covariance matrices are not equal.The singular problem and the selecting initial values problem are improved. We pointed out that the initial memberships of fuzzy c-mean algorithm which was based on Mahalanobis distance algorithm and the traditional Fuzzy c-means algorithm (FCM) Algorithm can’t be all equal. The other important issue is how to approach the global minimum value that can improve the cluster accuracy. The methods to detect the local extreme value were developed by this paper. Focusing attention to these two faults, an improved new algorithm, “Fuzzy C-Means based on Particle Swarm Optimization with Mahalanobis distance (PSO-FCM-HM)”, is proposed. We have two aims and goals of our research summary. One is to compare the classification accuracies of fuzzy clustering algorithms based on Mahalanobis distances and Euclidean distances. The other is to choose the initial membership to promote the classification accuracies.

Introduction

To overcome the drawback due to Euclidean distance, we could try to extend the distance measure to Mahalanobis distance (MD). However, Krishnapuram and Kim (1999)pointed out that the Mahalanobis distance can not be used directly in clustering algorithm. Gustafson-Kessel (GK) clustering algorithm and Gath-Geva (GG) clustering algorithm were developed to detect non-spherical structural clusters. In GK-algorithm, a modified Mahalanobis distance with preserved volume was used. However, the added fuzzy covariance matrices in their distance measure were not directly derived from the objective function. In GG algorithm, the Gaussian distance can only be used for the data with multivariate normal distribution.

Extending Euclidean distance to Mahalanobis distance, the well known fuzzy partition clustering algorithms, Gustafson-Kessel (GK) clustering algorithm and Gath-Geva (GG) clustering algorithm [3] were developed to detect non- spherical structural clusters, but these two algorithms fail to consider the relationships between cluster centers in the objective function, GK algorithm must have prior information of shape volume in each data class. Mahalanobis, an Indian statistician, introduced this distance in the 1930’s. The Mahalanobis distance is a distance using the inverse of the covariance matrix as the metric. It is a distance in the geometrical sense because the covariance matrices as well as its inverse are positive definite matrices [6]. The methods to compute the better initial value were developed by this paper. Focusing attention to these two faults, an improved new algorithm, “Fuzzy C-Mean based on Mahalanobis distance (FCM-M)”, is proposed [7]. The popular fuzzy c-means algorithm (FCM) is developed by using Picard Iteration through the first-order conditions for stationary points of the objective function. It converges to a local minimum of the objective function. Hence, different initializations may lead to different results. The important issue is how to avoid getting a bad local minimum value to improve the cluster accuracy. The particle swarm optimization (PSO) is a popular and robust strategy for optimization problems. A new improved Fuzzy Clustering Algorithm with particle swarm optimization is proposed(PSO-FCM-HM) [8].

Literature Review

The clustering plays an important role in data analysis and interpretation. It groups the data into classes or clusters so that the data objects within a cluster have high similarity in comparison to one another, but are very dissimilar to those data objects in other clusters.

FCM-HM Algorithm

For improving the stability of the clustering results, we replace all of the covariance matrices with the same common covariance matrix in the objective function in the FCM-M algorithm, An improve fuzzy clustering method, called FCM-HM is proposed. We can obtain the objective function of FCM-HM as following:

(

)

(

)

1 1

, , , ,

c n

m m

FCM CM ij j i

i j

J U A X _µ d x a

−

= =

Σ =

∑ ∑

(1)

[ ]

[ ]

1 1

[1, ); ; 0,1 , 1, 2, ..., , 1, 2,...,

1, 1, 2, ..., , 0 , 1, 2, ...,

ij cn ij

c n

ij ij

i j

m U i c j n

j n n i c

µ µ

µ µ

×

= =

∈ ∞ = ∈ = =

= = < < =

∑

∑

(2)

(

)

(

) (

)

(

) (

)

(

) (

)

1 1 1 1

2

1 1

ln ln 0

,

0 ln 0

j i j i j i j i

j i

j i j i

x a x a if x a x a

d x a

if x a x a

− − − −

− −

′ ′

− Σ − − Σ − Σ − − Σ ≥ =

′

− Σ − − Σ <

  

 (3) Minimizing the objective function respect to all parameters in Equation (1) with the constraint (2), we can obtain the following FCM-HM algorithm [9].

New Clustering Algorithm

FCM-NM Algorithm Not only z-score normalizing for each feature in the objective function in the

FCM-CM algorithm, but also replacing the threshold D, where ( ) ( )

(

)

(

)

0 0 0

1 1

0

i i

c n m

ij j j

i j

D _µ x a x a

= =

′

With the determinant value of the crisp correlation matrix, the new fuzzy clustering method, called the Fuzzy C-Means algorithm based on Homogeneous Mahalanobis distance (FCM-HM) is proposed.

Particle Swarm Optimization (PSO) is a quite convenient method for optimizing hard numerical function on metaphor of social behavior of flocks of birds and schools of fish. A swarm consist M individuals, called particles, which change their position over time. Each particle represents a potential solution to the problem of optimization. In FCM-HM, The problem of optimization is to minimize the value of the objective function. Let the particle k in a D-dimension space (D=nc) be represented as

(1) Let the particle k in a D-dimension space (D=nc) be represented as

(

)

(

)

1 2

1 1 1 2 1 2 1

2 2 2 1 2

, , . . . ,

, , . . . , , ,

, . . . , , . . . , , , . . . ,

1 , 2 , . . . ,

k k k k D

k k k n k

k k n k c k c k c n

k M

µ µ µ µ

µ µ µ µ

µ µ µ µ µ

=

=

=

(4) (2) Let the objective function of FCM be the fitness function as follows,

( ) 2

1 1

2

1 1 , ,

c n

m m

F C M i j i j

i j c n

m

i j j i i j

J U A X d

x a

µ

µ

= =

= =

=

= −

∑ ∑

∑ ∑

(5)

Experiment of Real Data

The features of the Iris data contain Length of Sepal, Width of Sepal, Length of Petal, and Width of Petal. The samples were assigned the original 3 clusters based on the clustering analysis. The results were shown in Table 1.

Table 1. The characteristics of 3 clusters for Iris data.

Cluster Samples size Concepts Average distance of center

1 ₅₀ Setosa 0.48170523644

2 ₅₀ Versicolor 0.70687020404

3 50 Virginica 0.81933940766

Table 2. Classification Accuracies for Iris data.

Algorithm Accuracies

FCM 0.8933

GG 0.7649

GK 0.9000

FCM-M 0.9000

FCM-HM 0.9279

PSO-FCM-HM 0. 9467

Table 3. The characteristics of 4 clusters for Math Teaching data.

Cluster Samples size Concepts Average distance of center

1 115 division 1.2576760

2 128 ordering 1.2968550

3 168 multiplication 1.1244569

Table 4. Classification Accuracies for Math Teaching data.

Algorithm Accuracies

FCM 0.8333

GG 0.7949

GK 0.8538

FCM-M 0.8618

FCM-HM 0.8947

PSO-FCM-HM 0. 9278

The performances of three clustering methods, FCM, GG, GK, FCM-M, FCM-HM and PSO-FCM-HM, are compared in the experiments. The Iris Data [10] with sample size 150 is used as first example. The classification accuracies testing samples were shown in Table 2.

A real data set with sample size 493 from elementary schools was selected. The main factors of the data were calculated by using Factor Analysis. According to the main factors, the samples were assigned to 4 clusters based on the clustering analysis. The results were shown in Table 3-4.

Conclusions and Suggestions

An improved new fuzzy clustering algorithm is developed to obtain better quality of fuzzy clustering results. The objective function includes the regulating terms about the covariance matrices. The update equations for the memberships and the cluster centers and the covariance matrices are directly derived from the Lagrange’s method. The fuzzy c-mean algorithm is different from the GK and GG algorithms. The singular problem and the selecting initial values problem are improved by the Eigenvalue method and the Ratio method. We also proposed a proposition which pointed out that the initial memberships of FCM-HM Algorithm and FCM algorithm can’t be all equal [11]. Finally, a numerical example shows that the new fuzzy clustering algorithm gives more accurate clustering results than the FCM algorithm for a real data set the ratio method which is proposed by us is the best of the six methods for selecting the initial values. The objective function includes the regulating terms about the covariance matrices. The update equations for the memberships and the cluster centers and the covariance matrices are directly derived from the Lagrange’s method. The fuzzy c-means algorithm is different from the GK and GG algorithms. The singular problem and detecting the local extreme value problem are improved by the Eigenvalue method and the algorithm of Particle Swarm Optimization. Finally, by using clustering accuracy and the Rand index, respectively, two numerical examples showed that the new fuzzy clustering algorithm (PSO-FCM-HM) gave more accurate clustering results than that of our proposed FCM-HM algorithm and the traditional FCM algorithm.

Acknowledgement

This study was financially supported by the Ministry of Science and Technology, Taiwan under the Grant No. MOST 104-2511-S-142 -003 -MY3

References

[1]J. C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms, Plenum, N.Y. (1981).

[2]D. E. Gustafson and W. C. Kessel, Proc. IEEE Conf. Decision Contr. San Diego, CA, 761 (1979).

[3]I. Gath, and A. B. Geva, IEEE Trans. Pattern Anal. Machine Intell. 11, 773 (1989).

[4]N. R. Pal, K. Pal and J. C. Bezdek, IEEE Transactions on Fuzzy Systems. 1, 98 (1993).

[6]D. G. Somesh, Indian J. Pure Apply Math. 485, 6, 26(1995).

[7]Liu, H. C., Yih, J. M., Lin, W. C, & Wu, D. B (2009, Dec). Fuzzy C-Means Algorithm Based on common Mahalanobis Distances, Journal of Multiple Valued Logic & Soft Computing, 15, 581-595.

[8]J. Kennedy, and R. C. Eberhart, Particle Swarm Optimization, Proc. of the IEEE International Conference on Neural Networks, Vol. IV, pp.1942-1948, 1995.

[9]B. Balasko, J. Abonyi and B. Feil " Fuzzy Clustering and. Data Analysis Toolbox For Use with Matlab" From http://www.mathworks.com/matlab central/fileexchange/7473

[10]R. A. Fisher. Annals of Eugenics. 7, 179 (1936).