4 Evaluation
4.2 Web Service Discovery Evaluation
We also evaluated the effectiveness of web service discovery based on the three probabilistic topic models (labeled PLSA, LDA and CTM). The probabilistic
methods are compared with a text-matching approach (labeled Text-Search). For this experiment, we use the services description collected from the WSDL corpus. As described previously, the services are divided into eight domains and some queries templates are provided together with a relevant response set for each query. The relevance sets for each query consists of a set of relevant service and each services has a graded relevance valuerelevance(s)∈ {1,2,3}where 3 denoteshigh relevance to the query and 1 denotes alow relevance.
In order to evaluate the accuracy of our approach, we compute two standard measures used inInformation Retrieval:Precision at n (Precision@n) andNor-
malised Discounted Cumulative Gain (N DCGn). These evaluation techniques
are used to measure the accuracy of a search and matchmaking mechanism.
– Precision@n: In our context,P recision@n is a measure of the precision of
the service discovery system taking into account the firstnretrieved services. Therefore, P recision@n reflects the number of services which are relevant to the user query. The precision@n for a list of retrieved services is given by Equation 10:
P recision@n=|RelevantServices∩RetrievedServices|
|RetrievedServices| (10)
Where the list of relevant services to a given query is defined in the test collection. For this evaluation, we have considered only the services with a graded relevance value of 3 and 2.
– Normalised Discounted Cumulative Gain: N DCGn uses a graded relevance
scale of each retrieved service from the result set to evaluate the gain, or usefulness, of a service based on its position in the result list. This measure is particularly useful in Information Retrieval for evaluating ranking results. TheN DCGn fornretrieved services is given by Equation 11.
N DCGn= DCGn
IDCGn (11)
WhereDCGnis the Discounted Cumulative Gain andIDCGn is the Ideal Discounted Cumulative Gain. TheIDCGnis found by calculating theDCGn of the firstnreturned services. The DCGn is given by Equation 12.
DCGn= n i=1 2relevance(i)−1 log2(1 +i) (12)
Wherenis the number of services retrieved andrelevance(s) is the graded relevance of the service in theith position in the ranked list.
We evaluated our service discovery approach by calculating the P recision@n and N DCGn. In this experiment, we have selected randomly 12 queries from the test collection. The text description is retrieved from the query templates and used as the query string. We consider that the size of the services to be returned was set to 50. The averageP recision@nandN DCGn are obtained over all 12 queries for CTM, LDA, PLSA and Text-Search. The results are shown in Figure 2(a) and 2(b).
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 5 10 15 20 25 30 35 40 45 50 Precision@n
Number of Services Retrieved Text-Search PLSA LDA CTM 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 5 10 15 20 25 30 35 40 45 50 Precision@n
Number of Services Retrieved Text-Search PLSA LDA CTM 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 5 10 15 20 25 30 35 40 45 50 Precision @ n
Number of Services Retrieved CTM with Conditional Probability LDA with Conditional Probability PLSA with Conditional Probability CTM with Multidimensional Angle LDA with Multidimensional Angle PLSA with Multidimensional Angle Text-Search
Fig. 2.(a) Comparaison of average Precision@n values over 12 queries. (b) Comparai- son of averageN DCGnvalues over 12 queries. (c) Comparaison of average Precision@n values over 12 queries for all methods using both Conditional Probability and Multi- dimensional Angle.
The comparison ofP recision@nshows that the CTM and LDA perform bet- ter than Text-Search and PLSA. The probabilistic methods based on CTM and LDA used the information captured in the latent factors to match web services based on the conditional probability of the user query. Text-Search and PLSA were unable to find some of the relevant web services that were not directly related to the user’s queries through CTM and LDA. The low precision results obtained by probabilistic method based on PLSA are due to limited number of concepts used for training the model. In this context, web service descriptions are similar to short documents. Therefore, the method based on PLSA model is not able to converge to a high precision using these limited concepts.
In Information retrieval,N DCGN gives higher scores to systems which rank a search result list with higher relevance first and penalizes systems which return services with low relevance. TheN DCGn values for all queries can be averaged to obtain a measure of the average performance of a ranking algorithm. In our experiments, we consider services with graded relevance values from 3 (high relevance) to 1 (low relevance) for this evaluation. N DCGn values vary from 0 to 1. The results obtained for N DCGn show that the both CTM and LDA perform better than the other search methods. Thus, the probabilistic methods based on both CTM and LDA give a higherN DCGnthan all other methods for any number of web services retrieved. This reflects the accuracy of the ranking mechanism used by our method. Text-Search and PLSA methods have a low N DCGnbecause, as shown in theP recision@nresults, both methods are unable to find some of the highly relevant services.
In order to compare the accuracy of our approach with existing approaches, we have implemented the approach proposed by Cassar et al. [6], which uses the proximity measure called Multidimentional Angle (also known as Cosine
Similarity); a measure, which uses the cosine of the angle between two vectors
[17]. In the first time, we represent the user’s query as a distribution over topics. Thus, for each topic zf we calculate the relatedness between query Q and zf based ontopic−worddistributionφusing Equation 13.
P(Q|zf) =
wi∈Q
Then, we calculate the similarity between the user’s query and a web service by computing the Cosine Similarity between a vector containing the query’s distribution over topicsqand a vector containing the service’s distribution over topicsp. The multidimensional angle between a vectorpand a vectorqcan be calculated using Equation 14:
Cos(p, q) = p.q p.q= t i=1piqi t i=1p2i t i=1qi2 (14)
wheretis the number of topics.
The comparison of average P recision@n (See Figure 2(c)) shows that the probabilistic method CP (i.e. Conditional Probability) performs better than the MA (i.e. Multidimensional Angle) for all the probabilistic topic models. The re- sults show that the CTM and LDA perform better than Text-Search and PLSA.
5
Conclusion
In this paper, we have used several probabilistic topic models (i.e. PLSA, LDA and CTM) to extract latent factors from web service descriptions. Then, the learned latent factors are used to group services into clusters. Indeed, the cate- gorization of services is often done with human intervention. To overcome this limitation, we propose to vary the number of topics (which can be considered as clusters, with one difference, which is that we can model the observations in a more compressed way than it would be if the model was based on clusters) to au- tomatically obtain the categories to which services belong. The accuracy of the three probabilistic clustering algorithms is compared with a classical clustering algorithm (i.e. K-means). The results show that the clustering method based on both CTM and LDA perform better than PLSA and K-means. In our work, we propose also to use the probabilistic topic models to discover the web services that match with the user query. We evaluated our service discovery approach by calculating theP recision@nandN DCGn. The comparison ofP recision@n andN DCGnshow that the CTM and LDA perform better than the other search methods (i.e. Text-Search and PLSA). This reflects the accuracy of the ranking mechanism used by our method. The probabilistic methods based on both CTM and LDA used the information captured in the latent factors to match web ser- vices based on the conditional probability of the user query. The obtained results show that the topic models provide a scalable and interoperable solution for au- tomated service discovery in large service repositories. Future work will focus on developing a new probabilistic model based on the latent factors to tag web services automatically.
References
1. Abramowicz, W., Haniewicz, K., Kaczmarek, M., Zyskowski, D.: Architecture for Web services filtering and clustering. In: ICIW 2007 (2007)
2. Atkinson, C., Bostan, P., Hummel, O., Stoll, D.: A Practical Approach to Web service Discovery and Retrieval. In: ICWS 2007 (2007)
3. Blei, D., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)
4. Blei, D., Lafferty, J.D.: A Correlated Topic model of Science. In: AAS 2007, pp. 17–35 (2007)
5. Cassar, G., Barnaghi, P., Moessner, K.: Probabilistic methods for service clustering. In: Proceeding of the 4th International Workshop on Semantic Web Service Match- making and Resource Retrieval, Organised in conjonction the ISWC 2010 (2010) 6. Cassar, G., Barnaghi, P., Moessner, K.: A Probabilistic Latent Factor approach to
service ranking. In: ICCP 2011, pp. 103–109 (2011)
7. Elgazzar, K., Hassan, A., Martin, P.: Clustering WSDL Documents to Bootstrap the Discovery of Web Services. In: ICWS 2010, pp. 147–154 (2010)
8. Dong, X., Halevy, A., Madhavan, J., Nemes, E., Zhang, J.: Similarity Search for Web Services. In: VLDB Conference, Toronto, Canada, pp. 372–383 (2004) 9. Heß, A., Kushmerick, N.: Learning to Attach Semantic Metadata to Web Services.
In: Fensel, D., Sycara, K., Mylopoulos, J. (eds.) ISWC 2003. LNCS, vol. 2870, pp. 258–273. Springer, Heidelberg (2003)
10. Hofmann, T.: Probabilistic Latent Semantic Analysis. In: UAI, pp. 289–296 (1999) 11. Kokash, N.: A Comparison of Web Service Interface Similarity Measures. Frontiers
in Artificial Intelligence and Applications, vol. 142, pp. 220–231 (2006)
12. Lausen, H., Haselwanter, T.: Finding Web services. In: European Semantic Tech- nology Conference, Vienna, Austria (2007)
13. Liu, W., Wong, W.: Web service clustering using text mining techniques. IJAOSE 2009 3(1), 6–26 (2009)
14. Ma, J., Zhang, Y., He, J.: Efficiently finding web services using a clustering semantic approach. In: CSSSIA 2008, pp. 1–8. ACM, New York (2008)
15. Nayak, R., Lee, B.: Web service Discovery with Additional Semantics and Cluster- ing. In: IEEE/WIC/ACM 2007 (2007)
16. Porter, M.F.: An Algorithm for Suffix Stripping. Program 1980 14(3), 130–137 (1980)
17. Platzer, C., Rosenberg, F., Dustdar, S.: Web service clustering using multidimen- tional angles as proximity measures. ACM Trans. Internet Technol. 9(3), 1–26 (2009)
18. Salton, G.: Automatic Text Processing: The Transformation, Analysis, and Re- trieval of Information by Computer. Addison-Wesley Longman Publishing Co., Inc., Boston (1989)
19. Sivashanmugam, K., Verma, A.P., Miller, J.A.: Adding Semantics to Web services Standards. In: ICWS 2003, pp. 395–401 (2003)
20. Steyvers, M., Griffiths, T.: Probabilistic topic models. In: Landauer, T., Mcnamara, D., Dennis, S., Kintsch, W. (eds.) Latent Semantic Analysis: A Road to Meaning. Lawrence Erlbaum (2007)
21. Yu, Q.: Place Semantics into Context: Service Community Discovery from the WSDL Corpus. In: Kappel, G., Maamar, Z., Motahari-Nezhad, H.R. (eds.) ICSOC 2011. LNCS, vol. 7084, pp. 188–203. Springer, Heidelberg (2011)
22. Zhao, Y., Karypis, G.: Empirical and theoretical comparisons of selected criterion functions for document clustering. In: Machine Learning 2004, vol. 55, pp. 311–331 (2004)
23. Zhao, Y., Karypis, G.: Evaluation of hierarchical clustering algorithms for docu- ment datasets. In: CIKM 2002 (2002)
K.-K. Lau, W. Lamersdorf, and E. Pimentel (Eds.): ESOCC 2013, LNCS 8135, pp. 34–48, 2013. © Springer-Verlag Berlin Heidelberg 2013