IIITH: Domain Specific Word Sense Disambiguation
Siva Reddy IIIT Hyderabad
India
Diana McCarthy Lexical Computing Ltd.
United Kingdom
Abhilash Inumella IIIT Hyderabad
India
Mark Stevenson University of Sheffield
United Kingdom
Abstract
We describe two systems that participated in SemEval-2010 task 17 (All-words Word Sense Disambiguation on a Specific Do-main) and were ranked in the third and fourth positions in the formal evaluation. Domain adaptation techniques using the background documents released in the task were used to assign ranking scores to the words and their senses. The test data was disambiguated using the Personalized PageRank algorithm which was applied to a graph constructed from the whole of WordNet in which nodes are initialized with ranking scores of words and their senses. In the competition, our systems achieved comparable accuracy of 53.4 and 52.2, which outperforms the most frequent sense baseline (50.5).
1 Introduction
The senses in WordNet are ordered according to their frequency in a manually tagged corpus, Sem-Cor (Miller et al., 1993). Senses that do not oc-cur in SemCor are ordered arbitrarily after those senses of the word that have occurred. It is known from the results of SENSEVAL2 (Cotton et al., 2001) and SENSEVAL3 (Mihalcea and Edmonds, 2004) that first sense heuristic outperforms many WSD systems (see McCarthy et al. (2007)). The first sense baseline’s strong performance is due to the skewed frequency distribution of word senses. WordNet sense distributions based on SemCor are clearly useful, however in a given domain these distributions may not hold true. For example, the first sense for “bank” in WordNet refers to “slop-ing land beside a body of river” and the second
to “financial institution”, but in the domain of “fi-nance” the “financial institution” sense would be expected to be more likely than the “sloping land beside a body of river” sense. Unfortunately, it is not feasible to produce large manually sense-annotated corpora for every domain of interest. McCarthy et al. (2004) propose a method to pre-dict sense distributions from raw corpora and use this as a first sense heuristic for tagging text with the predominant sense. Rather than assigning pre-dominant sense in every case, our approach aims to use these sense distributions collected from do-main specific corpora as a knowledge source and combine this with information from the context.
Our approach focuses on the strong influence of domain for WSD (Buitelaar et al., 2006) and the benefits of focusing on words salient to the do-main (Koeling et al., 2005). Words are assigned a ranking score based on its keyness (salience) in the given domain. We use these word scores as another knowledge source.
Graph based methods have been shown to produce state-of-the-art performance for unsu-pervised word sense disambiguation (Agirre and Soroa, 2009; Sinha and Mihalcea, 2007). These approaches use well-known graph-based tech-niques to find and exploit the structural properties of the graph underlying a particular lexical knowl-edge base (LKB), such as WordNet. These graph-based algorithms are appealing because they take into account information drawn from the entire graph as well as from the given context, making them superior to other approaches that rely only on local information individually derived for each word.
Our approach uses the Personalized PageRank algorithm (Agirre and Soroa, 2009) over a graph
representing WordNet to disambiguate ambigu-ous words by taking their context into consider-ation. We also combine domain-specific informa-tion from the knowledge sources, like sense distri-bution scores and keyword ranking scores, into the graph thus personalizing the graph for the given domain.
In section 2, we describe domain sense ranking. Domain keyword ranking is described in Section 3. Graph construction and personalized page rank are described in Section 4. Evaluation results over the SemEval data are provided in Section 5.
2 Domain Sense Ranking
McCarthy et al. (2004) propose a method for finding predominant senses from raw text. The method uses a thesaurus acquired from automat-ically parsed text based on the method described by Lin (1998). This provides the top k nearest neighbours for each target wordw, along with the distributional similarity score between the target word and each neighbour. The senses of a word ware each assigned a score by summing over the distributional similarity scores of its neighbours. These are weighted by a semantic similarity score (using WordNet Similarity score (Pedersen et al., 2004) between the sense ofwand the sense of the neighbour that maximizes the semantic similarity score.
More formally, let Nw = {n1, n2, . . . nk}
be the ordered set of the top k scoring neighbours of w from the thesaurus with associated distributional similarity scores {dss(w, n1), dss(w, n2), . . . dss(w, nk)}. Let
senses(w) be the set of senses of w. For each sense ofw(wsi ∈senses(w))a ranking score is
obtained by summing over thedss(w, nj)of each
neighbour (nj ∈ Nw) multiplied by a weight.
This weight is the WordNet similarity score (wnss) between the target sense (wsi) and the
sense of nj (nsx ∈ senses(nj))that maximizes
this score, divided by the sum of all such WordNet similarity scores for senses(w) and nj. Each
sense wsi ∈ senses(w)is given a sense ranking
scoresrs(wsi)using srs(wsi) =
X
njNw
dss(w, nj)× Xwnss(wsi, nj)
wsisenses(w)
wnss(wsi, nj)
wherewnss(wsi, nj) =
maxnsx∈senses(nj)(wnss(wsi, nsx))
Since this approach requires only raw text, sense rankings for a particular domain can be gen-erated by simply training the algorithm using a corpus representing that domain. We used the background documents provided to the partici-pants in this task as a domain specific corpus. In general, a domain specific corpus can be obtained using domain-specific keywords (Kilgarriff et al., 2010). A thesaurus is acquired from automatically parsed background documents using the Stanford Parser (Klein and Manning, 2003). We usedk= 5 to built the thesaurus. As we increasedkwe found the number of non-domain specific words occur-ring in the thesaurus increased and negatively af-fected the sense distributions. To counter this, one of our systems IIITH2 used a slightly modified ranking score by multiplying the effect of each neighbour with its domain keyword ranking score. The modified sense rankingmsrs(wsj) score of
sensewsiis msrs(wsi) =
X
njNw
dss(w, nj)× Xwnss(wsi, nj)
wsisenses(w)
wnss(wsi, nj)
×krs(nj)
wherekrs(nj)is the keyword ranking score of
the neighbournjin the domain specific corpus. In
the next section we describe the way in which we computekrs(nj).
WordNet::Similarity::lesk (Pedersen et al., 2004) was used to compute word similarity wnss. IIITH1 and IIITH2 systems differ in the way senses are ranked. IIITH1 usessrs(wsj)whereas
IIITH2 system uses msrs(wsj) for computing
sense ranking scores in the given domain. 3 Domain Keyword Ranking
krs(w) = XLL(w)
wi∈words(domain)
LL(wi)
The above score represents the keyness of the word in the given domain. Top ten keywords (in descending order ofkrs) in the corpora provided for this task arespecies, biodiversity, life, habitat, natura1, EU, forest, conservation, years, amp2.
4 Personalized PageRank
Our approach uses the Personalized PageRank al-gorithm (Agirre and Soroa, 2009) with WordNet as the lexical knowledge base (LKB) to perform WSD. WordNet is converted to a graph by repre-senting each synset as a node (synset node) and the relationships in WordNet (hypernymy, hyponymy etc.) as edges between synset nodes. The graph is initialized by adding a node (word node) for each context word of the target word (including itself) thus creating a context dependent graph (person-alized graph). The popular PageRank (Page et al., 1999) algorithm is employed to analyze this per-sonalized graph (thus the algorithm is referred as personalized PageRank algorithm) and the sense for each disambiguous word is chosen by choos-ing the synset node which gets the highest weight after a certain number of iterations of PageRank algorithm.
We capture domain information in the personal-ized graph by using sense ranking scores and key-word ranking scores of the domain to assign initial weights to the word nodes and their edges (word-synset edge). This way we personalize the graph for the given domain.
4.1 Graph Initialization Methods
We experimented with different ways of initial-izing the graph, described below, which are de-signed to capture domain specific information.
Personalized Page rank (PPR): In this method, the graph is initialized by allocating equal prob-ability mass to all the word nodes in the context including the target word itself, thus making the graph context sensitive. This does not include do-main specific information.
1In background documents this word occurs in reports de-scribing Natura 2000 networking programme.
2This new word”amp”is created by our programs while extracting body text from background documents. The HTML code”&”which represents the symbol”&”is converted into this word.
Keyword Ranking scores with PPR (KRS + PPR): This is same as PPR except that context words are initialized withkrs.
Sense Ranking scores with PPR (SRS + PPR): Edges connecting words and their synsets are as-signed weights equal tosrs. The initialization of word nodes is same as in PPR.
KRS + SRS + PPR: Word nodes are initialized withkrsand edges are assigned weights equal to srs.
In addition to the above methods of unsuper-vised graph initialization, we also initialized the graph in asemi-supervisedmanner. WordNet (ver-sion 1.7 and above) have a fieldtag cnt for each synset (in the file index.sense) which represents the number of times the synset is tagged in vari-ous semantic concordance texts. We used this in-formation,concordance score(cs) of each synset, with the above methods of graph initialization as described below.
Concordance scores with PPR (CS + PPR): The graph initialization is similar to PPR initialization additionally with concordance score of synsets on the edges joining words and their synsets.
CS + KRS + PPR: The initialization graph of KRS + PPR is further initialized by assigning con-cordance scores to the edges connecting words and their synsets.
CS + SRS + PPR: Edges connecting words and their synsets are assigned weights equal to sum of the concordance scores and sense ranking scores i.e. cs+srs. The initialization of word nodes is same as in PPR.
CS + KRS + SRS + PPR: Word nodes are ini-tialized withkrsand edges are assigned weights equal tocs+srs.
PageRank was applied to all the above graphs to disambiguate a target word.
4.2 Experimental details of PageRank
Tool: We used UKB tool3 (Agirre and Soroa,
2009) which provides an implementation of per-sonalized PageRank. We modified it to incorpo-rate our methods of graph initialization. The LKB used in our experiments is WordNet3.0 + Gloss which is provided in the tool. More details of the tools used can be found in the Appendix.
Normalizations: Sense ranking scores (srs) and keyword ranking scores (krs) have diverse ranges. We foundsrsgenerally in the range between 0 to
Precision Recall
Unsupervised Graph Initialization
PPR 37.3 36.8
KRS + PPR 38.1 37.6
SRS + PPR 48.4 47.8
KRS + SRS + PPR 48.0 47.4 Semi-supervised Graph Initialization
CS + PPR 50.2 49.6
CS + KRS + PPR 50.1 49.5 * CS + SRS + PPR 53.4 52.8 CS + KRS + SRS + PPR 53.6 52.9
Others
1stsense 50.5 50.5
[image:4.595.189.408.58.259.2]PSH 49.8 43.2
Table 1: Evaluation results on English test data of SemEval-2010 Task-17. * represents the system which we submitted to SemEval and is ranked 3rd in public evaluation.
1 andkrsin the range 0 to 0.02. Since these scores are used to assign initial weights in the graph, these ranges are scaled to fall in a common range of [0, 100]. Using any other scaling method should not effect the performance much since PageRank (and UKB tool) has its own internal mechanisms to normalize the weights.
5 Evaluation Results
Test data released for this task is disambiguated using IIITH1 and IIITH2 systems. As described in Section 2, IIITH1 and IIITH2 systems differ in the way the sense ranking scores are computed. Here we project only the results of IIITH1 since IIITH1 performed slightly better than IIITH2 in all the above settings. Results of1stsensesystem
pro-vided by the organizers which assigns first sense computed from the annotations in hand-labeled corpora is also presented. Additionally, we also present the results of Predominant Sense Heuristic (PSH) which assigns every wordwwith the sense wsj (wsj ∈ senses(w)) which has the highest
value ofsrs(wsj) computed in Section 2 similar
to (McCarthy et al., 2004).
Table 1 presents the evaluation results. We used TreeTagger 4 to Part of Speech tag the test data.
POS information was used to discard irrelevant senses. Due to POS tagging errors, our precision values were not equal to recall values. In the com-petition, we submitted IIITH1 and IIITH2 systems with CS + SRS + PPR graph initialization. IIITH1
4http://www.ims.uni-stuttgart.de/ projekte/corplex/TreeTagger/
and IIIH2 gave performances of 53.4 % and 52.2 % precision respectively. In our later experiments, we found CS + KRS + SRS + PPR has given the best performance of 53.6 % precision.
From the results, it can be seen when srs in-formation is incorporated in the graph, precision improved by 11.1% compared to PPR in unsuper-vised graph initialization and by 3.19% compared to CS + PPR in semi-supervised graph initializa-tion. Also little improvements are seen whenkrs information is added. This shows that domain specific information like sense ranking scores and keyword ranking scores play a major role in do-main specific WSD.
The difference between the results in unsu-pervised and semi-suunsu-pervised graph initializations may be attributed to the additional information the semi-supervised graph is having i.e. the sense dis-tribution knowledge of non-domain specific words (common words).
6 Conclusion
Acknowledgements
The authors are grateful to Ted Pedersen for his helpful advice on the WordNet Similarity Pack-age. We also thank Rajeev Sangal for supporting the authors Siva Reddy and Abhilash Inumella.
References
Eneko Agirre and Aitor Soroa. 2009. Personaliz-ing pagerank for word sense disambiguation. In
EACL ’09: Proceedings of the 12th Conference of the European Chapter of the Association for Compu-tational Linguistics, pages 33–41, Morristown, NJ, USA. Association for Computational Linguistics. Paul Buitelaar, Bernardo Magnini, Carlo Strapparava,
and Piek Vossen. 2006. Domain-specific wsd. In
Word Sense Disambiguation. Algorithms and Appli-cations, Editors: Eneko Agirre and Philip Edmonds. Springer.
Scott Cotton, Phil Edmonds, Adam Kilgarriff, and
Martha Palmer. 2001. Senseval-2. http://www.
sle.sharp.co.uk/senseval2.
A. Ferraresi, E. Zanchetta, M. Baroni, and S. Bernar-dini. 2008. Introducing and evaluating ukwac, a very large web-derived corpus of english. In
Proceed-ings of the WAC4 Workshop at LREC 2008, Marrakesh, Morocco.
Adam Kilgarriff, Siva Reddy, Jan Pomik´alek, and Avi-nesh PVS. 2010. A corpus factory for many
lan-guages. InLREC 2010, Malta.
Dan Klein and Christopher D. Manning. 2003. Accu-rate unlexicalized parsing. InACL ’03: Proceedings of the 41st Annual Meeting on Association for Com-putational Linguistics, pages 423–430, Morristown, NJ, USA. Association for Computational Linguis-tics.
Rob Koeling, Diana McCarthy, and John Carroll. 2005. Domain-specific sense distributions and
pre-dominant sense acquisition. InHLT ’05:
Proceed-ings of the conference on Human Language Tech-nology and Empirical Methods in Natural Language Processing, pages 419–426, Morristown, NJ, USA. Association for Computational Linguistics.
Dekang Lin. 1998. Automatic retrieval and
cluster-ing of similar words. In Proceedings of the 17th
international conference on Computational
linguis-tics, pages 768–774, Morristown, NJ, USA.
Associ-ation for ComputAssoci-ational Linguistics.
Diana McCarthy, Rob Koeling, Julie Weeds, and John Carroll. 2004. Finding predominant word senses in
untagged text. InACL ’04: Proceedings of the 42nd
Annual Meeting on Association for Computational Linguistics, page 279, Morristown, NJ, USA. Asso-ciation for Computational Linguistics.
Diana McCarthy, Rob Koeling, Julie Weeds, and John Carroll. 2007. Unsupervised acquisition of
pre-dominant word senses. Computational Linguistics,
33(4):553–590.
Rada Mihalcea and Phil Edmonds, editors. 2004.
Proceedings Senseval-3 3rd International Workshop on Evaluating Word Sense Disambiguation Systems.
ACL, Barcelona, Spain.
George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In
Proceedings of the ARPA Workshop on Human Lan-guage Technology, pages 303–308. Morgan Kauf-man.
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation rank-ing: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab, November.
Ted Pedersen, Siddharth Patwardhan, and Jason Miche-lizzi. 2004. Wordnet::similarity: measuring the
re-latedness of concepts. InHLT-NAACL ’04:
Demon-stration Papers at HLT-NAACL 2004 on XX, pages 38–41, Morristown, NJ, USA. Association for Com-putational Linguistics.
Paul Rayson and Roger Garside. 2000. Comparing
corpora using frequency profiling. InProceedings
of the workshop on Comparing corpora, pages 1– 6, Morristown, NJ, USA. Association for Computa-tional Linguistics.
Ravi Sinha and Rada Mihalcea. 2007. Unsupervised graph-basedword sense disambiguation using mea-sures of word semantic similarity. InICSC ’07: Pro-ceedings of the International Conference on Seman-tic Computing, pages 363–369, Washington, DC, USA. IEEE Computer Society.
Appendix
Domain Specific Thesaurus, Sense Ranking Scores and Keyword Ranking Scores are accessi-ble at
http://web.iiit.ac.in/˜gvsreddy/ SemEval2010/
Tools Used:
• UKB is used with options–ppr –dict weight. Dictio-nary files which UKB uses are automatically generated using sense ranking scoressrs.
• Background document words are canonicalized using KSTEM, a morphological analyzer
• The Stanford Parser is used to parse background docu-ments to build thesaurus
