Evaluation

We evaluate the Synset2Vec model in the task of fine-grained WSD (Alessandro Raganato and Navigli, 2017). Four unsupervised WSD systems based on synset and context similarity are used for evaluation in terms of measuring synset-synset similarity, synset-word similarity and synset-text similarity n the proposed joint semantic vector space of synsets and words. The goal of the experiments is to validate the following hypotheses:

• H1: The joint vector representation of synsets and words is effective.

• H2: Synset expansion and increasing the number of profiles can enhance the model.

We use the all-words fine-grained WSD evaluation datasets from (Alessandro Raganato and Navigli, 2017), as they uniformed the data format and sense inventory (WordNet 3.0) with several datasets, including Senseval-2 (Edmonds and Cotton, 2001), Senseval-3 (Snyder and Palmer, 2004), SemEval-07 (Pradhan et al., 2007), SemEval-13 (Navigli et al., 2013) and SemEval-15 (Moro and Navigli, 2015). Moreover, for enriching the synset profiles, we use the sense-annotated corpus SemCor (Miller et al., 1993), and a larger automatically constructed corpus OMSTI (One Million Sense-Tagged Instances) (Taghipour and Ng, 2015). Both datasets are publicly available. The statistics of evaluation dataset and training corpus are shown in Table 4.2, where #Sent and #Annotation denote the number of sentences and annotations, respectively. According to different amount of training data used for Synset2Vec model, we create four models M1, M2, M3, and M4 for comparison: (1) M1: WordNet synset and gloss; (2) M2: M1 with synset expansion; (3) M3: M2 with SemCor; (4) M4: M3 with OMSTI. We then run these four models in four WSD systems respectively in order to validate our hypotheses. For the evaluation metric of WSD, we use the standard definitions of precision, recall and F-measure (Navigli, 2009). As the datasets are used for fine-grained WSD where each target word instance corresponds to a synset in WordNet, we only show the F-measure score for evaluation since the scores of precision, recall and F-measure are the same.

In addition, as we are experimenting with knowledge-based WSD, the implemented WSD systems with different models are compared against the Most Frequent Sense (MFS) com-

puted from SemCor and WordNet first sense (WNFS). These two baseline systems are assumed to be difficult to challenge (Camacho-Collados et al., 2016). We also include a conventional semantic similarity metric (WN-JCN) using WordNet taxonomy and informa- tion content computed from SemCor (Jiang and Conrath, 1997) for comparison, which has been shown to yield the best performance in this task (Navigli, 2009). We run WN-JCN in WSD1 and WSD4. Moreover, we also use a word overlap based approach (Lesk, 1986) for comparison. Finally, as we are actually testing the word and synset inter-connectivity in a shared vector space, we also include results from a similar experiment from (Mancini et al., 2016), and denote them as SW2V (Mancini et al., 2016) and AutoExtend (Rothe and Schütze, 2015). Apart from the four Synset2Vec models, we also use a combination of WSD3, M4, and a back-off strategy (Camacho-Collados et al., 2016) (M4+WNFS) for comparing to SW2V, since such strategy is also included. We set the threshold value θ = 0.5 as the confidence to decide if use the similarity measure or WordNet First Sense (WNFS).

Our implementation is based on NLTK 1 and Gensim 2, using Python. We use the CBOW model of Synset2Vec and set 200 dimensions for synset and word vectors with con- textual window of 12. The evaluation results are shown in Table 4.3. In this table, we can see that the basic model M1 already obtains a fairly good performance when compar- ing to other systems, especially considering synset-synset relation (WSD1 and WSD4). M1 only contains information from WordNet itself, which shows the effectiveness of Synset2Vec. Note that WN-JCN only works for nouns and leaves other type of words with default WNFS. Furthermore, in M4+WNFS, the best F-scores are obtained comparing to other knowledge- based WSD systems, although it is still not as good as strong baselines MFS and WNFS. With such experimental observations, we validate our first hypothesis H1 that Synset2Vec is effective. Comparing M1, M2, and M3, the experimental results have shown that adding more profiles and performing synset expansion, the effectiveness of the models can be im- proved. This validates our hypothesis H2. However, comparison between M3 and M4 shows that automatically generated annotations (OMSTI) may not be as good as manually anno- tations (SemCor), due to the lower accuracy of the annotations. Comparing to SW2V and AutoExtend, although Synset2Vec does not rely on existing word embeddings from large corpora, it shows improvement in fine-grained sense representation based on WordNet and its annotation datasets. Moreover, the Synset2Vec does not train synset to synset relations like SW2V, but it obtains good performance in synset-synset similarity task based on vec- tors derived from related words. In other words, Synset2Vec actually obtains synset vector

1_{http://www.nltk.org/}

representation from words and related synsets.

Additionally, by investigating the difference between WSD3 and WSD4, we noticed that more annotation examples help to improve the synset-text performance. With larger training data, contextual words may appear frequently so that special word collocation patterns are more effective than individual words. This property of Synset2Vec is inherited from Doc2Vec, in which synsets and contextual words are used to predict target words together so that synset vectors memorize the meaning of frequent word patterns in context. This property is useful for applications requiring synset-text relation. In summary, through the experiments in WSD, we validate the hypotheses and show that the Synset2Vec model is effective in representing synsets and words in a shared semantic vector space, by illustrating the inter-connectivity of synset-synset, synset-word and synset-text.

System Senseval-2 Senseval-3 SemEval-07 SemEval-13 SemEval-15 MFS 65.6 66.0 54.5 63.8 67.1 WNFS 66.8 66.2 55.2 63.0 67.8 Lesk 50.6 44.5 32.0 53.6 51.0 SW2V - - 39.9 54,0 - AutoExtend - - 17.6 31.0 - WSD1 + WN-JCN 59.7 58.2 48.8 56.2 63.3 WSD4 + WN-JCN 55.3 51.9 45.1 55.1 61.9 WSD1 + M1 43.4 37.2 29.2 43.3 44.4 WSD1 + M2 43.2 39.6 34.7 47.1 45.5 WSD1 + M3 46.0 42.8 30.8 47.4 48.5 WSD1 + M4 41.6 37.9 28.1 43.1 46.6 WSD2 + M1 44.6 37.8 30.5 45.3 46.3 WSD2 + M2 46.6 42.9 32.3 48.7 50.1 WSD2 + M3 50.2 45.9 33.6 48.8 52.3 WSD2 + M4 48.7 44.0 33.4 49.8 53.7 WSD3 + M1 45.2 38.1 30.8 46.2 46.1 WSD3 + M2 45.4 43.1 32.5 49.1 49.5 WSD3 + M3 50.1 45.1 35.4 49.6 52.2 WSD3 + M4 50.0 44.7 34.9 50.9 54.1 WSD4 + M1 43.7 38.1 29.5 43.8 44.8 WSD4 + M2 44.3 40.1 31.9 46.9 46.9 WSD4 + M3 48.9 42.5 34.5 48.0 50.0 WSD4 + M4 43.8 39.3 33.2 45.1 49.4 M4+WNFS 61.2 58.9 49.2 58.2 63.6

Table 4.3: F-Measures percentage of different knowledge-based WSD systems and embed- ding models in five all-words fine-grained WSD datasets.