A Word Embedding based Approach for Word Sense Disambiguation

(1)

144 ISSN: 2005-4238 IJAST

A Word Embedding based Approach for Word Sense Disambiguation

Ilyas Ghanimi1, Elhabib Benlahmar1, Abderrahim Tragha1, Fadoua Ghanimi1 , 1 University Hassan II

Abstract

Word Sense Disambiguation (WSD) is main task in the area of natural language processing (NLP). Supervised learning methods are usually used to solve this problem. In this paper, we present a new supervised approach for Word Sense disambiguation based on word embedding which represents words, or concepts in a low-dimensional continuous space. It has the ability to capture semantic information from massive amounts of textual content. It’s shown how such representations of words and senses can be effectively applied for WSD purposes, as they encode rich semantic information.

Keywords - Word sense disambiguation, neural networks, word embeddings, Word2vec.

1 Introduction

Word Sense disambiguation in its broadest definition is nothing less than to determine the meaning of each word in its context, which seems to be an unconscious process of people. The WSD is an important step in the Natural Language Processing (NLP) process, such as text analysis, semantic interpretation, machine translation, information retrieval, text search and data acquisition.

Several Word Sens Disambiguation systems that rely on either supervised, unsupervised, or hybrid approaches.

The unsupervised methods have the advantage of not needing human intervention during the disambiguation process. While human intervention to do learning with a supervised approach can increase its performance.

Based on this idea, we present in this paper a supervised method for the Word Sense Disambiguation of English words. We show how a WSD system that makes use of word embeddings alone, if designed properly, can provide significant performance improvement.More precisely we considered Word2Vec [1] which is an implementation of word embedding. In our experiments, we used the pre- trained model on 100 billion words from Google News1.

This paper contains five sections; in the second section we investigate the WSD domain. The third section describes the Word2Vec method for word embedding.

In section fore, the new supervised approach for WSD and its main steps are introduced. The experimental results are described in section five. Finally, the sixth section is the conclusion.

2 Word Sense Disambiguation

Most commonly-used words in natural language have several meanings. In order to identify the intended meaning of a word one has to analyze the context in which it appears by directly exploiting information from raw texts.

(2)

145 ISSN: 2005-4238 IJAST

The task of automatically assigning predefined meanings to words in contexts, known as Word Sense Disambiguation, is a fundamental task in computational lexical semantics [3].

The approaches to lexical disambiguation are multiple and are generally categorized according to the nature of the resources used and their quantity. We thus distinguish:

2.1 Supervised methods

These methods make use of manually sense annotated data, elaborated by human experts. They are based on the assumption that a word’s context can provide enough evidence for its disambiguation. Supervised approaches, explore a large number of examples of words manually or automatically annotated in meaning, and which are usually linked to an automatic learning method such as a linear classifier [4][5] or more recently a network of recurrent neurons [6].

The machine-learning techniques are used to learn a classifier from labeled training sets, that is, sets of examples encoded in terms of a number of features together with their appropriate sense label (or class);

Supervised WSD uses machine-learning techniques for inducing a classifier from manually sense-annotated data sets. Usually, the classifier is concerned with a single word and performs a classification task in order to assign the appropriate sense to each instance of that word. The training set used to learn the classifier typically contains a set of examples in which a given target word is manually tagged with a sense from the sense inventory of a reference dictionary.

Generally, supervised approaches to WSD have obtained better results than unsupervised methods

2.2 Unsupervised methods

These methods create their own annotated corpus. only non-annotated texts are used, and the different meanings of the words are induced according to their contexts. Unsupervised models can make use of large amounts of data, but they are constrained by the fact that they can merely cluster distinct usages of words – they cannot employ precompiled lexicons with sense distinctions.

These methods lead to the difficulty of mapping their induced senses into a sense inventory and they still require manual intervention in order to perform such mapping [7].

2.3 Semi-supervised methods

Other methods, called semi-supervised, take a middle-ground approach. Here, a small manually annotated corpus is usually used as a seed for bootstrapping a larger annotated corpus. Examples of these approaches were presented by Mihalcea and Faruque[8]. A second option is to use a wordaligned bilingual corpus approach, based on the assumption that an ambiguous word in one language could be unambiguous in the context of a second language, hence helping to annotate the sense in the first language.

2.4 Knowledge-based methods

These methods are based on existing lexical resources, such as knowledge bases, semantic networks, dictionaries and thesauri. Their main feature is their coverage, since they function independently of annotated data and can exploit the graph structure of semantic networks to identify the most suitable meanings. These methods could obtain good performance using structured knowledge, rivaling supervised methods.

(3)

146 ISSN: 2005-4238 IJAST

Knowledge-based methods represent a distinct category in word sense disambiguation (WSD). Along with corpus-based methods, they represent one of the main categories of algorithms developed for word sense disambiguation. The performance of such knowledge intensive methods is usually exceeded by their corpus-based alternatives, but they have the advantage of a larger coverage.

Knowledge-based methods for WSD are usually applicable to all words in unrestricted text, as opposed to corpus-based techniques, which are applicable only to those words for which annotated corpora are available [11].

3 Word embedding with Word2vec

An embedding is a representation of a topological object, such as a manifold, graph, or field, in a certain space in such a way that its connectivity or algebraic properties are preserved [2].

The currently most popular word embeddings in the literature are provided by the Word2vec Toolkit [1][9]. That work introduced simpler neural architectures that are trainable in reasonable time, due to the fact that they do not have hidden non- linear layers. These shallow networks are less powerful than deep feedforward neural nets, let alone RNNs, but they compensate with much greater speed. It turns out that by training on large data (on the order of billions of words; e.g., the pre- trained Google vectors )[10].

These word embeeding networks can achieve good representations for individual words. This is demonstrated through experiments on word similarity and relatedness datasets, as well as on analogy making. Simple arithmetical operations on the vectors give impressive and immediately tangible results that suggest the representations do encode particular semantic features in the shared space [10].

Mikolov and al, first proposed two architectures (CBOW and Skip- gram) for learning word embeddings that are less expensive in terms calculation time than previous models. Then, they improved these models by improving learning speed and the performance of word embeddings [9][10].

These architectures offer two main advantages:

- They do not contain hidden layers;

- They allow the language model to take into account the context.

Word2vec belongs to the class of methods called “neural language models”. Using a scheme that is much simpler than previous work in this domain, where neural networks with many hidden units and several non-linear layers were normally constructed [8]. In what follows, we will present the two architectures CBOW and Skip-gram.

(4)

147 ISSN: 2005-4238 IJAST

Figure 1: The CBOW and Skip-gram architectures 3.1 CBOW

This model is named CBOW for Continuous Bag-of-Words or Continuous Bag- of-Words. It’s architecture allows predicting a word according to its context, which corresponds to the preceding words and the following words. In this architecture, the projection layer is shared by all words: all words are projected in the same position.

An input layer denotes the bag of words representation of the surrounding words, and contains one input element per glossary word. It is projected linearly to the hidden encoding layer. The hidden layer is then mapped to an output Huffman code representation of the given word.

Once the network is trained, the projections from each input unit to the middle encoding layer are used to represent the associated glossary word. Finally, the model compares its prediction with reality and makes the corrections by retro- propagation of the gradient. This model seeks to maximize the following equation:

(1) 𝑆 = ¹

𝑁 ^𝑁_𝑖=1log⁡𝑝(𝑚_𝑖\𝑚_𝑖−𝑛, … … , 𝑚_𝑖−1, 𝑚_𝑖+1, 𝑚_𝑖+𝑛)

The model receives a window of n words around the target word mi and N is to the number of words in the corpus.

One-word context

It’s the simplest version of the continuous bag-of-word model (CBOW) introduced in Mikolov et al [12]. We assume that there is only one word considered per context, which means the model will predict one target word given one context word, which is like a bigram model. Figure 2 shows the One-word context model, where V is the vocabulary size, and N is the hidden layer size.

(5)

148 ISSN: 2005-4238 IJAST

Figure 2: A simple CBOW model with only one word in the context Multi-word context

Figure 3 shows the CBOW model with a multi-word context setting. The CBOW model takes the average of the vectors of the input context words, and the average vector as the output, instead of directly copying the input vector of the input context word.

Figure 3: A CBOW model in multi word context 3.2 SKIP-GRAM

The skip-gram architecture is the opposite of CBOW. The target word is now used as input, and the context words are used as output. It allows predicting, for a given word, it’s context.

(6)

149 ISSN: 2005-4238 IJAST

In this architecture the input word is projected in the hidden layer then in the output layer to produce a vector. This vector is then compared to each of the context words and the network corrects itself by gradient backpropagation. In this way, the vector representation of the input word is close to the words producing the same context [12]. This architecture maximizes the following equation:

(2) 𝑆 = ¹

𝑁 ^𝑁_𝑖=1 _{−n≤j≤n≠0}log⁡p(𝑚_𝑖\𝑚_𝑖+𝑗)

The rationale as to why CBOW learning model was picked over skip-gram is that, Skip-gram is said to be accurate for infrequent words whereas CBOW is faster by a factor of window size which is also more appropriate for larger text corpora [12].

Figure 4: The skip-gram model.

4 The proposed approach

In this section, we explain how we integrated word embedding in Word Sense Disambiguation. More precisely, Word2vec algorithm is used to calculate the similarity between words. The proposed approach is developed in a supervised context where the different word senses are annotated.

(7)

150 ISSN: 2005-4238 IJAST

4.1 Vector space representation

For word representations, we use the word2vec Toolkit. The tool is based on a neural network language model, with modifications to make representation learning more efficient [1]. We make use of the skip-gram model, which takes each word in a sequence as an input to a log-linear classifier with a continuous projection layer, and predicts words within a certain range before and after the input word. We used the pre-trained model on 100 billion words from Google News1.

4.2 Disambiguisation method

This step consists of attributing for each ambiguous word its appropriate sense.

This is done by choosing the sense with the closest semantic proximity to its local and global context.

Similarity measurement between two vectors is calculated by the cosines similarity defined as follows:

(1) Cosines(v, w) = ^𝑣^𝑖

𝑖=𝑛 𝑖=1 .𝑤_𝑖

𝑣_𝑖². ^𝑖=𝑛_𝑖=0𝑤_𝑖²

𝑖=𝑛𝑖=0 .

Sense similarity is defined by the score Dimes defined below.

(2) Dismes(w, s) = ^{𝑐𝑜𝑠𝑖𝑛𝑒𝑠 𝑤}^𝑖,^𝑠

𝑑𝑖𝑠𝑡 𝑤_𝑖,𝑤

𝑛𝑖=0 × max_{0≤𝑖≤𝑁}𝑐𝑜𝑠𝑖𝑛𝑒𝑠( 𝑤_𝑖, 𝑠)/

𝑑𝑖𝑠𝑡(𝑤_𝑖, 𝑤)

In the formulas above, cosine(wi,s) denotes the value of the similarity cosine for the two words wi and s and dist(wi,w) denotes the distance between the two words wi and w where wi is a neighbor of w (We assume that stop words are removed) . Furthermore, n is window size of the surrounding words included in the sum and N is the size of the passage from what we will extract the sense of the word w.

4.3 The tuning procedure

The parameter to tune is the window size n of surrounding words will be included in the disambiguation process. The adjustment of the parameter n depends on the text size N. The Word Sense disambiguation procedure is applied with some values of n; procedure stability is then satisfied in the domain resulting from the smallest value of n where the sense extraction remains relevant.

4.4 Disambiguisation algorithm

Disambiguation process Algorithm

Input: ambiguous word

Output: sense of the ambiguous word

1: For each sense Sj of the ambiguous word w do:

(8)

151 ISSN: 2005-4238 IJAST

2: Calculate 𝑫𝒊𝒔𝒎𝒆𝒔(𝒘, 𝒔𝒋) 3: End:

4: Sense= S1

7: For each sense of the ambiguous word do:

8: If 𝑫𝒊𝒔𝒎𝒆𝒔 𝒘, 𝒔𝒋 > 𝐷𝑖𝑠𝑚𝑒𝑠(𝑤, 𝑠1) 9: Then: Sense = Sj

10: End

11: Assign the sense to the ambiguous word.

5 Experimental results 5.1 Example 1

As first example, we consider the word 'bank' and its two annotated senses. We applied the proposed approach to find out even the meaning is close to the sense of money or river. Thus, we used four short passages to figure out the performance of our method.

1. Bonnie and Clyde are two really famous criminals, I think they were bank robbers

2. My bank charges too much for an overdraft.

3. I went to the bank to deposit my check and get a new ATM card.

4. My grandfather planted his pole in the bank and got a great big catfish!

Simple passage Sens1: Money Sens2: River

1 0,14575376 0,0561954

2 0,15158529 0,0430811

3 0,04932138 0,0193134

4 0,00580921 0,0088824

Table 1: Results for WSD new approach applied to the four simple passages.

5.2 Example 2

As second example we will apply our new approach to a short text including some ambiguous words.

He got in the house. He opened the kitchen cabinet.

He found a jar of jam on the table. He spread some strawberry jam on a toast. He stood leaning against the entrance smoking his clay pipe and washing stars shining in the sky.

Figure 5: The short text used in example 2

Every ambiguous word has two annotated senses. The results are displayed in table 2. The ambiguous words are underlined in the text.

(9)

152 ISSN: 2005-4238 IJAST

Ambiguous word Annotated senses Sense extracted by the new approach

Cabinet ^Minister ^Wood

Wood

jam Jelly Jelly

Stuck

Table Kitchen Kitchen

Report

Pipe Smoke Smoke

Plumbing

Stars Movie Sky

Sky

Table 2: The extracted senses for the ambiguous words in the short text.

6 Conclusion

Word embeddings obtained through neural language models developed recently can capture semantic behaviors of words and very capably find their context features. In this paper, we presented a new supervised neural network approach for Word Sense Disambiguation that learns more semantic word representations by using both local and global context in learning. These learned word embeddings can be used to calculate the similarity between ambiguous words and their annotated senses.

To evaluate model’s abilities to capture sense of words in context, we applied our approach to some ambiguous words with human judgments in context sense. Our experiments show the usefulness of word embeddings in the WSD task.

References:

[1] Mikolov, T., Chen,K., Corrado, G. and Dean, J. (2013a). Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR.

[2] Matt Insall, Todd Rowland, and Eric W. Weisstein. 2015. “Embedding”. From MathWorld– A Wolfram Web Resource (access Sep 11, 2019).

[3] Jos´e Camacho-Collados, Mohammad Taher Pilehvar, and Roberto Navigli. 2015a. A Unified Multilingual Semantic Representation of Concepts. In Proceedings of the 53rd ACL, volume 1, pages 741–751, Beijing, China.

[4] Chan Y. S., Ng H. T., Zhong Z., « NUS-PT : Exploiting Parallel Texts for Word Sense Disambiguation in the English All-words Tasks », Proceedings of the 4th International Workshop on Semantic Evaluations, SemEval ’07, Association for Computational Linguistics, Stroudsburg,PA, USA, p. 253- 256, 2007b.

[5] Zhong Z., Ng H. T., « It Makes Sense : A Wide-coverage Word Sense Disambiguation System for Free Text », Proceedings of the ACL 2010 System Demonstrations, ACLDemos ’10, Association for Computational Linguistics, Stroudsburg, PA, USA, p. 78-83, 2010.

[6] Raganato A., Delli Bovi C., Navigli R., « Neural Sequence Learning Models for Word Sense Disambiguation », Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, p. 1167-1178, 2017.

[7] Navigli R, Word Sense Disambiguation: A Survey ACM Computing Surveys, Vol. 41, No. 2, Article 10, February 2009.

[8] Mihalcea, R., and Faruque, E. 2004. SenseLearner: Minimally supervised word sense disambiguation for all words in open text. In In Proc. of ACL/SIGLEX Senseval-3.

[9] Iacobacci, I., M. T. Pilehvar, and R. Navigli (2016, August). Embeddings for word sense disambiguation: An evaluation study. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, pp. 897–907. Association for Computational Linguistics.

(10)

153 ISSN: 2005-4238 IJAST

[10] Alexander Popov, Neural Network Models for Word Sense Disambiguation,An Overview, Cybernetics and Information Technologies Volume 18, No 1,2018.

[11] Mihalcea R. (2007) Knowledge-Based Methods for WSD. In: Agirre E., Edmonds P. (eds) Word Sense Disambiguation. Text, Speech and Language Technology, vol 33. Springer, Dordrecht

[12] Mikolov, W.T Yih, G. Zweig. 2013a. Linguistic Regularities in Continuous Space Word Representations. Proceedings of NAACL-HLT 2013, 746–751.

[13] SYNERGISTIC UNION OF WORD2VEC AND LEXICON FOR DOMAIN SPECIFIC SEMANTIC SIMILARITY, KEET

SUGATHADASA,BUDDHI AYESH, Nisansa de SilVA,AMAL SHEHAN PERERA AND Vindula Jayawardana, IEEEINTERNATIONAL CONFERENCE ON INDUSTRIAL AND INFORMATION SYSTEMS (ICIIS),2017.