Building Lexical Chains

Considering the algorithms introduced in Sections 3.3.1 and 3.3.2 we provide an illus- trative example of how our lexical chains are built. Let us consider the following sentence:

Beets, carrots, and potatoes are grandma and grandpa’s favorite dish for lunch!

As explained in Algorithms 3 and 4, we require a document composed of synsets to build our lexical chains. Before extracting the most suitable synset for each relevant token, we lowercase all the words, remove all punctuation and common English stopwords. After

pre-processing the sentence, we end up with the following list of tokens {beets, carrots, potatoes, grandma, grandpa, favorite, dish, lunch}. Next, we apply the MSSA algorithm (Section 3.2) to obtain the proper synset for each word in that list, as shown in Table 3.1. To make our toy example clearer, we reference the synsets for each word using the notation in row Filtered Synset Sentence. For each synset Synset(‘word.pos.number’) we name it as word.pos.number, where word is the word itself, pos is the part-of-speech tag used, and number is the synset number in WordNet.

Table 3.1: Sentence example transformation from words into synset.

Type Sentence

Original Sentence Beets, carrots, and potatoes are grandma and grandpa 's favorite dish for lunch!

Pre-Processed Sentence {beets, carrots, potatoes, grandma, grandpa, favorite, dish, lunch}

Synset Sentence

{Synset(‘beet.n.02’), Synset(‘carrot.n.03’), Synset(‘potato.n.01’), Synset(‘grandma.n.01’), Synset(‘grandfather.n.01’),

Synset(‘favorite.n.01’), Synset(‘dish.n.02’), Synset(‘lunch.n.01’)} Filtered Synset Sentence {beet.n.02, carrot.n.03, potato.n.01, grandma.n.01, grandfather.n.01,

favorite.n.01, dish.n.02, lunch.n.01}

Once we have a specific synset for each token, we can start to build our chains. For the FLLC II algorithm, we extract all related synsets from each synset in the Filtered Synset Sentence (Table 3.1). The related synsets are other synsets extracted from 19 attributes in WordNet [1], such as hypernyms, homonyms, and meronyms to name a few. In Table 3.2, we illustrate some of the extracted synsets, in which a “-” represents the absence of a related synset.

In Table 3.2, it is possible to notice that some synsets have an overlap with respect to their related synsets. For example, the synsets {beet.n.02, carrots.n.03, potatoes.n.01} have root_vegetable.n.01 as a common relation, which provides a semantic connection between them. The same characteristic is also present when considering grandparent.n.01 for the synsets {grandma.n.01, grandfather.n.01}. For the cases where there are no common related synsets, a new chain must be built, as in the case of favorite.n.01, which has no

Table 3.2: Sample of related synsets extracted from WordNet. Main Synsets Related Synset 01 Related Synset 02 Related Synset 03 beet.n.02 root_vegetable.n.01 sugar_beet.n.02 beetroot.n.01 carrot.n.03 carrot.n.02 root_vegetable.n.01 -

potato.n.01 baked_potato.n.01 french_fries.n.01 root_vegetable.n.01 grandma.n.01 grandparent.n.01 nan.n.01 -

grandfather.n.01 grandparent.n.01 - -

favorite.n.01 choice.n.01 - -

dish.n.02 nutriment.n.01 meal.n.01 ingredient.n.03 lunch.n.01 meal.n.01 business_lunch.n.01 -

semantic relation with any of its adjacent neighbors. For the FXLC II algorithm, we take a different approach and group the synsets according to an arbitrary chunk size cs. In Table 3.3, we illustrate how the lexical chains are grouped for the FLLC II and FXLC II algorithms, considering cs = 3 for the latter. The chains are tagged in order and according to the algorithm used, where (A) represents the chains from FLLC II and (B) the chains from FXLC II. In case cs is greater than the number of available synsets in the document, that chain will be smaller.

Table 3.3: Lexical chains construction.

Lexical Chain Type Lexical Chain

A - FLLC II {A1:{beet.n.02, carrot.n.03, potato.n.01}, A2:{grandma.n.01, grandfather.n.01}, A3:{favorite.n.01}, A4:{dish.n.02, lunch.n.01}}

B - FXLC II {B1:{beet.n.02, carrot.n.03, potato.n.01}, B2:{grandma.n.01, grandfather.n.01, favorite.n.01}, B3:{dish.n.02, lunch.n.01}}

Now that we have grouped the possible synsets to represent our chains, let us assume we have a pre-trained synset embeddings model for all the tokens in our corpus. For the sake of simplicity, our model has only five dimensions as Table 3.4 shows.

The next step is to represent each lexical chain considering the vectors of its inner elements. Two tasks are necessary to find the best representation for our chains: (i) calculate the average for each sub-chain, considering their vector values, in Table 3.3 and (ii) select the synset with the highest cosine similarity within each sub-chain with respect to their average calculated in (i). Table 3.5 shows the average vector calculated for each

Table 3.4: Fictional pre-trained synset embeddings model. Synsets Dim-01 Dim-02 Dim-03 Dim-04 Dim-05 beet.n.02 0.926 0.620 -0.278 -0.380 0.717 carrot.n.03 -0.525 0.361 0.839 -0.989 -0.420 potato.n.01 -0.675 -0.172 1.000 0.964 0.868 grandma.n.01 0.159 -0.795 0.663 0.405 0.617 grandfather.n.01 -0.035 0.469 -0.183 0.659 0.361 favorite.n.01 0.121 0.581 -0.603 0.148 0.046 dish.n.02 0.667 -0.257 -0.379 0.300 0.723 lunch.n.01 -0.874 0.458 0.945 0.717 0.314 sub-chain in Table 3.3.

Table 3.5: Vector average for lexical chains centroids.

Lexical Chain Type Centroid Dim-01 Dim-02 Dim-03 Dim-04 Dim-05

A - FLLC II A1 -0.091 0.270 0.520 -0.135 0.388 A2 0.062 -0.163 0.240 0.532 0.489 A3 0.121 0.581 -0.603 0.148 0.046 A4 -0.104 0.101 0.283 0.509 0.519 B - FXLC II B1 -0.091 0.270 0.520 -0.135 0.388 B2 0.082 0.085 -0.041 0.404 0.341 B3 -0.104 0.101 0.283 0.509 0.519

Table 3.6 shows the cosine similarity between the elements of each chain and their respective average synset vector (centroid). Bold values represent the highest similarity value for that centroid with respect to a specific synset in the sub-chain. The closer to 1, the most similar a synset is to its centroid, and a better candidate to represent a chain it is. On the other hand, the closer to 0, the most dissimilar a synset is to its centroid, and the worst candidate it is. It is no surprise that if a chain is composed of one synset that synset will have the value of 1.0.

In Table 3.7 we show the resulting lexical chains for both FLLC II and FXLC II algo- rithms considering the highest cosine similarity values from Table 3.6. Even though this illustrates a simple example, we can see how the configuration of our chains can change depending on how their synsets are organized. In FLLC II the chain {grandma.n.01, grandfather.n.01} has grandma.n.01 as a representative since it has the highest cosine sim-

Table 3.6: Cosine similarity between lexical chains elements and their centroids.

FLLC II FXLC II

| Synset Centroid CosineSim Synset Centroid CosineSim beet.n.02 A1 0.264 beet.n.02 B1 0.264 carrot.n.03 A1 0.509 carrot.n.03 B1 0.509 potato.n.01 A1 0.578 potato.n.01 B1 0.578 grandma.n.01 A2 0.815 grandma.n.01 B2 0.420 grandfather.n.01 A2 0.572 grandfather.n.01 B2 0.882 favorite.n.01 A3 1.000 favorite.n.01 B2 0.341 dish.n.02 A4 0.365 dish.n.02 B3 0.365 lunch.n.01 A4 0.746 lunch.n.01 B3 0.746

ilarity value. However, when considering FXLC II, the synset favorite.n.01 gets merged to {grandma.n.01, grandfather.n.01} as a result of the selected chunk size. Thus, grandfa- ther.n.01 is chosen as a new synset representative for our chain based on its cosine similarity value.

Table 3.7: Final chains for FLLC II and FXLC II algorithms.

Lexical Chain Type Lexical Chain

FLLC II {potato.n.01, grandma.n.01, lunch.n.01} FXLC II {potato.n.01, grandfather.n.01, lunch.n.01}

Once the synset input corpus is transformed to either flexible or fixed lexical chains, we feed them into a word2vec implementation, as described in Section 3.3.3. As a result, we produce a synset (chains) embeddings model that can be used in any NLP downstream task. For the lexical chains algorithms, we explore the document classification task to validate our techniques, detailed in Section 4.3.

CHAPTER IV