Automatic Parsing of Text - Unsupervised Dependency-based Patterns

CHAPTER 3: STEM SENTENCES SELECTION VIA IE

3.2 Unsupervised Dependency-based Patterns

3.2.1 Automatic Parsing of Text

Syntactic analysis of text, also known as parsing, is a process of determining the grammatical structure of its sentence constituents. In syntactic analysis, a sentence is recursively decomposed into smaller units called constituents or phrases. These constituents are then categorised into noun phrases or verb phrases according to their internal structures. Syntactic analysis is generally represented in the form of a parse tree. Syntax plays an important role in making language useful for communication. Syntax in linguistics attempts to describe the language in terms of certain rules. In relation to automatic parsing, many theoretical approaches are presented so far in the area of syntax.

Dependency trees are regarded as a suitable basis for semantic pattern acquisition as they abstract away from the surface structure to represent relations between elements (entities) of a sentence. Semantic patterns represent semantic relations between elements of sentences. One of the advantages of using dependency trees is that they provide a useful structure for the sentences by annotating edges with dependency functions e.g. subject, object etc. (Fundel et al., 2007). In a dependency tree a pattern is defined as a path in the dependency tree passing through zero or more intermediate nodes within a dependency tree (Sudo et al., 2001). Stevenson and Greenwood (2009) provided an insight of the usefulness of dependency patterns in their work (see Section 2.6.3.6). In their work, they revealed that dependency parsers have the

advantage of generating analyses which abstract away from the surface realisation of text to a greater extent than phrase structure grammars tend to, resulting in semantic information being more accessible in the representation of the text which can be useful for IE.

Several approaches in IE have relied on dependency trees in order to extract patterns for the automatic acquisition of IE systems (Yangarber et al., 2000; Sudo et al., 2001; Sudo et al., 2003; Stevenson and Greenwood, 2005 and Greenwood et al., 2005) (see Sections 2.6.3). Apart from IE, Lin and Pantel (2001) used dependency trees in order to infer rules for question answering while Szpektor et al. (2004) had made use of dependency trees for paraphrase identification. Moreover, dependency parsers are used most recently in the systems which identify protein interactions in biomedical texts (Katrenko and Adriaans, 2006; Erkan et al., 2007 and Saetre et al., 2007).

All the abovementioned approaches have used different pattern models based on the particular part of the dependency analysis. The motive behind all of these models is to extract the necessary information from text without being overly complex. All of the pattern models have made use of the semantic patterns based on the dependency trees for the identification of items of interest in text. These models vary in terms of their complexity, expressivity and performance in an extraction scenario.

3.2.2 Our Approach

In our dependency-based approach, we employed two dependency tree pattern models: SVO pattern model (SVO patterns) and an adapted version of the linked chain pattern model. We used SVO pattern model (Yangarber et al., 2000; see Section 2.6.3.1 for more details) as a baseline in our experiments. In the SVO model, we extracted all subject-verb-object tuples from the dependency parse of a sentence and discarded the remainder of the dependency parse.

Our adapted linked chain pattern model approach (Afzal et al., 2011) is based on the linked chain pattern model presented by Greenwood et al. (2005) (see Section 2.6.3.4). Linked chain pattern model combines the pairs of chain in a dependency tree

which share common verb root but no direct descendants. We selected the linked chain dependency pattern model as it is the best performing pattern model and its performance is consistently better than the collective performance of both SVO and chain dependency pattern models (Stevenson and Greenwood, 2009).

In our approach, we have treated every Named Entity (NE) as a chain in a dependency tree if it is less than 5 dependencies away from the verb root and the word linking the NEs to the verb root are from the category of content words (Verb, Noun, Adverb and Adjective) along with prepositions. We consider only those chains in the dependency tree of a sentence which contain NEs, which is much more efficient than the subtree model of Sudo et al. (2003) (see Section 2.6.3.3), where all subtrees containing verbs are taken into account. This allows us to extract more meaningful patterns from the dependency tree of a sentence. We extract all NE chains which follow aforementioned rule from a sentence and combine them together. The extracted patterns are then stored in a database along with their frequencies.

3.2.3 Extraction of Candidate Patterns

As with the learning of surface-based patterns, our general approach to learn dependency-based patterns consists of the same two main stages: (i) the construction of potential patterns from an unannotated domain corpus and (ii) their relevance ranking.

3.2.3.1 Pre-processing steps

The first step in constructing candidate patterns is to perform NE recognition in an unannotated domain corpus. We will explain the whole process of candidate patterns extraction from the dependency trees with the help of an example shown below:

We used the GENIA32 tagger for NER and the following example shows the NER from a biomedical text:

<protein> Fibrinogen </protein> activates <protein> NF-kappa B </protein> in <cell_type> mononuclear phagocytes </cell_type>.

Once the NEs are recognised in the domain corpus by the GENIA tagger, we replace all the NEs with their semantic class respectively, so the aforementioned sentence is transformed into the following sentence.

PROTEIN activates PROTEIN in CELL.

The transformed sentences are then parsed by using the Machinese Syntax33 parser (Tapanainen and Järvinen, 1997). The Machinese Syntax parser uses a functional dependency grammar for parsing. The Machinese Syntax parser first labels each word with its all possible function types and then applies a collection of handwritten rules to introduce links between specific types in a given context and remove all the other function types. The Machinese Syntax parser was evaluated in terms of correct identification of attached heads on three different genres in the Bank of English (Järvinen, 1994) data. Table 11 shows the results in terms of precision and recall.

Precision Recall

Broadcast 93.4% 88.0%

Literature 96.0% 88.6%

Newspaper 95.3% 87.9%

Table 11: Percentages of heads correctly attached

Stevenson and Greenwood (2007) used three different parsers including the Machinese Syntax parser in order to compare different IE models. They carried out their experiments on two different corpora: MUC-6 corpus and a biomedical corpus (see Section 2.6.3.6).

32_{http://www-tsujii.is.s.u tokyo.ac.jp/GENIA/tagger/}

Figure 20 shows the dependency tree produced by the parser for the aforementioned adapted sentence example.

Figure 20: Dependency tree of ‘PROTEIN activates PROTEIN in CELL’

The analyses produced by the Machinese Syntax parser are encoded to make the most of information they contain and ensure consistent structures from which patterns could be extracted. Figure 21 shows the encoded output of a biomedical text.

<W ID="2" LEMMA="protein" POS="N" FUNC="SUBJ" DEP="3">PROTEIN</W> <W ID="3"LEMMA="activate" POS="V" FUNC="+FMAINV"DEP="1">activates</W> <W ID="4" LEMMA="protein" POS="N" FUNC="OBJ" DEP="3">PROTEIN</W> <W ID="5" LEMMA="in" POS="PREP" FUNC="ADVL" DEP="3">in</W>

Figure 21: Encoded biomedical text 3.2.3.2 Dependency-based patterns

After the encoding, the patterns are extracted from the dependency trees using the methodology described in Section 3.2.2. For example the following SVO pattern was extracted from the Figure 21.

[V/activate] (subj[PROTEIN] + obj[PROTEIN])

The following adapted linked chain patterns were extracted from the same example (Figure 21):

[V/activate] (subj[PROTEIN] + obj[PROTEIN])

[V/activate] (obj[PROTEIN] + prep[in] + p[CELL_TYPE])

For dependency tree patterns representation, we employed a similar sort of formalism to that used by Sudo et al. (2003). Each node in the dependency tree is represented in the format a[B] e.g. subj[PROTEIN] where a is the dependency relation between this node and its parent (subj) and B is the semantic class of the named entity. The relationship between nodes is represented as X (A+B+C) which indicates that nodes

A, B and C are direct descendants of X. The patterns along with their frequencies are

stored in a database. Similar to surface-based patterns, we also filtered out the patterns containing only stop-words in dependency-based patterns too. In SVO patterns, we extracted only those SVO patterns where both subject and object are named entities. Table 12 shows some examples of the most frequent SVO patterns along with their frequencies extracted from the GENIA corpus.

Patterns Frequency

[V/contain] (subj[DNA] + obj[DNA]) 34

[V/activate] (subj[PROTEIN] + obj[PROTEIN]) 32 [V/contain] (subj[PROTEIN] + obj[PROTEIN]) 19 [V/induce] (subj[PROTEIN] + obj[PROTEIN]) 18

[V/encode] (subj[DNA] + obj[PROTEIN]) 17

[V/express] (subj[CELL_TYPE] + obj[PROTEIN]) 16 [V/inhibit] (subj[PROTEIN] + obj[PROTEIN]) 14 [V/form] (subj[PROTEIN] + obj[PROTEIN]) 6 [V/regulate] (subj[PROTEIN] + obj[PROTEIN]) 6 [V/stimulate] (subj[PROTEIN] + obj[PROTEIN]) 6

Table 12: SVO patterns along with their frequencies

The total numbers of SVO patterns extracted from the GENIA corpus is very small and one of the main reasons for this is that the SVO pattern model does not perform well in the biomedical domain. This fact was also highlighted by Stevenson and Greenwood (2009) and they argued that the reason behind this is that in the

biomedical domain named entities are described in ways that the SVO pattern model is unable to represent as it is restricted to verbs and their direct arguments only. In their work they compared various pattern models using two domains: MUC-6 and biomedical data (see Section 2.6.3.6)

Table 13 shows some examples of the most frequent adapted linked chain patterns along with their frequencies extracted from the GENIA corpus.

Patterns Frequency

[V/contain] (subj[DNA] + obj[DNA]) 34

[V/activate] (subj[PROTEIN] + obj[PROTEIN]) 32 [V/contain] (subj[PROTEIN] + obj[PROTEIN]) 19 [V/induce] (subj[PROTEIN] + app[PROTEIN]) 19

[V/activate] (a[DNA] + obj[PROTEIN]) 18

[V/induce] (subj[PROTEIN] + obj[PROTEIN]) 18 [V/interact] (subj[PROTEIN] + prep[in] + p[PROTEIN]) 17 [V/induce] (subj[PROTEIN] + obj[phosphorylation] + prep[of] + p[PROTEIN])

[V/encode] (subj[DNA] + obj[PROTEIN]) 17

[V/induce] (subj[PROTEIN] + subj[PROTEIN]) 17

Table 13: Adapted linked-chain patterns along with their frequencies

In our experiments we preferred to use an adapted linked chain pattern model as it is possible to encode more of the information present in a sentence than compared to SVO pattern model (Section 2.6.3.1) or chain pattern model (Section 2.6.3.2) and this fact was also highlighted by Stevenson and Greenwood (2009). Moreover, SVO pattern model performed very poorly in the biomedical domain as compared to linked chain pattern model (Stevenson and Greenwood, 2009).

3.2.4 Pattern Ranking

In order to rank extracted candidate patterns, we employed the same information theoretic concepts: Information Gain (IG), Information Gain Ratio (IGR), Mutual Information (MI), Normalised Mutual Information (NMI) and statistical tests of association: Log-likelihood (LL) and Chi-Square (CHI), along with meta-ranking and tf-idf ranking methods which we used in the surface-based approach (see Section 3.1.4 for further details).

3.2.5 Evaluation

We used the same experimental data as used in the surface-based patterns experiments (see Section 3.1.5 for further details). The numbers of adapted linked chain dependency patterns extracted from each corpus are: GENIA 5066, WEB 13653, GENIA+WEB 17694, BNC 419274 and GENIA EVENT 3031. The quality of extracted patterns is evaluated by employing the same approach as described in Section 3.1.5.2.

3.2.6 Results

We conducted our experiments on all 3 corpora (GENIA, WEB and GENIA+WEB). Similar to the surface-based approach, here we will discuss the results for the GENIA corpus only while the complete results for all three corpora in terms of precision, recall and F-scores are given in Appendix C. Figure 22 shows the rank-thresholding results for adapted linked chain dependency patterns using the GENIA corpus. Here precision scores are represented along the Y-axis (for complete results see Table 13 in Appendix C).

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Top 100 Top 200 Top 300

P re c is io n IG IGR MI NMI LL CHI Meta tf-idf

Figure 22: Rank-thresholding results for adapted linked chain patterns using GENIA corpus

Figure 22 shows that similar to the surface-based approach CHI and NMI are the best performing ranking methods while MI is the worst. Moreover, IG, IGR and LL achieved quite similar results.

In the next step we used score-thresholding measure for each corpus similar to the surface-based approach. Here too, we are considering only those threshold scores that give us high precision scores (see Table 14 in Appendix C for complete results for each corpus). Figure 23 shows the results of score-thresholding measures for adapted linked chain dependency patterns using the GENIA corpus.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 >0.1 >0.2 >0.3 Threshold Scores P re c is io n IG IGR MI NMI LL CHI Meta tf-idf

Figure 23: Rank-thresholding results for adapted linked chain patterns using GENIA corpus

3.2.6.1 Ranking methods

We carried out our experiments using both ranking measures: rank-thresholding and score-thresholding. In both set of experiments, similar to the surface-based approach (Section 3.1.6) CHI is the best performing ranking method but recall scores are very low. MI is the worst performing ranking method while IG, IGR and LL attained quite similar results. Moreover, we found that there is a no statistical significant difference (p < 0.05) between IG and LL, IGR and LL. Similar to the surface-based approach tf- idf achieved quite reasonable results but it is not the best performing ranking method. Figure 24 shows the precision scores of the best performing ranking method (CHI) in the score-thresholding method for dependency patterns.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 >0.08 >0.09 >0.1 >0.2 >0.3 >0.4 Threshold Scores Pr e c is io n _GENIA WEB GENIA+WEB

Figure 24: Precision scores of best performing ranking method for adapted linked chain dependency patterns in score-thresholding

3.2.6.2 Score vs. rank thresholding

We also found that the score-thresholding method produces better results than the rank-thresholding as we are able to achieve higher precision with the former measure.

3.2.6.3 Precision vs. F-measure optimisation

As mentioned earlier, CHI is the best performing ranking method in terms of precision scores while recall scores are very low. Using NMI ranking method we are able to achieve quite reasonable results in terms of both precision and recall scores. Figure 25 and Figure 26 show precision, recall and F-score for the GENIA corpus using these ranking methods (CHI and NMI).

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 >0.06 >0.07 >0.08 >0.09 >0.1 >0.2 >0.3 Threshold Scores Precision Recall F-score

Figure 25: Precision, recall and F-score for adapted linked chain dependency patterns in score-thresholding measure using CHI

Figure 26: Precision, recall and F-score for adapted linked chain dependency patterns in score-thresholding measure using NMI

Similar to the surface-based approach (Section 3.1.6.4); in the dependency-based approach the score-thresholding measure achieves higher precision than the rank- thresholding. Applications such as MCQ generation, as mentioned earlier, rely on high precision so the score-thresholding method gives us the opportunity to attain higher precision but low recall.

3.3 Comparison between Surface-based and Dependency-

In document Unsupervised relation extraction for e-learning applications (Page 92-103)