As we have seen, a critical issue for implementing QA systems is the need to design answer search and extraction modules that capture the salient characteristics of pairs of text. These modules encode handcrafted rules based on syntactic patterns that detect the relations between a question and its candidate answers. Such rules are triggered when patterns in the question and the passage are found.
To reduce the burden of manual feature engineering for QA, we propose structural models based on kernel methods [Severyn et al., 2013a,b]. These systems are based on similar ideas from the frameworks in Chapter 4. The main adaptation for factoid QA is that we perform question analysis, and design a shallow syntactic representation of the text which contains relational tags.
Semantic Linking in Convolution Tree Kernels 83
In this section, we describe our structural relational model for QA, and then we show how to enrich the semantic representation of question/answer (q/a) pairs with the infor- mation provided by NLP annotators, i.e., question category (QC) and focus classifiers (FC) and Named Entity Recognizers (NERs). FC determines the constituent of the ques- tion to be linked to the named entities (NEs) of the answer passage. The target NEs are selected based on their compatibility with the category of the question, e.g., an NE of type PERSON is compatible with a category of a question asking for a human (HUM). This model greatly improves on IR baseline, e.g., BM25, by 40%, and on previous reranking models, up to 10%. We refer the interested reader to Severyn et al. [2013a] for details about all the experiments.
In this thesis, we only report the experiments on the TrecQA dataset from Severyn et al. [2013b], since we will evaluate our neural QA model from section 6.4 in the same answer sentence selection setting.
6.2.1 Learning to Rank with Kernels
Our reranking framework is similar to the one described in Chapter 4: given a query ques- tion, a search engine retrieves a list of candidate passages ranked by their relevancy. In alternative, a list of candidate passages can already be provided in the QA dataset. Then, various NLP components in a UIMA pipeline1 analyze each question together with its can-
didate answers, e.g., POS tagging, chunking, named entity recognition, constituency and dependency parsing, etc. These annotations are used to produce structural models, which are enriched with the output of a question focus detector and question type classifiers, such as to establish relational links for a given q/a pair. The resulting tree pairs are used to train a kernel-based reranker, that refines the initial ordering of the answer passages.
We use tree structures as our base encoding since they provide sufficient flexibility in representation, and an easier feature extraction process than, for example, graph struc- tures. We rely on the Partial Tree Kernel (PTK) [Moschitti, 2006] to handle feature engineering over the structural representations.
6.2.2 Relational Structural Models of Q/A Pairs
Our starting point for representing the text in a q/a pair is the shallow syntactic tree. In a shallow syntactic representation, first explored for QA in Severyn and Moschitti [2012], a question and its candidate answer are encoded into a tree where POS tags are located at the pre-terminal level, and word lemmas are at the leaf level. In the same model, a special REL tag is used to encode relationships between the question and the answer.
1
84 Semantic Linking in Convolutional Models
Dataset STK STKbow PTK
Mooney 81.9 81.5 80.5 SeCo-600 94.5 94.5 90.0 Bunescu 98.3 98.2 96.9
Table 6.1: Accuracy (%) of focus classifiers.
The adopted strategy is simple: lemmas shared between the question and the answer get their parents (POS tags) and grandparents (chunk labels) marked with a REL tag.
The idea of marking related fragments in the question and answer tree representations has been shown to yield more accurate relational models. However, such approach relies on a basic hard matching between word lemmas. Below, we propose a novel strategy to establish relational links using named entities extracted from the answer along with question focus and category classifiers. In particular, we use a question category to link the focus word of a question with the named entities extracted from the candidate answer.
Question Focus Classifier
For the identification of the focus, we train an SVM classifier with tree kernels applied to the constituency tree representation of the questions. To generate examples for training the reranker, we produce a number of trees where the parent (node with the POS tag) of each candidate word is annotated with a special FOCUS tag. Trees with the correctly tagged focus word constitute positive examples, while the others are negative examples. To detect the focus for a new question, we classify all possible trees obtained by tagging each single question word as a focus. The tree which receives the highest classification score reveals us the predicted focus word.
Our focus identifier is evaluated on three datasets: SeCo-600 [Quarteroni et al., 2012], Mooney GeoQuery [Damljanovic et al., 2010], and the dataset from Bunescu and Huang [2010]. The SeCo dataset contains 600 questions. We removed from that set multi-focus questions and non-interrogative queries. Mooney GeoQuery contains 250 questions mainly asking for U.S. geographical information. The first two datasets are very domain specific, so we also evaluate our classifier on the dataset from Bunescu and Huang [2010], which contains the first 2000 questions from the answer type dataset from Li and Roth [2006]. We discard questions with implicit and multiple focuses.
Table 6.1 displays the accuracies obtained by our question focus detector on the three datasets using different kernels: STK, STKbow (bag-of-words feature vector is added) and
Semantic Linking in Convolution Tree Kernels 85
Dataset STKbow PTK
UIUIC [Li and Roth, 2006] 86.1 82.2
TREC 11-12 79.3 78.1
Table 6.2: Accuracy (%) of question classifiers.
QA pipeline to automatically detect the question focus.
Question Category Classifier
Previous work in question classification demonstrates the power of syntactic/semantic tree representations coupled with tree kernels to train state-of-the-art models [Bloehdorn and Moschitti, 2007]. Hence, we opt for an SVM multi-class classifier that uses tree kernels to extract the question class. We train the former according to a one-vs-all strategy.
Table 6.2 contains the accuracies of the question classifier on the UIUIC dataset Li and Roth [2006] and the TREC 11-122 questions that we also use for testing the reranker.
We select the STKbow and PTK kernels.
Linking Focus Word with Named Entities using Question Class
We exploit the question category (automatically identified by a question type classifier) along with named entities found in the answer to establish relational links between the tree structures of a given q/a pair. In particular, once the question focus and question category are determined, we link the focus wordwf ocus in the question, with all the named entities
whose type matches the question class. Table 6.3 shows the mapping between question classes and named entity types. We perform tagging at the chunk level, and use two types of relational tags: the plain REL-FOCUS, and a tag which is furtherly typed with a question class, e.g.,REL-FOCUS-HUM. Figure 6.1 shows an example q/a pair where the typed relational tag is used in the shallow syntactic tree representation to link the chunk containing the question focusname to the named entities of the corresponding type Person (according to the mapping defined in Table 6.3), i.e., samuel langhorne clemens is linked tomark twain.
6.2.3 Feature Vector Representation
While the primary focus of our study is on the structural representations and relations between question and answer pairs, we also include basic features widely used in QA:
2
86 Semantic Linking in Convolutional Models
NER: Person NER: Person focus
Figure 6.1: Shallow tree structure CH with a typed relation tagREL-FOCUS-HUMto link a question focus wordname with the named entities of typePersoncorresponding to the question category (HUM).
Question Category Named Entity types
HUM Person
LOC Location
NUM Date, Time, Money, Percentage ENTY Organization, Person
Table 6.3: Mapping between question classes and named entity types.
Term-overlap features. Cosine similarity between question and answer: simCOS(q, a),
where the input vectors are composed of n-grams (up to tri-grams) of word lemmas and POS tags, and dependency triplets. For the latter, we simply hash the string value of the predicate defining the triple together with its argument, e.g. poss(name, twain).
PTK score. For the structural representations we also define a similarity based on the PTK score: simP T K(q, a) = P T K(q, a), where the input trees can be both depen-
dency trees and shallow chunk trees. Note that this similarity is computed between the members of a q/a pair.
NER relatednessrepresents a match between a question category and the related named entity types extracted from the candidate answer. It counts the proportion of named enti- ties in the answer that correspond to the question type returned by the question classifier.
We also extend this feature vector to obtain an advanced feature vector Vadv, which
contains 43 features in total. The latter include all the features already described in Section 4.4.6, and the similarity scores from a translation model (METEOR).
Comparison with State-Of-The-Art Systems 87