Evaluation

From Table4.12and Table4.13, we can see only popularity models (from either English or Chinese Wikipedia) can achieve better results. It is similar to the observation in TAC EL 2010 [42]: “a na¨ıve candidate ranking approach based on web popularity alone can achieve 71% micro-averaged accuracy.” The context model along achieves less linked accuracy but better NIL accuracy. This is due to mismatching of non-target Wikipedia pages, which contain many occurrences of query tokens and contexts. The context model leads to worse clustering performance, which proves that it recognizes many linked entities as NIL in ranking results. The joint model of popularity and context knowledge can achieve a balance of linked and NIL accuracy, with circa 5% increasing of micro averaged accuracy, as well as at least 4.5% clustering B-Cubed+ F-score over individual models. The new model achieves a significant performance improvement over our old system, and can reach the second position in Chinese EL comparing with participant systems in TAC 2011.

All PER ORG GPE

English Popularity Overall 0.710 0.548 0.864 0.802 in-KB 0.617 0.339 0.707 0.789 NIL 0.783 0.657 0.945 0.833 Chinese Popularity Overall 0.716 0.560 0.859 0.807 in-KB 0.615 0.330 0.747 0.771 NIL 0.794 0.681 0.918 0.892 Chinese Context Overall 0.710 0.665 0.850 0.629 in-KB 0.538 0.511 0.647 0.502 NIL 0.845 0.745 0.955 0.925 English Popularity + Chinese Context Overall 0.742 0.618 0.875 0.794 in-KB 0.657 0.507 0.713 0.746 NIL 0.809 0.676 0.959 0.908 Chinese Popularity + Chinese Context Overall 0.752 0.637 0.871 0.805 in-KB 0.680 0.529 0.747 0.763 NIL 0.807 0.693 0.935 0.900

Table 4.12.. Micro-averaged accuracy of the Generative EL Model strategy for Chinese queries on TAC 2011evaluation data.

4.8. Conclusion B–Cubed+ Precision B–Cubed+ Recall B–Cubed+ F-score English Popularity 0.605 0.706 0.652 Chinese Popularity 0.618 0.711 0.661 Chn. Context 0.608 0.674 0.639 Eng. Popularity + Chn. Context 0.643 0.725 0.682 Chn. Popularity + Chn. Context 0.656 0.731 0.691

Table 4.13.. Clustering evaluation of generative model EL strategy for Chinese queries on TAC 2011evaluation data.

4.8. Conclusion

We have developed a CLEL system for TAC KBP 2011, which uses a parallel monolingual architecture mainly consisting of a document retrieval and an entity clustering module. We show the feasibility of using Chinese Wikipedia as the KB to connect cross-lingual knowledge and avoid the propagation of translation errors to the retrieval module. To reach a better solution for the CLEL problem, we propose a simple generative EL model, which utilizes new entity generation and reranking model to improve the EL performance.

Part II.

Learning with Crowdsourced

Annotations

Introduction

Linguistic annotation is fundamental and indispensable to various tasks in natural language processing (NLP), information extraction (IE) and information retrieval (IR), since several supervised and semi-supervised machine learning methods trained on annotated data have been successfully introduced and applied in those fields and achieved state-of-the-art performances. Many open-domain evaluations such TREC and TAC evaluations, and professional linguistic resource annotation societies, such as Linguistic Data Consortium (LDC) 11 have devoted a lot of effort to collecting, annotating, and sharing linguistics resources in order to meet the gradually increasing needs for annotated data for various tasks.

Although those resources have promoted the research and applications in IR and NLP, those traditional in-house annotation procedures suffer from some inherent shortcomings. Training annotators takes a lot of effort and is expensive. The process of annotation usually takes a long time to accomplish, which makes it hard to support emergent tasks. On the other hand the access of annotation is not very convenient. Users have to participate in evaluations or become paying members to be granted access to specific datasets. The expenditure is not suitable in some situations. For individual research, the cost of training, supervising and managing a network of skilled and responsible annotators often outweighs the value of completing the annotation. Therefore we need alternate data collection and annotation paradigms, which are more flexible, faster and cheaper.

In recent years the emergence of crowdsourcing provided considerable opportunities for

a neologism defined as

the act of taking a task traditionally performed by an employee or contrac- tor, and outsourcing it to an undefined, generally large group of people or community in the form of an open call.

The usage of crowdsourcing can occur both online and offline. The traditional annotation can be viewed as offline crowdsourcing with comparably small groups of annotation, while the online crowdsourcing addressed in this thesis takes advantage of aggregating crowd intelligence. The development of online crowdsourcing platform, such Amazon Mechanical Turk (AMT)12and Crowdflower 13make it possible for requesters to publish and distribute their tasks to the large crowd of online workers and get work done effectively and efficiently.

In this part of the dissertation, first we will focus on using AMT to create a corpus of supporting passages for list question answering. Due to the variety of worker’s expertise and task complexity, the results of crowdsourcing annotation are not perfect. To improve the quality of annotation, we also employ various methods to learn gold standard-annotation from noisy AMT annotations. We train learning to ranking models with the enhanced annotation and achieve start-of-the-art performance.

Note that most of the work in Chapters 6 and 7 has been published as [116, Xu and Klakow, 2010]. Some of the work in the Chapter8has been published as [117, Xu and Klakow, 2012].

12_{http://www.mturk.com} 13_{http://crowdflower.com/}

5. Background on Question Answering

Asking questions has been one of the main activities for people to learn and acquire knowledge. With the development of internet and search engine techniques in recent years, people can easily access massive information. Coming with the epochal phenomenon of “information explosion”, it becomes more difficult for many people to easily manage and effectively organize information, which leads to the problem of information overload.

The development of automatic Question Answering (QA) systems offers a possible solution to information overload. The goal of QA is to extract answers for natural language questions. It can help us to filter and summarize knowledge from tremendous electronic information. To answer a question such as “List capital of European countries”, a QA system can properly comprehend the meaning of the question and automatically give the list of answers. Advanced QA technologies, which deeply learn the breadth of relevant content to more precisely extract and justify answers, “can help support professionals in critical and timely decision making in areas like compliance, health care, business intelligence, knowledge discovery, enterprise knowledge management, security, and customer support”. [28]

As a research field, QA is primarily concerned with developing theories, principles, algo- rithms and systems to help a user find correct answer(s) to a question from a collection of text documents. Recently lots of research activities have emerged to solve the QA problem from the perspective of Information Retrieval (IR) and Natural Language Processing (NLP). The surge was initiated by QA tracks of TREC (Text REtrieval Conference)1_{for English}

language in 1999 [106]. In the coming eight years increasingly powerful systems have been developed and tested on QA benchmarks. Following this trend, other evaluations have been organized to promote research, innovation, and development of multilingual and cross-lingual QA. There are CLEF (Cross Language Evaluation Forum)2for European languages [77] and NTCIR (NII-NACSIS Test Collection for IR Systems) for east Asian lan- guages3_{[30]. Those conferences defined some specific types of questions with restricted}

forms of answers which are easy to evaluate.

Over the decades, those evaluations have contributed to solve critical techniques in QA, and generated a resurgence of research topics for IR and NLP communities. Researchers have developed a suite of automatic QA technologies, to deal with various questions on specific knowledge in restricted domains and open questions about universal topics which are accessible to human beings.

This chapter focuses on the creation of question and answer datasets for the development and research in QA. We do not intend to give a though survey on research topics in QA, however we will give a general overview of underlying techniques in QA and how our following work on passage ranking can fit in the overall framework of QA system.

Section5.1briefly reviews the early history of QA. Section5.2introduces the chronicle of QA evaluation tracks. Section5.3presents the architecture of our latest QA systems for TAC 2008 to demonstrate how the overall system and underlying models work.