While there are several further measures to evaluate information retrieval systems (see, e.g., Baeza-Yates and Ribeiro-Neto, 1999; Manning et al., 2008), the presented measures of precision at k, average precision at k, and normalized discounted cumulative gain will be used in this work (e.g., in Chapter 5).
Note that all three measures have been formulated to evaluate single queries. Of course, they can be used for sets of queries by summing over all scores for the queries and dividing the sum by the number of queries (Manning et al., 2008: p.160). In the case of average precision, the measure is then called mean average precision. Note that “each information need [is weighted] equally in the final reported number, even if many documents are relevant to some queries whereas very few are relevant to other queries” (Manning et al., 2008: p.161).
2.7 Summary of the Chapter
In this chapter, we placed the thesis into its research context of natural language processing and text mining, and in particular of information extraction and information retrieval. In addition, we introduced basic concepts such as named entity recognition and its subtasks of recognition, classification, and normalization. We pointed out the importance of the normalization subtask which makes it possible to not only extract named entities as a specific type of entity (e.g., a person) but which also allows to link entity mentions to respective entities (e.g., a specific person).
Furthermore, the concepts of temporal and geographic information have been introduced, their key characteristics have been explained and compared, and it was described how temporal and geographic information – in particular temporal and geographic expressions – occur in textual documents. Finally, we briefly described the basics of the UIMA framework and presented several evaluation measures which we do not only refer to when evaluating our own approaches in this work but also when discussing related approaches.
In the next chapter, we will address the information extraction task of temporal tagging by surveying the research area and presenting our contributions to the research field.
3 Cross-domain Temporal Tagging
In this chapter, the task of temporal tagging is addressed with a special focus on multilingual and cross- domain temporal tagging. In Section 3.1, we demonstrate the importance of temporal taggers and give a brief description of the research field of temporal information extraction, of which temporal tagging is a subtask. The state-of-the-art of temporal tagging is described in Section 3.2 by presenting annotation standards, research competitions, annotated corpora, as well as state-of-the-art approaches to temporal tagging and existing temporal taggers. This section will be closed with a discussion of open issues.
Motivated by these open issues in state-of-the-art temporal tagging, we then outline differences and challenges of temporal tagging documents from different domains (genres) in Section 3.3 and multilingual temporal tagging in Section 3.4. These differences and challenges in domain-sensitive and multilingual temporal tagging and since they have rarely been addressed in previous work are also the main reason why we developed the multilingual, cross-domain temporal tagger HeidelTime, which is detailed in Section 3.5. After an extensive evaluation of HeidelTime in Section 3.6, we close the chapter by discussing possible future work related to HeidelTime (Section 3.7) and summarizing the chapter of temporal tagging (Section 3.8).
3.1 Introduction and Motivation
The task of temporal tagging can be defined as the extraction and normalization of temporal expressions from text documents according to some annotation guidelines. In general, temporal tagging is thus a specific type of named entity recognition and normalization (cf. Section 2.2).
Since temporal information is prevalent in many kinds of documents – as it is exemplarily shown in Figure 3.1 for three types of documents – the extraction and normalization of temporal expressions from documents are important preprocessing steps for many natural language processing and understanding tasks. For example, in information retrieval, temporal information can be used, among others, for temporal clustering of documents along timelines and querying a document collection using temporal constraints – an issue also addressed later in this thesis (Chapter 5). While Alonso et al. (2007) gave an overview of the value of temporal information in information retrieval, we described a wide range of research trends in temporal information retrieval together to re-emphasizes this importance (Alonso et al., 2011).
Information Retrieval is usually not the research area for which rich natural language understanding is necessary. Thus, in research areas requiring rich natural language understanding, such as information extraction, document summarization, machine translation, and question answering, temporal information is often utilized a fortiori. For example, the ultimate goal of temporal information extraction can be summarized as “[t]he automatic identification of all temporal referring expressions (timexes), events, and temporal relations within a text” (UzZaman et al., 2013). Thus, a prerequisite of the final task of temporal annotation, i.e., the identification of temporal relations between events, and between events and temporal expressions, is to extract and normalize temporal expressions.
3 Cross-domain Temporal Tagging
(a) News document. (b) History document. (c) Biography document.
Figure 3.1: Examples of three types of documents in which temporal information occurs frequently. Sources: (a): http://www.bloomberg.com/.
(b): http://en.wikipedia.org/wiki/Soviet_war_in_Afghanistan. (c): http://en.wikipedia.org/wiki/Alexander_von_Humboldt.
Independently of their specific goal, all applications using temporal information mentioned in text documents rely on high quality temporal taggers. Due to its importance for many tasks, temporal tagging has become an active research field over the past few years. This resulted in the development of standards for temporal annotation, the creation of annotated corpora, as well as several competitions, which were organized to evaluate temporal taggers. These topics are surveyed in the following section together with an overview of state-of-the-art approaches to temporal tagging and existing temporal taggers.