Aside from experimental work, these language resources were generated in the course of this PhD.
A.2.1
CAVaT
CAVaT is an open source, modular checking utility for statistical analysis of features specific to temporally-annotated natural language corpora (Derczynski and Gaizauskas,2010a). It provides reporting, highlights salient links between a variety of general and time-specific linguistic features, and also validates a temporal annotation to ensure that it is logically consistent and sufficiently annotated. Uniquely, CAVaT provides analysis specific to TimeML-annotated temporal informa- tion. CAVaT includes an API for loading TimeML documents into a portable database format, a command line query interface for interrogating and summarising TimeML data and a set of temporal consistency and evaluation checking tools.
A.2.2
RTMML
RTMML is a markup language for the tenses of verbs and temporal relations between verbs. There is a richness to tense in language that is not fully captured by existing temporal annotation schemata. Following Reichenbach we present an analysis of tense in terms of abstract time points, with the aim of supporting automated processing of tense and temporal relations in language. This allows for precise reasoning about tense in documents, and the deduction of temporal relations between the times and verbal events in a discourse. RTMML differs from TimeML (Pustejovsky et al., 2004) in that (1) it chiefly only annotates verbs that indicate events, (2) the information annotated about verbs is more nuanced, and (3) inter-verb links are defined using Reichenbach’s three abstract points instead of event boundaries. RTMML has a syntax that can be adopted as an extension to TimeML. See also Section6.6andDerczynski and Gaizauskas (2011b).
A.2.3
TB-sig
TimeBank contains some signal annotations over text that describes the nature of a temporal link. However, these annotations are incomplete. TB-sig is a hand-curated version of TimeBank with improved signal annotations, adding extra timexes, events and temporal links where applicable. See also Section5.5.
A.2. LANGUAGE RESOURCES 163
A.2.4
TempEval-2 analysis
TempEval-2 saw the comparison of diverse approaches to temporal link labelling over a fixed corpus and in a tightly controlled environment. This thesis includes an extensive formal analysis of the errors made by almost all the participants in the task, constituting the largest most recent survey of temporal link labelling efforts and difficulties. This is given in Section4.2.
A.2.5
TIMEN
Automatically annotating temporal expressions is a research goal of increasing interest. Recog- nising them can be achieved with supervised machine learning, but interpreting them accurately (normalisation) is a complex task requiring human knowledge. TIMEN (Llorens et al.,2012a) is a community-driven tool for temporal expression normalisation. TIMEN is derived from current best approaches and is an independent tool, enabling easy integration in existing systems. It is argued that temporal expression normalisation can only be effectively performed with a large knowledge base and set of rules. Our solution is a framework and system with which to capture this knowledge for different languages.
A.2.6
T2T3 v.2
Saquete and Pustejovsky (2011) describe a technique for converting TIMEX2 to TIMEX3 an-
notations and present the T2T3 tool as an implementation of it. As some things annotated as TIMEX2s were no longer considered parts of temporal expressions in TimeML and instead as- signed to other functions, T2T3 generates not only TIMEX3s but also extra TimeML elements. We upgraded the tool to better process complex TIMEX2s, using resources including temporal signals and mapping of Reichenbach’s reference point (seeDerczynski et al.(2012)).
A.2.7
TIMEX3 extended corpora
Applying T2T3 to earlier TIMEX2 corpora gave a 6x increase in the number of available TIMEX3s over the sum total of prior TIMEX3 resources. This extended dataset was useful for training state- of-the-art timex recognition tools and generated improved recognition accuracy. Timexes in the generated corpora were difficult to recognise using just existing TIMEX3 data, perhaps due to the limited variation of expression in previous newswire-only TIMEX3 data (Derczynski et al.,2012).
A.2.8
TempEval-3
Temporal annotation is a time-consuming task for humans, which has limited the size of annotated data in previous TempEval exercises. Current systems, however, are performing close to the inter- annotator reliability for entity recognition. This suggests that larger corpora could be built from automatically annotated data with minor human reviews. As part of TempEval-3, we explore whether there is value in adding a large automatically created silver standard to a hand-crafted gold standard. TempEval-3 differs from its ancestors in the following respects:
(i) size of the corpus: the dataset used comprises about 500K tokens of silver standard data and about 100K tokens of gold standard data for training, compared to the corpus of roughly 50K tokens corpus used in TempEval 1 and 2;
(ii) temporal relation task: the temporal relation classification tasks are to be performed from raw text, i.e. participants need to extract events and temporal expressions first, determine which ones to link and then obtain the relation types;
(iii) tasks not independent: participants must annotate temporal expressions and events in order to do the relation task;
(iv) temporal relation types: the full set of temporal interval relations in TimeML is used, rather than the reduced set used in earlier TempEvals;
(v) annotation: most of the corpus was automatically annotated by the state-of-the-art systems from TempEval-2, a portion of the corpus, including the test dataset, that is human reviewed;
(vi) evaluation: we will report a temporal awareness score for evaluating temporal relations, to help to rank systems with a single score.
This will be the largest temporal link labeling exercise to date, in terms of datasets available; see alsoUzZaman et al.(2012).
Appendix B
Annotated Corpora and
Annotation Tools
B.1
Introduction
TimeML is a standard for annotating time in natural language. It includes annotations for the lexicalised entities TIMEX3, EVENT and SIGNAL, and for the abstract entities TLINK, SLINK, ALINK and MAKEINSTANCE. The syntax is XML-like, with inline annotation. For the temporal link labelling task, one is interested in TIMEX3, EVENT, SIGNAL and TLINK. The MAKEIN- STANCE tag gives events extra information and instantiates them for use in TLINKs, and so also contains useful information. TimeML has recently become a non-free ISO standard ISO-TimeML, which incorporates a few changes to event description and permits stand-off annotation. As almost all prior work and all existing resources use TimeML or an extension thereof, this thesis considers only TimeML in general.