Mention clustering models

Mention clustering models recast coreference resolution as an incremental clustering task instead of modelling it as a binary classification problem. One of the first of such approaches was presented in Cardie et al. (1999). The approach of Cardie et al. first put all mentions in their own coreference set, i.e. they were treated as singletons. Working in the reversed text direction, mentions were compared to preceding ones and incrementally clustered into coreference sets w.r.t. a distance metric based on features commonly used in coreference resolution.

The approach featured one crucial operation. When two clusters (each containing one or more mentions) were considered for merging, compatibility between all the mentions in both clusters was asserted. If two mentions from the two clusters did not agree in e.g. number and gender, the merge was prevented. Doing so, the problem of contradicting morphological properties (but also semantic properties, such as animacy) in coreference chains, as present in the mention-pair model, was avoided.

In evaluation, the approach yielded a ranking in the middle field compared to contempo- rary approaches, despite its simple feature set. However, Cardie et al. did not compare the model directly to a mention-pair approach in the same experimental setup.

Figure 2.2: _{Modelling coreference as a graph. Vertices denote mentions, and edges} denote potential pair-wise coreference relations and their weights. The circled entity

clusters indicate gold coreference clusters. Example due to Culotta et al. (2007).

Another model of mention clustering is presented in graph partitioning approaches to coreference (Nicolae and Nicolae, 2006, Culotta et al., 2007, Cai and Strube, 2010a, inter alia). Mentions are stored as vertices (i.e. nodes), and edges between them signify potential coreference relations. Initially all mentions are connected through edges. The edges carry weights based on the binary and unary features of the connected mention pairs, as shown in figure 2.2. A graph cut algorithm then cuts edges based on their weights to extract the coreference partition from the graph. In figure 2.2, a cut algorithm would ideally cut all edges which connect the two circled clusters. The stopping criterion

Chapter 2. Discourse processing models for coreference resolution 24

for the cut algorithm is determined empirically with the help of machine learning. Once the cutting algorithm has stopped, the established subgraphs or vertex clusters denote coreference sets.

Nicolae and Nicolae (2006) applied the BestCut method to remove edges from the graph and achieved state-of-the-art results in evaluation. However, they needed to treat pro- nouns separately, i.e. pronouns were not included in the graph, but attached according to the highest ranked pair-wise decision. Also, Nicolae and Nicolae did not include any cluster-level features, unlike e.g. Cardie et al. (1999). The approach of Nicolae and Nicolae did however outperform a mention-pair baseline and the Luo et al. (2004) entity-mention system.4

Cai and Strube (2010a) extended this approach by introducing hyperedges. Hyperedges denote features spanning (possibly) multiple mentions (i.e. nodes in the graph). For ex- ample, a hyperedge denoting head string match connects all mention nodes in the graph whose heads match (e.g. [US President Barack Obama - Barack Obama - Obama] ). Spectral clustering was applied to cluster these subhypergraphs formed by the hyper- edges into coreference clusters. However, like Nicolae and Nicolae (2006), the approach did not feature any means to enforce consistency regarding gender etc. in the emerging coreference clusters, and Cai and Strube noted that they found such inconsistencies in the coreference chains in their system output. Despite this drawback, the system out- performed two strong mention-pair baselines and ranked among the top systems in the CoNLL 2011 shared task (Pradhan et al., 2011).

Culotta et al. (2007) applied first-order logic to capture cluster- (or subgraph-)level features of arbitrary clusters of potentially coreferring mentions. First, a feature encoded how many of the mention pairs (All, Most-True, Most-False) in an arbitrarily generated cluster shared a particular feature (such as a WordNet class or gender agreement). A second feature encoded how many of the mention pairs in the cluster were coreferent (All-True, Most-True, Most-False), as well as whether the maximum and minimum pair- wise scores were above a given threshold. Additionally, cluster size and the distribution of mention types were used as features. The latter is particularly interesting, since it tries to model regularities in mention type distributions in coreference sets, which Culotta et al. speculated would help prevent the formation of sets comprised of pronouns only. Learning and inference then aimed at finding relations between feature sharing and cluster purity (e.g. Most-True for feature sharing and coreference). This approach outperformed a mention-pair baseline in evaluation by large margins. However, Culotta et al. did not evaluate the impact of the cluster-level features separately.

Chapter 2. Discourse processing models for coreference resolution 25

An advantage of the clustering- and graph-based approaches over the mention-pair model is that they perform coreference resolution in a single step, i.e. once the cutting or clus- tering algorithm has finished, the coreference partition is established. Furthermore, graph-based models consider multiple coreference links at once, instead of processing individual pairs of markables in isolation. The cut algorithm also adheres to the transi- tivity and exclusiveness restrictions of the coreference relations, as it places a vertex in only one subgraph or cluster. Clustering approaches also feature the benefit of having access to all mentions of the incrementally established cluster to determine compatibility with an anaphor at hand, which ameliorates underspecification of certain mentions. A related approach to clustering which avoids inconsistencies in coreference sets enforces global constraints during the merging of classified pair instances in the mention-pair model. An additional layer, e.g. Integer Linear Programming (ILP), guides the pair merging step and ensures transitivity and exclusiveness (Finkel and Manning, 2008, Klenner and Ailloud, 2008, Denis and Baldridge, 2009, Klenner and Ailloud, 2009). The weights of the pair-wise decisions serve as input, and the ILP layer optimizes the clustering given the global coreference constraints. This has been shown to improve per- formance, but requires considerable engineering and computational effort. Additionally, the approach has the drawback of still relying on the pair generation mechanics of the mention-pair model.