5.5 Error analysis
5.5.2 Error examples
In this section, we investigate a set of hand-selected examples of pronoun resolution errors that the models produce. Although we do not provide an extensive error analysis and we do not present a systematic error categorization, we pick examples that subsume
Chapter 5. Empirical validation of our entity-mention model 128
several similar errors we encountered during manual error tracking. We limit our analysis to the wrong linkage class, since false positive and false negative errors are due to preprocessing errors, which we do not cover here.
The first example shows an instance of a possessive pronoun that the entity-mention model correctly resolves which the mention-pair model incorrectly handles.
(6) Sie1 denkt an den Tag danach, wenn die KollegInnen wom¨oglich ihr1 Bild in der
Zeitung sehen.
She thinks about the day after, when the colleagues perhaps see her/their1 pic-
ture in the newspaper.
In this example, the instance of the possessive pronoun [ihr]1 is genuinely ambigu-
ous. That is, without discourse context, the pronoun can be linked to either [Sie]1 or
[KollegInnen], and both resolutions yield semantically valid utterances. However, the previous sentences in this context have only mentioned the entity denoted by [Sie]1,
but not [KollegInnen], i.e. [KollegInnen] is a discourse-new entity, while [Sie]1 de-
notes a discourse-old entity this context. Due to its local confinement, the mention-pair model has no notion of information status of entities and NPs denoting them. Therefore, the mention-pair model incorrectly attaches [ihr]1 to [KollegInnen] based on proxim-
ity preferences. The entity-mention model, which incorporates the discourse-new and discourse-old distinction, correctly links it to [Sie]1 based on its bias towards favoring
discourse-old entities as antecedents.
The next example shows a relative pronoun which both models incorrectly resolve.
(7) Hier werden Beitr¨age1 kleiner Leute veraast, die1 von ehrenamtlichen Kassierern
f¨unf Mark weise gesammelt werden.
Donations1of ordinary people are being wasted, that1are collected by volunteers,
5 Mark per donation.
Here, the strategy to select the most recent compatible antecedent candidate for relative pronouns fails. Both models resolve [die1] to [Leute]. To correct this error, the models
would need a notion of semantic compatibility to infer that volunteers generally collect donations rather than ordinary people. However, a simple model of verb selectional preferences might not suffice, since people might be a relevant direct object of “sammeln” in such a model. The discriminatory factor here is the adverbial phrase “5 Mark a piece” which modifies the verb, since it is unlikely that one collects or gathers people at a 5 Mark rate.
Chapter 5. Empirical validation of our entity-mention model 129
In the next example, the models fail because the pronoun occurs in direct speech and refers to an antecedent in a previous direct speech segment.
(8) Endlich kommt einer und fragt, warum der Mann1 nicht sein zweites Bein be-
nutzt. “Warum wohl?”, fragt Kilian das Publikum und gibt sich selbst die Antwort. “Er1 hatte es einfach nicht bemerkt.”
Finally, someone approaches and asks why the man1 does not use his second leg.
“Why?”, Kilian asks the audience and gives the answer himself. “He1 simply
had not noticed it.”
Both models here resolve [Er1] to [Killian] because given our feature set, it is a more
probable antecedent when direct speech is disregarded. Clearly, our approach needs an elaborate strategy to disentangle direct speech segments from other parts in the discourse to resolve such pronoun instances.
Finally, the following example in figure 5.7 illustrates the major differences of the models by looking at a longer discourse segment, i.e. beyond pair-wise decisions.41
We use the subscript to indicate entity IDs and superscript to enumerate mentions, i.e. lexemementionID
entityID . Thus, lexemes with identical entity IDs are coreferent. ’*’ in the
responses denotes that mentions or entities are invented or overlooked. Table 5.23 lists the coreference chains of the key and the two responses.
Key: [Jusef1, Er2, er3, seine4, seinen5, Jusef9, seine10, ich11, meiner12, meiner13, Jusef14] 1
[V ater6, ihn7, er8] 2
E-M: [Jusef1, Er2, er3, seine4, seinen5, ihn7, er8, Jusef9, seine10, Jusef14] 1
[ich11, meiner12, meiner13] 2
M-P: [Jusef1, Er2, er3, seine4, Jusef9, seine10, Jusef14] 1
[Regime∗, seinen5, ihn7, er8] 2
[ich11, meiner12, meiner13] 3
Table 5.23: Gold and predicted coreference chains for the segment.
We see that the entity-mention model makes two errors. First, it links ihn7 and er8 to the Jusef1 entity instead of to V ater2. Here, the weight for favoring discourse old
entities misguides resolution. Secondly, the model is unable to attach the first person pronouns in the direct speech segment to the Jusef1entity, since we only allow matching
first person pronouns to antecedents governed by a communication verb at most one sentence away. The right mention here would be Jusef114. However, it occurs after the first person pronouns and we do not allow pronouns to link to postcedents.
41English translation: Jusef gets up. He moves his eyes away from the wall and starts to talk. Of Afghanistan, where he and his family led a happy life. Until the new regime came. One day, the Taliban stood at the door, took his father and shot him, because he was a communist, allegedly. The mother thereafter had Jusef and his sister taken out of the country by their uncle. “No I am here and I have no contact with my mother and my family”, Jusef says.
Chapter 5. Empirical validation of our entity-mention model 130
(9) Key:
Da steht Jusef11 auf. Er21 wendet den Blick von der Wand und f¨angt an zu erz¨ahlen. Von Afghanistan, wo er3
1 und seine41 Familie ein gl¨uckliches Leben
f¨uhrten. Bis das neue Regime kam. Eines Tages standen die Taliban vor der T¨ur, nahmen seinen51 Vater62 mit und erschossen ihn72, weil er82 angeblich Kommunist war. Die Mutter ließ daraufhin Jusef91 und seine101 Schwester Abeda vom Onkel außer Landes schaffen. “Jetzt bin ich11
1 hier und habe keinen Kontakt zu meiner121
Mutter, auch nicht zu meiner131 Familie”, sagt Jusef141 . Entity-mention response:
Da steht Jusef11 auf. Er21 wendet den Blick von der Wand und f¨angt an zu erz¨ahlen. Von Afghanistan, wo er31 und seine41 Familie ein gl¨uckliches Leben f¨uhrten. Bis das neue Regime kam. Eines Tages standen die Taliban vor der T¨ur, nahmen seinen51 Vater6∗ mit und erschossen ihn71 *, weil er81 angeblich Kommu- nist war. Die Mutter ließ daraufhin Jusef9
1 und seine101 Schwester Abeda vom
Onkel außer Landes schaffen. “Jetzt bin ich112 * hier und habe keinen Kontakt zu meiner122 Mutter, auch nicht zu meiner132 Familie”, sagt Jusef141 .
Mention-pair response: Da steht Jusef1
1 auf. Er21 wendet den Blick von der Wand und f¨angt an zu
erz¨ahlen. Von Afghanistan, wo er31 und seine41 Familie ein gl¨uckliches Leben f¨uhrten. Bis das neue Regime∗2 kam. Eines Tages standen die Taliban vor der T¨ur, nahmen seinen52*Vater6∗mit und erschossen ihn72, weil er82 angeblich Kom- munist war. Die Mutter ließ daraufhin Jusef91 und seine101 Schwester Abeda vom Onkel außer Landes schaffen. “Jetzt bin ich11
3 *hier und habe keinen Kontakt zu
meiner123 Mutter, auch nicht zu meiner133 Familie”, sagt Jusef141 .
Figure 5.7: Example responses of the models. Incorrectly resolved pronouns are marked by a star (*).
The mention-pair model, on the other hand, makes three resolution errors. It also in- correctly resolves the two personal pronouns (ihn7, er8) that denote the V ater
2 entity.
However, it identifies Regime∗as the antecedent entity. This happens because the model forms the pairs [Regime∗ − seinen5], [seinen5 − ihn7], and [ihn7− er8]. The transi-
tive merge then yields the coreference chain which features inconsistent morphology (i.e. Regime∗ and ihn7, and Regime∗ and er8 are exclusive). Finally, the mention-pair model also fails to link the first person pronouns to the Jusef1 entity, since we apply
Chapter 5. Empirical validation of our entity-mention model 131