Improving Mention Classification

In the work reported so far, the instance class labels predicted from the case-base of examples use a nearest neighbor approach with a variant of the Jaccard distance to retrieve cases and classify new instances. This approach achieves relatively high classification accuracy but we experimented with potential refinements. In this work we improve the classification results by using the previously identified coreference resolution information (see Figure4.4).

Table 4.4: Confusion matrix for predictions in the 15 class labels in our classification process with counts for all the 21 stories using the leave-one-story-out protocol. The two letter labels stand for (from top to bottom): “N/A” for parsing errors, AA: anthropomorphic animal charac- ter, AN: animal (non-character), AO: anthropomorphic object character, FE: female character, GR: group of characters, HA: happening, MA: male character, MB: magical being character, OB: object or prop, PA: part of characters, PO: part of non-characters, SC: scenery that is mentioned, SS: locations that the characters visit, and ST: temporal references. Bold face indi- cates correct predictions (diagonal) and the color gradient illustrates the normalized value over the total count of instances for each class.

N/A AA AN AO FE GR HA MA MB OB PA PO SC SS ST Recall Prec.

N/A 0 24 1 7 8 17 30 37 4 166 11 0 9 150 47 0 0 AA 0 39 1 2 31 10 1 29 13 22 2 0 0 3 5 0.247 0.151 AN 0 4 2 6 0 2 2 8 2 49 0 0 1 3 7 0.023 0.133 AO 0 1 0 0 0 22 1 7 2 20 1 0 0 7 0 0 0 FE 0 14 0 0 510 3 8 9 0 24 0 0 0 17 4 0.866 0.765 GR 0 10 3 34 62 56 9 55 0 120 2 0 0 5 0 0.157 0.308 HA 0 3 2 1 2 2 4 7 1 60 4 0 6 21 10 0.033 0.033 MA 0 72 1 1 30 37 17 799 17 71 3 0 0 69 11 0.708 0.76 MB 0 34 1 9 1 5 0 57 52 58 0 0 21 1 2 0.216 0.433 OB 0 36 3 11 13 13 30 14 26 375 48 0 16 119 50 0.497 0.318 PA 0 5 1 8 7 5 5 4 0 56 33 0 1 16 0 0.234 0.308 PO 0 0 0 0 0 0 0 0 1 2 1 0 1 1 0 0 0 SC 0 8 0 0 0 2 2 1 1 19 0 0 2 21 6 0.032 0.032 SS 0 4 0 4 1 6 9 13 1 94 1 0 4 283 14 0.652 0.387 ST 0 5 0 0 2 2 2 12 0 42 1 0 2 16 57 0.404 0.268

mentions and there exist edged between nodes when two mentions refer to the same entity. It is therefore intuitive to assume that given that there is an assignment of one class label per entity, all the mentions of the same entity should have the same class label. Therefore, we use a majority voting approach among the predictions for each coreference group, that is, given a mentione,Vozidentifies its coreference groupcoref(e), i.e., all the other mentions that are linked toein the coreference graph

G. Then, a final classification is generated by assigning to each mentionethe majority class in the coreference groupcoref(e). For example, if three mentions incoref(e)were labeled ascharacter and

only one as non-character, then all the mentions in coref(e) will be labeled as character. In the

(unusual) case of tie, the class of the earliest occurring mention in the text is chosen.

Once all the mentions have been classified, the output of the coreference resolution is used to refine the results. Given a mention e ∈ E, we identify its coreference group coref(e), i.e., all the other mentions that are linked to e in the coreference graph G. Then, the class assigned to e is replaced by the majority among all the classes of all the mentions in the coreference groupcoref(e).

For example, given a mentionelabeled asnon-character, if two mentions incoref(e)were labeled as

Table 4.5: Effect of coreference information on the majority voting processes. Rows report results without coreference information, using the automatically computed coreference graph and using the coreference from the ground truth. The columns report the accuracy, precision and recall on the binary classification task. Note that the dataset used for this experimental evaluation is larger than the one used in the results reported in Tables4.1and4.2.

Approach Acc. Prec. Rec.

Without Voting 0.86 0.844 0.876 Auto Coref. 0.868 0.859 0.878 GT. Coref. 0.868 0.896 0.839

character and only one asnon-character, then the label assigned toewill be replaced bycharacter.

Experimental Results

For the experimental evaluation we used both a ground truth coreference graph and an automatically extracted coreference graph (from the Stanford CoreNLP). With either approach we observed an improvement on accuracy and f-measure from 0.860 to 0.868. Table 4.5reports detailed accuracy, precision and recall comparison between these approaches. Note that the dataset used for this experimental evaluation is larger than the one used earlier in Section4.3.3.

4.4

A Machine Learning Approach to Identifying Narrative Roles from

Characters

In this section we focus on a method for identifying a particular high-level feature of narrative theory, namely, narrative roles for characters. Narrative roles for characters are a recurrent feature present in several narrative theories (e.g. Propp’s25 _{or the Monomyth}77_{). These roles identify prototypical} character actions for specific narrative roles. Typically, the protagonist of the story takes the role of thehero. Another character roles is that of thevillain that triggers or motivates the heroe’s actions. Let us consider the except shown in Figure4.7. In Propp’s theory, the dragon (character) fulfills the specific function of villain (role). This structure-level narrative information of roles is important to understand the story as well as its relation to others in its domain. However, as the word “villain” or “hero” rarely appears explicitly in the text, extracting the role information requires combining NLP and narrative theory. A key Proppian insight that we use is that each role has a “sphere of action.” It defines the core actions of whatever characters fulfilling that role. For example, no matter

One day, somewhere near Kiev, a dragon appeared, who demanded heavy tribute from the people. He demanded every time to eat a fair maiden: and at last the turn came to the Tsarevna, the princess. But the dragon would not eat her, she was too beautiful. He dragged her into his den and made her his wife. [...]

When she wrote a letter to her father and mother she used to tie it to the neck of her little dog. [...]

The Tsarevna got every day on more intimate terms with her dragon in order to discover who was stronger. At last he owned that Nikita, the tanner at Kiev, was the stronger. [...]

The Tsarevna at once wrote to her father [...] So the Tsar looked for Nikita, and went to him himself to beg him to release the land from the cruelty of the dragon and redeem the princess. [...]

Figure 4.7: An excerpt of a story from our dataset.

whether the villain is a dragon or a wizard, its sphere of action centers on villainy, struggle, and pursuit.

In order to automatically identify a character’s narrative role we rely precisely on the recurring actions and interactions with other characters as described in the narrative theory; Propp’s in our case. In the work described in this section we introduce the idea of representing the “sphere of action” of a character role (their prototypical actions in Propp’s narrative theory) as a matrix encoding interactions indicated by verbs linking characters with different roles167_{. Initially we used} an annotated dataset to compute a matrix from a story and compare it against a reference matrix using Wordnet to find similarities. Then we folded this approach into the automated mention and feature extraction process described earlier and we integrated it with the machine learning approach used to classify characters described in the previous section129_.

In our initial work on modeling narrative character interactions using “spheres of action”, we hypothesize that,given a particular narrative convention, information about how characters behave towards one another can help identify their roles. This is based on our observation of recurring patterns in the relationship between different roles. To test our hypothesis, in this work we used function definitions from Propp’s narrative theory and an annotated dataset of 8 Russian and Slavic folktales to build a knowledge base of common actions and interactions between different character roles.