Analysis of Confusion Errors

In the previous section, we have seen that better morphological information leads to improved parsing accuracy for case-marked functions. Clearly, the parser utilizes the mor- phological information to better model the syntactic structures annotated in its training data. We now go one step deeper and look at confusion errors that the parser commits during parsing. Our hypothesis states that the mistakes that the parser makes are (among other reasons) due to errors in the morphological information which are caused by syn- cretic word forms that the morphological analyzer cannot resolve properly. If this is true, then one should be able to predict the errors based on the ambiguity in the morphosyntac- tic systems of the language. For example, with automatically predicted case values, we would expect the parser to confuse subjects and accusative objects when parsing Czech and German, since there exist regular syncretisms between these functions. When using gold-standard information, we would expect the parser only to confuse functions that

62 4 Error Propagation between Morphology and Syntax

are marked by the same case value, e.g., subjects and nominal predicates. Conversely, we would expect the models without morphological information to commit confusion errors all over the place since they do not have the necessary information.

Subjects. To start with the last expectation, we examine the confusion errors with sub- jects made when using no morphological information (NO-M). Subjects are marked by

nominative case in all three languages, with Czech allowing for dative and genitive sub- jects under special circumstances. TheNO-Mmodels do not have access to morphological information and should therefore mix up functions regardless of the case value that would usually distinguish them. Table 4.6 shows the top five confusion errors made by theNO-M

models on the subject function. The values are split for correct and incorrect head selection to tease apart simple label classification errors from errors involving label classification and attachment.

correct head wrong head rank label freq label freq

1 Obj4 4996 Atr 2644 2 Pnom 1261 Obj4 981 3 Adv 811 Sb M 948 4 Obj3 752 Adv 273 5 Obj7 380 Obj M 245 (a)Czech.

correct head wrong head rank label freq label freq

1 OA 2680 OA 1498 2 PD 776 NK 906 3 DA 458 DA 431 4 EP 301 AG 313 5 MO 219 CJ 296 (b)German.

correct head wrong head rank label freq label freq

1 OBL 3029 ATT 1116 2 OBJ 1505 Exd 574 3 PRED 250 COORD 313 4 ATT 185 OBL 311 5 DAT 152 OBJ 139 (c)Hungarian.

Table 4.6:Top 5 functions with which subjects were confused when parsing withNO-Mmodels.

M marks a coordinated function in Czech.

As expected, the results in Table 4.6 show confusion errors with functions that are marked with different case values: For Czech, when the head was chosen correctly, Obj4, Obj3,

4.3 Error Propagation 63

and Obj7 (accusative, dative, and instrumental objects respectively) are all marked by a different case value and their confusion rates follow their frequency in the data. Pnom (nominal predicates) are marked by nominative case like subjects. If the head was chosen incorrectly, the parser assigns Obj4 and coordinated subjects and objects (Sb M, Obj M). Adverbial (Adv) and attributive functions (Atr) are expected as they mark adjunct functions that can be filled by nominal elements. For German, we see confusions with the object functions (accusative OA and dative objects DA), predicates (PD) and the EP function marking expletive pronouns in subject position. Furthermore, the parser confuses subjects with MO, NK, and AG, which are the three adjunct functions that can be filled by nominal elements (e. g. AG marks genitive adjuncts). CJ finally marks coordinated elements, which is an expected error if the head was chosen incorrectly, but unlike in the Czech treebank, we cannot tell by the coordination label the particular function the element would have if it were not coordinated. In Hungarian, we also find errors across the board, with argument functions not marked by nominative case (accusative objects OBJ, dative objects DAT), the predicate function PRED, and all types of adjuncts (ATT (attributives) and OBL (obliques)). Obliques are especially interesting in Hungarian since the language has only a small number of prepositions. Most oblique adjunct functions are realized by a particular case (hence the about 20 different case values) which for a parsing model using no morphological information makes it rather difficult to distinguish them from the core argument functions. In summary, we find the expected picture of confusion errors across the case paradigm.

Turning now to theGOLD-M models, we can see whether the parser is able to learn the mapping between case and its associated functions. If so, we expect confusion errors with functions that are all compatible with the case value of the correct function. Table 4.7 shows the top five confusion errors that theGOLD-Mmodels make on the subject function. Here, we see a completely different picture compared to theNO-M model errors in Table 4.6. In all three languages, we find — regardless if the head is correct or not — confusions only with functions that are compatible with the nominative case. In Czech, subjects are mostly confused with predicates (Pnom) and coordinated subjects (Sb M). ExD marks suspended nodes in an elliptical construction. The label does not tell whether the node would be a subject with regard to the empty node but it may be, so it is compatible with nominative case. Atr between nominal elements may mark close appositions, which would be marked as nominative by default. ObjX marks objects with no annotated case value (mostly for foreign words). Of all the functions, only Obj4 cannot be signaled by nominative case. If one checks those 69 cases, only 22 are annotated with accusative case

64 4 Error Propagation between Morphology and Syntax

correct head wrong head rank label freq label freq

1 Pnom 583 Sb M 1142 2 ObjX 102 Atr 711 3 Adv 102 ExD M 162 4 Obj4 69 ExD 145 5 ExD 45 Pnom 65 (a)Czech.

correct head wrong head rank label freq label freq

1 PD 773 NK 555 2 EP 323 CJ 245 3 MO 117 PNC 139 4 OA 112 PD 129 5 PH 96 APP 127 (b)German.

correct head wrong head rank label freq label freq

1 PRED 264 ATT 678 2 Exd 102 Exd 494 3 OBL 94 COORD 249 4 ATT 90 NE 32 5 OBJ 50 DET 22 (c)Hungarian.

Table 4.7: Top 5 functions with which subjects were confused when parsing with the GOLD-M

models. M marks a coordinated function in Czech.

in the gold standard, the rest consists mostly of various, high-frequent numerals in neuter gender and quantifiers, most of which are ambiguous between nominative and accusative. In these cases, the case feature seems to be overruled by other information. We get the same picture for German and Hungarian, both models making errors that are compatible with the nominative case value. Of the 112 errors with accusative objects (OA) in German, only 36 have the correct case value in the gold standard. We thus conclude that the parser learns that subjects are marked by nominative case.

Direct Objects (Accusative). We now switch to accusative objects and compare the performance of theGOLD-Mmodels with their respectivePRED-Mcounterparts. We make

the switch to show results on another function and not just on subjects. The basic pattern is the same in both cases. Table 4.8 shows the confusion errors for accusative objects. On the left, theGOLD-Merrors are shown, thePRED-Merrors are displayed on the right. For