Criticism

The traditional parser evaluation remains very popular up to the present day, because of the above mentioned advantages. However, it is not sufficient for numerous reasons.

First of all, it covers only one aspect of parser quality, namely the ac- curacy with which it is able to replicate some gold standard. Some other

important properties like efficiency, size of the data required to train the system and the usability of the system have not been a subject to eval- uation. The efficiency evaluation would comprise the assessment of the training and parsing speed of a parser. The size of the required data is especially important for multilingual parsers working with resource-poor languages. Usability of a system involves the availability of preprocess- ing tools, different possibilities for the output formats, facility of feature model specifications, as well as the hardware requirements.

Additionally, as far as the accuracy evaluation is concerned, the tra- dition evaluation is not perfect neither. In fact there are numerous ar- guments why this kind of evaluation does not truly reflect the quality of parser’s results:

• the parsers are tuned to a specific treebank, which might impair their performance for other domains, because they are overfit to some data and lose their ability to generalise.

• the scores do not differentiate between various dependency rela- tions of different importance.

• the scores are high (around 90%), suggesting that the results are very good, however, when used in applications, parser errors are omnipresent.

As far as another major advantage, namely the facility of parser comparison, is concerned the methodology also has some drawbacks:

• treebanks contain errors and a higher score when replicating the treebank might mean that there is not real improvement, but merely the errors are replicated.

• there are different annotation schemes, which vary in granularity. Some are easier to produce and some others are more difficult. E.g. the Stanford scheme is slightly easier than the CoNLL scheme.

• one parser might be better at more important and difficult rela- tions, but they contribute equally towards the score as some other easier and less relevant dependencies.

Therefore a parser with a higher score not necessarily is better and a more detailed analysis is necessary.

3.1.3

Modified Evaluation

The disadvantages of the usual attachment scores practice lead to the necessity of a modified evaluation:

1. The problem of attachment scores being the average over many different relation types must be addressed.

2. The missing 10%, which can not be replicated, must be analysed and compared in order to determine whether they are common for different systems or whether the parsers have difficulties with various pieces of data.

3. The grade of tuning and overfitting of systems to a particular tree- bank must be assessed.

The first problem is well-known and there are numerous works deal- ing with it. The proposed solution is to define a set of relations relevant for one’s needs and to compute attachment scores only over these depen- dencies. Examples of such work are the evaluation of non-local (long- distance dependencies, non-dependencies, phrasal modifiers and subtle arguments) (Bender et al., 2011) and unbounded dependencies (object extraction from a relative clause, object extraction from a reduced rela- tive clause, subject extraction from a relative clause, free relatives, object question , right node raising and subject extraction from an embedded clause) (Nivre et al., 2010). Both papers provide a comprehensive moti- vation why exactly these types of dependencies are particularly impor- tant.

However, the two above-mentioned categories of dependencies are not relevant for one’s needs in general. It is completely dependent on one’s application what kind of relations are important and it can also be true that the simplest ones are actually the relevant ones. Therefore to my mind no ultimate set of useful relations can be defined and that is why I was rather interested in developing a methodology of determining the set of relevant dependencies given a certain NLP task.

The notion of relevant or necessary dependencies has also already been discussed in several works. The basis for these works is the book by Anderson et al.: Entailment. The logic of relevance and necessity (Anderson et al., 1992). The general idea is that given some logic for- mulae and a task, relevant parts are those shared by the formulae, e.g. atomic units like variables and constants, whereas the necessary parts are those actually required to solve the task. E.g. for argument validation,

the information contained in the premises should also occur in the con- clusion in order to be relevant, but only some of such premises provide (or take away) support and are necessary in order to decide the valid- ity. The same idea can be transferred to the information-based textual tasks: the relevant fragments are those which share the same informa- tion, whereas the necessary ones are those which are actually required for task purposes. E.g. in the shared task PETE (Parser Evaluation using Textual Entailments) (Yuret et al., 2010) a necessary dependency is a dependency whose correct recognition is a requirement in order to correctly predict the entailment relation between two texts.

The exact distinction between relevant and necessary dependencies in a certain application is difficult and depends on how they are used in order to solve the actual task. However, it is obvious that at least as far as the evaluation of dependency parsing is concerned, it is very sensible to discard the irrelevant dependencies. Even though a further distinction between necessary dependencies and relevant ones would be a plus, the first step is already a big improvement.

The second point has also been addressed by several works. Es- pecially the works on the integration of the transition-based and graph based approaches, as e.g. (Nivre and McDonald, 2008), discuss the differ- ences between the results of various systems. These analyses have shown that different parsers perform differently for various parts of speech, de- pendency types and certain sentence constructions and that is why ba- sically an integration of both makes sense. The Ensemble (Surdeanu and Manning, 2010) system shows that even within one approach differ- ent parsing algorithms lead to different performance and that combining their strengths significantly improves the overall result. There is also an interesting article on Deniz Yuret’s webblog (Yuret)1_{on the missing}

10% problem. His belief is that a certain amount of the errors is due to inter-annotator disagreement, but does not perform an actual study.

The third point includes the efforts to study the problem of domain adaptivity for dependency parsing. The CoNLL 2007 shared task in- cluded two tracks: the multilingual track, which is the ordinary track where the training and test data for different languages were from the same resource, and the domain adaptation track, where the test data was different from what the parser was trained on. In terms of the domain adaptation track numerous works on the topic of domain adaptivity and overfitting were published. The finding was that the performance of

1_{http://denizyuret.blogspot.de/2010/11/next-generation-parser-evaluation.html,}

the systems significantly dropped when they were applied to the out-of- domain data. The best approaches of that shared task, including those who achieved the top scores (Dredze et al., 2007)(Sagae, 2007), used the knowledge about the new domain in order to improve the original model. However, it would be better to have a model which is as general as pos- sible in the first place before adapting it to a different domain and this point has not been investigated. To my mind it is due to the fact that the community is not very aware of a considerable amount of erroneous annotations in the treebanks, which I have already broadly discussed in the Part I of the thesis. The overfitting is therefore dangerous, because a model tuned to such a treebank, might be simply replicating these errors.

This concludes the overview of the traditional evaluation method- ology and the newer developments which are similar in nature. I will present my work in this field in section 3.3, but first I have to introduce the extrinsic evaluation methodology, which is on the other side of the spectrum.