In this section a set of novel simple linguistic-based metrics based on detailed Part-of-Speech (POS) tags is described and evaluated. Although the idea of using the POSBLEU score was men- tioned several years ago,1 none of the experimental results in this direction have been reported yet in the literature. The following metrics are investigated in this work:
• POSBLEU
The standard BLEU score calculated on the POS tags instead of words; • POSWER
The standard word error rate calculated on the POStags instead of words; • POSR4GRAM
Recall measure based on POS-n-grams, n∈{1,...,4}: percentage of n-grams in the refer-
ence which are also present in the hypothesis; • POSP4GRAM
POS-n-gram precision: percentage of n-grams in the hypothesis which have a counterpart in the reference;
• POSF4GRAM
POS-n-gram-based F-measure: takes into account all n-grams which have a counterpart, both in the reference and in the hypothesis.
For the n-gram-measures, two types of n-gram averaging are investigated: geometric mean and aritmetic mean. Geometric mean is already widely used in the BLEU score, but is also argued not to be optimal because the score becomes equal to zero even if only one of n-gram counts is equal to zero.
All evaluation metrics are based on detailed POS tags. The prerequisite is availability of the appropriate POS tagger for the target language. It should be noted that the POStags cannot be only basic (noun, verb, etc.) but must contain all morpho-syntactic details (e.g. verb tenses, cases, number, gender, etc.).
B.2.1 Evaluation set-up
The results are presented on the English, French, Spanish and German European Parliament texts generated by different translation systems in the framework of the shared task on the 2006 HLT-NAACL Workshop [Koehn & Monz 06] and 2007 ACL Workshop on Statistical Machine Translation [Callison-Burch & Fordyce+ 07]. The objective of the shared task was translation between European languages, namely French, German, Spanish and English. The translation directions were from each of the languages into English, and vice versa. Training and testing
B.2 Syntax-oriented evaluation measures
was based on the EUROPARL corpus. In addition, editorials from the Project Syndicate web- site2 were collected and used as out-of-domain test data. The corpus statistics of all test data is shown in Tables B.1 and B.2. About fifteen different groups from different institutions par- ticipated in the share tasks. Most of the groups use some variant of a phrase-based statistical system. However, SYSTRAN uses a rule-based system which is not task-specific, and in the evaluation 2007 SYSTRAN and NRC submitted joint translation outputs from a hybrid system using both rule-based and statistic approaches. More details about the shared tasks, the data and the participants can be found in [Koehn & Monz 06] and [Callison-Burch & Fordyce+ 07].
Table B.1: Test data for the shared task 2006.
English Spanish French German
EUROPARL Sentences 2 000 Words 59 307 61 824 66 783 55 533 Distinct words 6 031 7 719 7 230 8 812 out-of-domain Sentences 2 000 Words 59 307 61 824 66 783 55 533 Distinct words 6 031 7 719 7 230 8 812
Table B.2: Test data for the shared task 2007.
English Spanish French German
EUROPARL Sentences 2 000 Words 53 531 55 380 53 981 49 259 Distinct words 8 558 10 451 10 186 11 106 out-of-domain Sentences 2 007 Words 43 767 50 771 49 820 45 075 Distinct words 10 002 10 948 11 244 12 322
The new metrics are evaluated on outputs from all translation directions: Spanish, French and German into English and vice versa. Morpho-syntactic annotation of the English and German references and hypotheses is performed using the constraint grammar parser EN- GCG [Voutilainen 95] and GERCG [Haapalainen & Majorin 95]. Spanish texts are annotated using the FreeLing analyser [Carreras & Chao+ 04], and French texts usin the Tree Tagger3. In this way, all references and hypotheses are provided with detailed POS tags.
Correlation with human judgments: Spearman’s rank correlation coefficientρ is used in the officional evaluation of various metrics in the share tasks. Therefore we calculate the same coefficient in order to measure correlation of the new automatic metrics with adequacy and fluency scores. Spearman’s rank correlation coefficient is equivalent to Pearson correlation on ranks, and its advantage is that it makes fewer assumptions about the data. The possible values ofρ are between 1 (if all systems are ranked in the same order) and -1 (if all systems are ranked
2http://www.project-syndicate.com/
in the reverse order). Thus the higher value ofρ for an automatic metric, the more similar it is to the human metric.
B.2.2 Results
For each new metric, the Spearman ρ coefficient with the adequacy and with the fluency score is calculated on the document level. Then the results are summarised by averaging the obtained coefficients over all translation outputs. Both the mean and the median value are calculated. Median value reflects better how often the certain correlation is above or below this value. Average correlations are calculated separately for the 2006 and 2007 data as for the 2007 data the TER and METEOR scores are also available for comparison whereas for the 2006 data only the BLEUscore is available.
Average correlations are shown in Table B.3 and Table B.4. In Table B.3 it can be seen that the new measures have a rather high ρ coefficient both with respect to the adequacy and to the fluency score. POSBLEU and POSF4GRAM using the geometric mean have especially high correlation coefficients, followed by POSWER and other n-gram-based metrics. Furthermore, all these measures outperform the BLEU score. It can also be seen that the geometric mean averaging of n-grams performs slightly better than the arithmetic mean.
Table B.3: Mean and median Spearman correlations for the 2006 data.
2006 adequacy fluency
mean median mean median BLEU 0.478 0.660 0.495 0.624 POSBLEU 0.546 0.727 0.541 0.645 POSWER 0.729 0.708 0.659 0.698 POSF4GRAM gm 0.597 0.788 0.566 0.739 am 0.577 0.751 0.588 0.740 POSR4GRAM gm 0.508 0.652 0.525 0.612 am 0.472 0.623 0.494 0.522 POSP4GRAM gm 0.602 0.712 0.532 0.654 am 0.569 0.610 0.500 0.554
Average correlation coefficients for the 2007 data are presented in Table B.4. Again, POS- BLEU, POSWER and POSF4GRAM have high correlation coefficients with both adequacy and fluency, and for this data POSR4GRAMtoo. These measures outperform the BLEUscore as well as the METEORand the TER metric.
Results of an additional experiment on the 2007 data are shown in Table B.5. This table presents the percentage of the documents on which the new measure outperforms one of the well-known measures, i.e. BLEU, METEOR and TER. It can be seen that in the majority of the cases the POSBLEU metric outperforms all three standard measures, especially with re- spect to the fluency score. The POSWER and geometric mean POSF4GRAM show a similar behaviour, outperforming the standard measures in the majority of cases but slightly less often than the POSBLEU. The POSR4GRAM score is worse than POSF4GRAM, but still outperforms
B.2 Syntax-oriented evaluation measures
Table B.4: Mean and median Spearman correlations for the 2007 data.
2007 adequacy fluency
mean median mean median BLEU 0.674 0.669 0.584 0.710 METEOR 0.596 0.604 0.538 0.643 TER 0.598 0.68 0.479 0.59 POSBLEU 0.723 0.742 0.697 0.778 POSWER 0.676 0.726 0.651 0.742 POSF4GRAM gm 0.576 0.693 0.538 0.736 am 0.590 0.693 0.556 0.736 POSR4GRAM gm 0.626 0.714 0.618 0.804 am 0.600 0.732 0.586 0.827 POSP4GRAM gm 0.508 0.642 0.438 0.572 am 0.500 0.607 0.429 0.572
Table B.5: Percentage of documents where a particular new measure outperforms the standard measures – test data 2007.
adequacy fluency
BLEU METEOR TER BLEU METEOR TER
POSBLEU 77.3 58.3 75.0 81.8 83.3 83.3 POSWER 63.6 66.7 75.0 63.6 75.0 83.3 POSF4GRAMgm 72.7 58.3 75.0 63.6 75.0 83.3 am 68.2 58.3 75.0 63.6 66.7 66.7 POSR4GRAMgm 63.6 75.0 58.3 68.1 66.7 58.3 am 54.5 75.0 58.3 63.6 58.3 50.0 POSP4GRAMgm 63.6 50.0 75.0 45.4 50.0 58.3 am 54.5 41.7 66.7 36.4 50.0 58.3
the standard measures in 50-70% of cases, and the POSP4GRAM score has the lowest percent- age, 30-60%. Additionaly, it can be seen that, again, the geometric mean averaging of the
n-grams performs better than the artimetic mean.
B.2.3 Conclusions
The use of lingustic-based evaluation measures oriented on the syntactic structure of the sen- tence is proposed. BLEU and WER are calculated on the detailed Part-of-Speech (POS) tags instead on the words. In addition, precision, recall and F-measure obtained on POS-n-grams are investigated. The new measures are tested on the data of the second and third shared task of Sta- tistical Machine Translation Workshop [Koehn & Monz 06, Callison-Burch & Fordyce+07]. An extensive analysis of the correlation coefficients between the linguistic-based automatic evaluation metrics and the human judgments is carried out. The obtained results show that the
new metrics correlate very well with human judgments, namely adequacy and fluency scores, as well as that some of the new metrics outperform the standard evaluation measures BLEU, METEOR and TER in the majority of cases. POSBLEU, geometric mean POSF4GRAM and POSWER seem to be especially promising.
Combinations of different evaluation metrics could be investigated in the future, for example the combination of different linguistic-based metric as well as the combination of linguistic metrics with standard metrics.