Why Evaluation? Machine Translation. Evaluation. Evaluation Metrics. Ten Translations of a Chinese Sentence. How good is a given system?

(1)

Machine Translation

Evaluation

Based on Philipp Koehn’s slides from Chapter 8

Why Evaluation?

• How good is a given system?

• Which one is the best system for our purpose? • How much did we improve our system? • How can we tune our system to become better?

But MT evaluation is a difficult problem!

Evaluation of Machine Translation 1

Ten Translations of a Chinese Sentence

Israeli officials are responsible for airport security. Israel is in charge of the security at this airport.

The security work for this airport is the responsibility of the Israel government. Israeli side was in charge of the security of this airport.

Israel is responsible for the airport’s security. Israel is responsible for safety work at this airport. Israel presides over the security of the airport. Israel took charge of the airport security.

The safety of this airport is taken charge of by Israel.

This airport’s security is the responsibility of the Israeli security officials.

(a typical example from the 2001 NIST evaluation set)

Evaluation Metrics

• subjective judgments by human evaluators • automatic evaluation metrics

• task-based evaluation, e.g.: – how much post-editing e↵ort? – does information come across?

(2)

Human vs. Automatic Evaluation

• Human evaluation is

– Ultimately what we are interested in, but – Very time consuming

– Not re-usable • Automatic evaluation is

– Cheap and re-usable, but – Not necessarily reliable

Adequacy and Fluency

• Human judgement

– given: machine translation output – given: source and/or reference translation

– task: assess the quality of the machine translation output • Metrics

Adequacy: Does the output convey the same meaning as the input sentence? Is part of the message lost, added, or distorted?

Fluency: Is the output good fluent English?

This involves both grammatical correctness and idiomatic word choices.

Fluency and Adequacy: Scales

Adequacy Fluency

5 all meaning 5 flawless English

4 most meaning 4 good English

3 much meaning 3 non-native English

2 little meaning 2 disfluent English

1 none 1 incomprehensible

Human Evaluation

• Source: Estos tejidos estan analizados, transformados y congelados antes de ser almacenados en Hema- Quebec, que gestiona tambien el unico banco publico de sangre del cordon umbilical en Quebec.

• Reference: These tissues are analyzed, processed and frozen before being stored at Hema-Quebec, which manages also the only bank of placental blood in Quebec.

• Translation:These weavings are analyzed, transformed and frozen before being stored in Hema-Quebec, that negotiates also the public only bank of blood of the umbilical cord in Quebec.

What is your judgement in terms of adequacy and fluency?

Adequacy Fluency

5 all meaning 5 flawless English

4 most meaning 4 good English

3 much meaning 3 non-native English

2 little meaning 2 disfluent English

1 none 1 incomprehensible

(3)

Evaluators Disagree

• Histogram of adequacy judgments by di↵erent human evaluators

1 2 3 4 5 10% 20% 30% 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 (from WMT 2006 evaluation)

Measuring Agreement between Evaluators

• Kappa coefficient

K =p(A) p(E)

1 p(E)

– p(A): proportion of times that the evaluators agree – p(E): proportion of time that they would agree by chance

(5-point scale!p(E) =1₅)

• Example: Inter-evaluator agreement in WMT 2007 evaluation campaign

Evaluation type P (A) P (E) K

Fluency .400 .2 .250

Adequacy .380 .2 .226

Ranking Translations

• Task for evaluator: Is translation X better than translation Y? (choices: better, worse, equal)

• Evaluators are more consistent:

Evaluation type P (A) P (E) K

Fluency .400 .2 .250

Adequacy .380 .2 .226

Sentence ranking .582 .333 .373

Human Evaluation

Rank Sentences

You have judged 25 sentences for WMT09 Spanish-English News Corpus, 427 sentences total taking 64.9 seconds per sentence.

Source: Estos tejidos están analizados, transformados y congelados antes de ser almacenados en Hema-Québec, que gestiona también el único banco público de sangre del cordón umbilical en Quebec.

Reference: These tissues are analyzed, processed and frozen before being stored at Héma-Québec, which manages also the only bank of placental blood in Quebec.

Translation Rank

These weavings are analyzed, transformed and frozen before being stored in Hema-Quebec, that negotiates also the public only bank of blood of the umbilical cord in Quebec.

1 Best

2 3 4 5

Worst

These tissues analysed, processed and before frozen of stored in Hema-Québec, which also operates the only public bank umbilical cord blood in Quebec.

1 Best

2 3 4 5

Worst

These tissues are analyzed, processed and frozen before being stored in Hema-Québec, which also manages the only public bank umbilical cord blood in Quebec.

1 Best

2 3 4 5

Worst

These tissues are analyzed, processed and frozen before being stored in Hema-Quebec, which also operates the only public bank of umbilical cord blood in Quebec.

1 Best

2 3 4 5

Worst

These fabrics are analyzed, are transformed and are frozen before being stored in Hema-Québec, who manages also the only public bank of blood of the umbilical cord in Quebec.

1 Best

2 3 4 5

Worst

Annotator: ccb Task: WMT09 Spanish-English News Corpus

Instructions:

Rank each translation from Best to Worst relative to the other choices (ties are allowed). These are not interpreted as absolute scores. They are

(4)

Human Evaluation

General Goals for Evaluation Metrics

Correct: metric must rank better systems higher

Meaningful: score should give intuitive interpretation of translation quality Low cost: reduce time and money spent on carrying out evaluation

Useful for tuning: automatically optimize system parameters towards metric Consistent: repeated use of metric should give same results

Automatic Evaluation Metrics

• Goal: computer program that computes the quality of translations • Advantages: low cost, fast, re-usable

• Basic strategy

– given: machine translation output – given: human reference translation – task: compute similarity between them

Precision and Recall of Words

Israeli officials

responsibility of

airport

safety

Israeli officials are responsible for airport security

REFERENCE: SYSTEM A: • Precision correct output-length= 3 6= 50% • Recall correct reference-length= 3 7= 43%

• F-measure precision⇥ recall

(precision + recall)/2=

.5⇥ .43

(.5 + .43)/2= 46%

(5)

Precision and Recall

Israeli officials

responsibility of

airport

safety

Israeli officials are responsible for airport security

REFERENCE:

SYSTEM A:

airport security Israeli officials are responsible

SYSTEM B:

Metric System A System B

precision 50% 100%

recall 43% 100%

f-measure 46% 100%

flaw: no penalty for reordering

Word Error Rate

• Minimum number of editing steps to transform output to reference match: words match, no cost

substitution: replace one word with another insertion: add word

deletion: drop word • Levenshtein distance

wer =substitutions + insertions + deletions reference-length

Example

offi

cials

Israeli responsibility of airport safety

0 1 Israeli 2 3 4 5 1 officials 1 2 3 4 2 1 are 1 2 3 4 3 2 responsible 2 3 4 4 3 for 3 3 3 4 5 4 airport 4 4 4 6 5 security 5 5 4 4 0 3 2 Israeli 2 officials 3 are 4 responsible 5 for airport 6 security airport 1 2 3 4 5 6 security 2 3 3 4 5 6 6 Israeli 3 4 5 6 7 offi cials 3 3 3 4 5 6 are 4 4 3 3 4 5 responsible 5 2 2 5 5 2 2 1 2 3 4 5 6 2 3 4 5 7 1 0 6 1 2 3 4 5 6 0 1 2 3 4 5 6 7

word error rate (wer) 57% 71%

BLEU

• N-gram overlap between machine translation output and reference translation • Compute geometric mean of n-gram precisions (typically size 1 to 4):

P = pn

precision1⇤ precision2⇤ ... ⇤ precisionn = (precision1⇤ precision2⇤ ... ⇤ precisionn)

1 n₌ n Y i=1 precisioni !1 n

• Add brevity penalty for short translations: BP

⇢

1 if output-length c > reference-length r

(6)

BLEU

• More efficient: P = (Qn_i=1precisioni) 1

n_{= exp} 1 n

Pn

i=1loge(precisonn) • Putting everything together (for 1 to 4-grams):

BLEU = min ✓ 1, exp ✓ 1 reference-length output-length ◆◆ exp 1 n N X n=1 loge(precisonn) !

• Typically computed over the entire test corpus, not single sentences • Can you figure out why?

Example

airport security Israeli officials are responsible Israeli officials responsibility of airport safety Israeli officials are responsible for airport security

REFERENCE: SYSTEM A:

SYSTEM B:

4-GRAM MATCH 2-GRAM MATCH

2-GRAM MATCH 1-GRAM MATCH

precision (1gram) 3/6 6/6 precision (2gram) 1/5 4/5 precision (3gram) 0/4 2/4 precision (4gram) 0/3 1/3 brevity penalty 6/7 6/7 bleu 0% 52%

Modified N-gram Precision

Avoid counting correct N-grams more often than they appear in any reference translation!

countclip= min (countcandidate, maxcountref erence) Candidate: the the the the the the the.

Reference 1: The cat is on the mat.

Reference 2: There is a cat on the mat.

countclip(the) = 2

precision1= 2/7 (unigram precision)

Multiple Reference Translations

• To account for variability, use multiple reference translations – n-grams may match in any of the references

– closest reference length used • Example

Israeli officials responsibility of airport safety

Israeli officials are responsible for airport security Israel is in charge of the security at this airport

The security work for this airport is the responsibility of the Israel government

Israeli side was in charge of the security of this airport REFERENCES:

SYSTEM:

2-GRAM MATCH 2-GRAM MATCH 1-GRAM

(7)

Typical BLEU Scores

BLEU scores for 110 statistical machine translation systems (Koehn 2005)

% da de el en es fr fi it nl pt sv da - 18.4 21.1 28.5 26.4 28.7 14.2 22.2 21.4 24.3 28.3 de 22.3 - 20.7 25.3 25.4 27.7 11.8 21.3 23.4 23.2 20.5 el 22.7 17.4 - 27.2 31.2 32.1 11.4 26.8 20.0 27.6 21.2 en 25.2 17.6 23.2 - 30.1 31.1 13.0 25.3 21.0 27.1 24.8 es 24.1 18.2 28.3 30.5 - 40.2 12.5 32.3 21.4 35.9 23.9 fr 23.7 18.5 26.1 30.0 38.4 - 12.6 32.4 21.1 35.3 22.6 fi 20.0 14.5 18.2 21.8 21.1 22.4 - 18.3 17.0 19.1 18.8 it 21.4 16.9 24.8 27.8 34.0 36.0 11.0 - 20.0 31.2 20.2 nl 20.5 18.3 17.4 23.0 22.9 24.6 10.3 20.0 - 20.7 19.0 pt 23.2 18.2 26.4 30.1 37.9 39.0 11.9 32.0 20.2 - 21.9 sv 30.3 18.9 22.8 30.2 28.6 29.7 15.3 23.9 21.9 25.9

-Evaluation of Machine Translation 24

Correlation with Human Judgement

METEOR: Flexible Matching

• Partial credit for matching stems

system Jimwenthome

reference Joe goes home

• Partial credit for matching (near) synonyms

system Jimwalkshome

reference Joe goes home

• Use of paraphrases

Critique of Automatic Metrics

• Ignore relevance of words

(names and core concepts more important than determiners and punctuation) • Operate on local level

(do not consider overall grammaticality of the sentence or sentence meaning) • Scores are meaningless

(scores very test-set specific, absolute value not informative) • Human translators score low on BLEU

(8)

Evidence of Shortcomings of Automatic Metrics

Post-edited output vs. statistical systems (NIST 2005)

2 2.5 3 3.5 4 0.38 0.4 0.42 0.44 0.46 0.48 0.5 0.52 Human Score Bleu Score Adequacy Correlation

Evidence of Shortcomings of Automatic Metrics

Rule-based vs. statistical systems

2 2.5 3 3.5 4 4.5 0.18 0.2 0.22 0.24 0.26 0.28 0.3 Human Score Bleu Score Adequacy Fluency SMT System 1 SMT System 2 Rule-based System (Systran)

Metric Research

• Active development of new metrics – syntactic similarity

– semantic equivalence or entailment – metrics targeted at reordering – trainable metrics

– etc.

• Evaluation campaigns that rank metrics

Automatic Metrics: Conclusions

• Automatic metrics essential tool for system development • Not fully suited to rank systems of di↵erent types • Evaluation metrics still open challenge

(9)

Task-Oriented Evaluation

Does machine translation output help accomplish a task?

browsing quality: Is the translation understandable in its context? (its main contents is clear to find information I need)

post-editing quality: How many edit operations are required to turn it into a good translation?

publishing quality: How many human interventions are necessary to make the entire document ready for printing?

Post-Editing Machine Translation

• Measuring time spent on producing translations – baseline: translation from scratch

– post-editing machine translation

But: time consuming, depend on skills of translator and post-editor • Metrics inspired by this task

– ter: based on number of editing steps

Levenshtein operations (insertion, deletion, substitution) plus movement – hter: manually construct reference translation for output, apply ter

(very time consuming, used in DARPA GALE program 2005-2011)

Content Understanding Tests

• Given machine translation output, can monolingual target side speaker answer questions about it?

1. basic facts: who? where? when? names, numbers, and dates 2. actors and events: relationships, temporal and causal order 3. nuance and author intent: emphasis and subtext

• Very hard to devise questions

• Sentence editing task (WMT 2009–2010) – person A edits the translation to make it fluent

(with no access to source or reference) – person B checks if edit is correct

! did person A understand the translation correctly?

Other Evaluation Criteria

When deploying systems, considerations go beyond quality of translations Speed: we prefer faster machine translation systems

Size: fits into memory of available machines (e.g., handheld devices) Integration: can be integrated into existing workflow

(10)

Summary

• MT evaluation is important – System development – Parameter tuning – Task-oriented performance • MT evaluation is difficult

– Human evaluators are expensive and disagree – Automatic metrics ar not always reliable ! Be careful when arguing about MT quality!