Kendall’s τ

A somewhat more complex statistical technique than simply observing aligned words is that of Kendall’sτ[Kendall, 1938]. Designed as a correlation coefficient for evaluating

ranked data, it counts pairwise comparisons between such ranks. Given two ordered sequences, such as series of numbers, the number of concordant pairs is considered to be the number of pairs of entries which occur in the same relative order in both sets, whilediscordantpairs are the opposite. These are compared against the total number of possible matches using a simple mathematical formula:

τ = |pairsconcordant| − |pairsdiscordant| |pairsconcordant|+|pairsdiscordant|

(3.2) The range of this formula is the same as that for many, if not all, correlation coeffi- cients: a correlation of 1 indicates that the order of all pairs of entries match – i.e. the

3.5. BASELINE TOOLS 35

ranks are identical – while a correlation of -1 indicates that the ranks in one sequence are the exact inverse of those in the other. Intermediate values denote more limited similarity, with a correlation of 0 indicating that as many pairs match as do not match, suggesting that the two sequences are entirely independent of each other.

Note that the coefficient does not in any way account for the absolute identities of, or differences between, elements of the sequences: it merely compares their relative order- ing. Thus,(3,5)is concordant with(1,2), (3,5)and(1,7)due to shared monotonicity, while being discordant with any decreasing sequence such as(7,1).

Algorithm 1Kendall’sτwrapper

1: procedureTAU WRAPPER(apairs)

2: ahyp ←[hypothesis indices fromapairs in ascending order]

3: aref ←[reference indices fromapairs in ascending order]

4: amix ←[ ]

5: for alliref inaref do .Process each index inaref in order

6: ihyp ←[hypothesis indices which map toiref inapairs]

.Replace reference index with hypothesis one 7: amix ←amix +ihyp .Append hypothesis index to new ‘reference’ list

8: returnτ(ahyp, amix) .Calculate Kendall’sτon the resulting indices While intended for generic rank comparisons, Kendall’sτ applies almost directly to

the field of translation order evaluation. It has in the past been used to evaluate other systems’ success in ordering tasks [Lapata, 2003], and more recently has been applied directly to the evaluation of sentences [Birchet al., 2010].

The key adaptation required when using Kendall’s τ in our field, as introduced in

Section2.3.1, is to simplify full sentences into comparable ordered sequences. To do this, we use a very simple bespoke technique. Once again we make use of word alignments generated by third-party tools. These produce series of paired indices for equivalent words, which can then be converted to comparable sequences before calculating a normal

τvalue representing the order similarity of the two sentences.

The simple conversion process is shown in Algorithm1: from the pairwise mappings provided by alignment tools (e.g.1-1 1-2to indicate hypothesis word 1 being aligned to reference words 1 and 2), both sets of indices are extracted separately. Indices referring to reference words are replaced with those of the corresponding hypothesis word(s) to en- sure like is compared with like, though the reference words’ order is retained. Kendall’s

τis thus applied to the hypothesis indices in the order they occur in the hypothesis tree,

and the same indices in the order their counterparts occur in the reference tree.

Note that the [-1,1] range of Kendall’sτis not entirely in keeping with the norm for

machine translation evaluations, which range instead from 0 to 1. We do not consider this to be a problem, as the relative ranks of two sentences can still be meaningfully interpreted, with a lower score in all cases representing a lower level of similarity than a score closer to 1. Fundamentally, while in many statistical domains a negative correlation can indicate a useful effect – e.g. time to reach a destination vs. speed of transport – we consider that in the case of relative word ordering a negative correlation is simply worse than no correlation at all.

Component Calculation Result r length of reference 6 c length of candidate 7 N maximumnchosen 3 All weightswn 1/N 0.33 Precision withn = 1(p1) 4/6 0.67 Precision withn = 2(p2) 2/5 0.4 Precision withn = 3(p3) 1/4 0.25 Brevity penalty (BP) 1 ifc > r e(1−r/c) ifc≤r 1 BLEU BP·exp N X n=1 wnlogpn ! 0.322

n Referencen-grams n-gram matches

1 cats / had / sat / on / a / mat had / sat / on / mat 2 cats had / had sat / sat on / on a / a mat had sat / sat on 3 cats had sat / had sat on / sat on a / on a mat had sat on 4 the cat sat on / cat sat on the / sat on the mat none

Table 3.3: Calculations performed by BLEU on the example sentence pair

Table 3.2 shows which pairings are considered correct or incorrect in our example sentence pair. Note that each pair is listed twice, in the rows corresponding to each of its two components. There are thus 3 matching or concordant pairs of words, and 7 mismatching or discordant ones. We can pass this information to Equation3.2to find a final score for the sentence pair of 7−3

7+3 = 4

10 = 0.4.