Non-Transductive algorithms

Length-based Functions Length-based selection methods aim to select sen- tences based on the intuition that sentence pairs whose source and target side differ a lot (compared the average length difference) may be noisy and they should be left out.

In the literature we can find different approaches of measuring the difference: Taghipour et al. (2010) propose to use the length difference (|t| − |s|) or the pro- portion (|t|/|s|) to remove sentences. Another proposal (Khadivi and Ney, 2005) removes sentences whose length proportion exceeds a certain threshold t (keeping sentences which fulfill the constraint that |s|_|t| < t and |t|_|s| < t).

Alignment-based Functions Taghipour et al. (2010) use sentence-alignment en- tropy to remove noisy data from the training set. In their work, they compute the entropies of the distribution of the alignment links in a sentence.

hs, ti, the probability of a word to be aligned is defined in Equation (2.34): Pal(ws) = a(ws) X wt∈N gr1(t) a(wt) (2.34)

where ws is the word ws ∈ N gr1(s) and a(w) is a function that retrieves how many

words in t (or hNULLi tokens) are aligned to w.

The probabilities computed in Equation (2.34) are used to compute the normal- ized sentence-alignment entropies as in Equation (2.35):

scoreALIG(s) = 1 −

−P|s|

i=1Pal(xi) log(Pal(xi))

log(|s|) . (2.35)

We substract the normalized entropy from 1 so that the value of scoreALIG(s) is

higher the lower the entropy is. Sentences with high entropy indicate that the alignment of the words is evenly distributed, i.e. the mapping of the words is close to one-to-one mapping.

In contrast, for sentence pairs hs, ti with lower entropies, this indicates that words are not evenly aligned, which is an indicator that s and t are inaccurate translations of each other.

For this reason, sentence with lower entropies should be promoted (higher value of scoreALIG).

Language Model-Based Functions Moore and Lewis (2010) propose to use

an in-domain language model LMI and an out-of-domain language model LMO to

obtain sentences that are closer to the in-domain data. They therefore define the Cross-Entropy Difference (CED) as in Equation (2.36):

scoreCED(s) = HLMI(s) − HLMO(s). (2.36)

Axelrod et al. (2011) extend Equation (2.36) by using language models in both the source-side and target-side languages, defining the Bilingual Cross-Entropy Dif-

ference (BCED) in (2.37):

scoreBCED(s) = (HLMIsrc(s) − HLMOsrc(s)) + (HLMItrg(s) − HLMOtrg(s)). (2.37)

N-gram Coverage This approach consist of selecting a subset L of sentences aiming to achieve the maximum coverage possible with the minimum amount of sentences. In order to do that, Eck et al. (2005b) select those sentences containing n-grams which have not yet been added to L, rewarding those that are more frequent as defined in Equation (2.38): scoreCOV(s, L) = X ngr /∈N grn(L) CS(ngr) |s| . (2.38)

TF-IDF Coverage The proposal of Eck et al. (2005a) is to retrieve sentences that are the most different to the selected pool L based on TF-IDF distance. Those sentences that differ the most to the selected pool L are the best candidates to be added to L as they are the most informative.

A sentence s ∈ U is scored as the cosine difference between the TF-IDF vector of the sentence ds (each term w is weighted as tfidf (w, s, S)) and the vector of the

selected pool dL(each term weighted as tfidf (w, L, S)) as defined in Equation (2.39):

scoreT F IDF COV(s, L) = disttfidf(ds, dL). (2.39)

Density Weighted Diversity Sampling (DWDS) (Ambati et al., 2011) selects sentences that contain the most representative n-grams which have not yet been seen in the selected pool. Therefore the score of a sentence s is based on:

• Density d(s, L): This indicates how features are distributed in the selected pool. It is computed as the probability of the n-gram ngr in the candidate pool in proportion to the count of these n-grams (CL(ngr)) in the selected

pool, defined as in Equation (2.40): d(s, L) = X ngr∈N grn(s) PU(ngr)e−CL(ngr) |N grn(s)| . (2.40)

• Uncertainty u(s, L), the (normalized) number of non-selected n-grams con- tained in s, defined as in Equation (2.41):

u(s, L) = X ngr∈N grn(s) (1 − IN grn(L)(ngr)) |N grn(s)| (2.41)

where IA(x) is the indicator function (which retrieves 1 if the element x is in

the set A and 0 otherwise), so (1 − IN grn(L)(ngr)) indicates whether ngr is not

included in the selected pool L.

Ambati et al. (2011) define scoreDW DS(s) as the harmonic mean of the density

d(s) and the uncertainty u(s) as in Equation (2.42):

scoreDW DS(s, L) =

2d(s, L)u(s, L)

d(s, L) + u(s, L) (2.42)

Log-probability Ratios Haffari et al. (2009) propose to select sentences that contain n-grams that are common in the candidate pool and rare in the selected pool (the more frequent a phrase is in L, the less important it is). The score of a sentence is measured as in (2.43): scoreLP R(s, L) = 1 |N grnn(s)| X ngr∈N grnn(s) logPU(ngr) PL(ngr) . (2.43)

Perplexity Ratio Mandal et al. (2008) propose to select sentences based on the perplexity ratio to select sentences that are novel with respect to an initially selected

pool L. The sentences in the candidate pool U are scored as in Equation (2.44):

scoreP P R(s, L) =

P PLML(s)

P PLMS(s)

. (2.44)

The aim is to select sentences from U that are informative regarding L. Sentences that are rare (high perplexity) in L but common in S (low perplexity) are promoted as they are the most informative. In contrast, those sentences that are rare in both L and S (or only rare in S) are considered outliers and so, the score computed in Equation (2.44) will be low.