Rening translation grammars through paraphrase
clustering
MSc Thesis (Afstudeerscriptie)
written by Ekaterina Garmash
(born June 16th 1988 in Kiev, USSR)
under the supervision of Dr. Khalil Sima'an and Gideon Maillette de Buy Wenniger, MSc, and submitted to the Board of Examiners in partial
fulllment of the requirements for the degree of
MSc in Logic
at the Universiteit van Amsterdam.
Date of the public defense: Members of the Thesis Committee: December 17th, 2012 Dr. Alexandru Baltag
Dr. Khalil Sima'an Dr. Raquel Fernandez
Abstract
Contents
1 Introduction 3
1.1 Translation grammars in SMT . . . 4
1.2 Rening one-nonterminal hierarchical translation grammar: syn-tactic and semantic perspectives . . . 7
1.3 Contributions . . . 9
1.4 Outline . . . 11
2 Background 12 2.1 Statistical Machine Translation (SMT) . . . 12
2.1.1 Word-based models . . . 14
2.1.2 Phrase-based models . . . 15
2.1.3 Hierarchical models . . . 17
2.1.4 Extensions of the state-of-the-art SMT systems: struc-tural perspective . . . 19
2.1.5 Evaluation metrics for SMT . . . 20
2.2 Automatic extraction of semantic information . . . 21
2.2.1 Distributional semantic models . . . 21
2.2.2 Semantics in the framework of multilingual learning . . 23
2.3 Paraphrase extraction methods . . . 23
2.3.1 Distributional approach . . . 24
2.3.2 Approach using parallel corpora . . . 24
2.4 Summary and outlook . . . 26
3 Paraphrase clustering for label generation 28 3.1 Determining a starting element of a paraphrase cluster . . . . 29
3.1.1 The baseline model . . . 29
3.1.2 Choosing a representative phrase for a nonterminal . . 30
3.2 Paraphrase clustering: choosing a paraphrase model and a
clus-tering method . . . 34
3.2.1 Two types of models for a measure function . . . 34
3.2.2 Clustering algorithm . . . 35
3.3 Basic models for type 1 and 2 . . . 37
3.3.1 Basic model of type 1 . . . 37
3.3.2 Basic model of type 2 . . . 38
3.4 Assumptions and predictions associated with the dened pro-cedure . . . 39
3.5 Summary . . . 40
4 Experiments 42 4.1 Experimental setup . . . 42
4.1.1 Data setup . . . 42
4.1.2 Implementation details . . . 43
4.2 Experiments . . . 43
4.2.1 Experiments with fully labeled grammars . . . 44
4.2.2 Modifying the labeling algorithm . . . 47
4.3 Discussion of results and future work . . . 51
Chapter 1
Introduction
The ultimate goal of machine translation (MT) is to automatically translate texts from one language to another at the same qualitative level that pro-fessional human translators do. The problem therefore consists in nding or dening translation equivalence units between languages and using them to nd an optimal translation of an input text. By translation equivalence we understand a relation between linguistic units from two dierent languages such that they are translations of one another.
dene an algorithm to automatically extract patterns directly from data. In empirical linguistics, data is large collections of texts corpora which one assumes to be a good approximation of a studied linguistic system (i.e. one assumes that the patterns found in a corpus and their distribution is close to the one in the whole actual linguistic system). In MT, one studies not just one linguistic system, but the correspondence between two languages, so the data is bilingual corpora: corpora consisting of texts in two languages being translations of each other.
Example-based machine translation works with a corpus of pairs of sen-tences translating each other (typically produced by human translations). Given these, translation correspondences between smaller units of text are extracted based on some denition of string correspondence. Statistical ma-chine translation (SMT) does not just extract patterns from data collections, but attempts to model its probability distribution. It sees translation as a stochastic process where denitions of random variables are based on transla-tion equivalence. A probabilistic model allows to dene a functransla-tion assigning probability scores to pairs of sentences as an estimation of the likelihood that they are mutual translations. This way we can get a decision procedure of choosing the best translation hypothesis, which is based not on the structural properties of the sentences but on the probabilistic model in which the struc-tural properties and components of the sentences are values of the variables constituting it. In this thesis we work on an extension of an existing SMT model, and the rest of the introduction is dedicated to SMT.
1.1 Translation grammars in SMT
In this section we look at the structure of translation equivalence. There are three major paradigms of SMT with respect to the translation equivalence structure: word-based, phrase-based and hierarchical, each next one can be seen as a generalization of the previous one. We rst describe them briey, and then show that each of them can be formalized as a synchronous1 translation
grammar (although it is not typical to do so for the rst two paradigms), in order to compare them and to explain why and how we want to rene then most advanced one (hierarchical translation grammar).
Figure 1.1: A set of possible phrase alignments of constrained by word align-ments (represented by lines). (Picture taken from lecture notes to Element of language processing and learning).
translation equivalence units as pairs of single words. Word pairs constitute word alignment of sentences. Phrase-based SMT extends the denition by al-lowing pairs of any contiguous strings of words which respect word alignments (i.e. for any word in the string all of the words aligned to it should be in this phrase pair). Figure 1.1 represents all the possible phrases extractable from aligned phrases, given their word alignments. A translation hypothesis in a a word-based or phrase-based translation grammar is a result of concatena-tion of translaconcatena-tion units. The conceptual dierence between word-based and phrase-based SMT is that the former assumes absolute compositionality: that a correct translation of a sentence can be composed of translations of its small-est subunits. With the phrase-based model, on the other hand, it is possible to take contextual information into account by letting the nal translation hypothesis be composed of units of any length2 and letting the probabilistic
model decide about the best derivation. Hierarchical SMT incorporates the benets of phrase-based SMT, but allows to generalize further. Construction of a translation hypothesis in hierarchical SMT is better described as a deriva-tion in a grammar which generates strings with both terminal (nonsubstitable) and nonterminal (substitutable) symbols.
As an example, consider a German-English sentence pair (Sie kam sofort an, She arrived immediately), with word alignment pairs (sie, she), (kam, arrived), (sofort, immediately), (an, arrived). Some of the possible rules in
word-based (1.2), phrase-based (1.3) and hierarchical (1.4) grammars are given below.3 We see that in order to make a derivation in a word-based and a
phrase-based grammar we need additional rules of the form X → XX (for
concatenation of substrings). For the hierarchical grammar, we can just con-catenate strings of rules (a) and (j) and then apply rule (b).
a. X → hsie, shei
b. X → hsofort, immediatelyi
c. X → hkam, arrivedi
d. X → han, arrivedi
Figure 1.2: Some possible rules in word-based grammar
e. ...everything from the word-based grammar... f. X → hkam sofort an, arrived immediatelyi
Figure 1.3: Some possible rules in a phrase-based grammar
g. ...everything from the phrase-based grammar... j. X → hkam X an, arrived Xi
Figure 1.4: Some possible rules in an hierarchical grammar
From the above example we can also see that the hierarchical formalism allows to make better generalizations of the data. In a phrase-based framework there is really no way to learn that the German an is the so-called separable verb prex4: the best it can do is learn dierent exemplars of the general rule
(kam sofort an, kam langsam an, kam plötzlich an). The hierarchical model learns the pattern with a string kam X an, which is a generalization over the dierent examples.
A hierarchical grammar allows to introduce more abstract representations for the translation equivalence units by distinguishing between dierent non-terminal symbols. For instance, we may have translation pairshADJ NP, NP
ADJifor English-French;hget Xstate, werden Xstatei,hget Xobject, bekommen Xobjecti
for English-German, where werden is `become', and bekommen is `receive'. In the next section we will show that rening translation grammar through la-beling of nonterminal symbols can be benecial from the machine translation perspective and motivate the contribution of the present thesis.
1.2 Rening one-nonterminal hierarchical
trans-lation grammar: syntactic and semantic
per-spectives
The current baseline in hierarchical SMT is a system called Hiero [Chiang07]. Its translation grammar has only one nonterminal X (in chapter 2 we give a
formal denition)5 , which implies that for any step in a derivation, if a current
string contains X, then any rule can potentially be applied there (provided it
ts into the terminal string that is being parsed). One straightforward way to alleviate the problem above is to label nonterminal symbols so that the set of rules that can be applied at any step in a derivation reduces. In this thesis we explore precisely this way of rening a hierarchical grammar. In this section section we explain what kind of labeling we want to employ.
In chapter 2 we review some recent literature on how to rene a one-nonterminal hierarchical grammar formalism through labeling. All of the re-viewed proposals derive labels based on some syntactic denitions: their un-derlying hypothesis is that broad syntactic classes are good for specifying a restriction on the class of translation equivalence units that are applicable at some part of a derivation.
rate syntactic labels: categorial grammar tags, in addition to syntactic class, also capture position relative to other phrases in a sentence. In this thesis we attempt to capture semantic equivalence of translation units in a more di-rect fashion: we group translation equivalence units based on some notion of semantic equivalence. The advantage of proper semantic labeling is that it captures more subtle cases of ambiguity. A possible inferiority in comparison to syntactic labeling is that it might overlook the syntactic well-formedness factor (one reason being that semantic classes are typically much narrower).
For an illustration of our motivations, consider an English-Russian lan-guage pair and suppose we have unlabeled rules (1.1 - 1.2) extracted from data:
X → hX is the crown of creation, X venec evoljuciii (1.1)
X → hman, cheloveki | hman, muzhchinai (1.2)
The English man has at least two interpretations: `human' and `male'. In Russian there are two separate words for that (and no word that has both of the interpretations): chelovek (`human') and muzhchina (`male'). For the sake of illustration let us assume that `male' interpretation is more frequent in the corpus. Then with rules (1.1 - 1.2), a more likely derivation for a sentence Man is the crown of creation would be (1.3-1.4) (i.e. `Male is a the crown of creation') rather than (1.5-1.6).
X → hX is the crown of creation, X venec evoljuciii → (1.3) hman is the crown of creation, muzhchina venec evoljuciii (1.4)
X → hX is the crown of creation, X venec evoljuciii → (1.5) hman is the crown of creation,chelovek venec evoljuciii (1.6)
If we could label rules in (1.2) as in (1.7) and (1.8) and have two separate rules (1.9) and (1.10) instead of (1.1), it would be very likely that rule (1.9) has a very high probability weight, and (1.10) a very low one. Then the derivation combining rules (1.9) and (1.7) would be likely to get a higher probability score than the one combining (1.10) and (1.8).6
XL2 → hman,muzhchinai (1.8)
X→ hXL1 is the crown of creation, XL1 venec evoljuciii (1.9)
X → hXL2 is the crown of creation, XL2 venec evoljuciii (1.10) This is a typical example of a problem caused by semantic ambiguity, which, as we pointed out at the beginning of the chapter, is very relevant in the eld of MT. We showed that using labeling it is possible to distinguish between narrow semantic classes of words. Obviously, syntactic labels would not work here: they are too general to distinguish between such notions as `human individual' and `male individual'. One way to solve such a problem is to use some kind of semantic labeling: for instance, based on our example, one could use features as male/female/undened for labeling. The problem is the already mentioned complexity of natural language: it is not known a priori what kind of semantic labeling is optimal and it hard to come up with one that would account for all the cases of ambiguity in a language. Therefore, for the same reason that data-based approaches in MT are more favorable than rule-based approaches, we would like to have a procedure that automatically clusters items that are of the same semantic nature, in a way that is relevant to translation equivalence relation.
In the next section we describe the strategy of unsupervised semantic la-beling that we explore in this thesis.
1.3 Contributions
In this thesis we explore the following strategy of unsupervised semantic la-beling: we do not come up with an ad hoc set of semantic labels, but dene a general notion of similarity between translation pairs. The dened seman-tic similarity allows to derive a quantitative measure which is used to cluster translation pairs that are close enough. The resulting clusters are hypothesized to be semantic equivalence classes.
In the computational linguistics literature semantically close linguistic items are usually called paraphrases, and the task of extracting them paraphrase ex-traction. We illustrate with the same example from above how we want employ paraphrases and paraphrase clustering for translation grammar labeling.
ones from group 2. On the other hand, it is likely that phrases from group 2 have high co-occurrence scores with a phrase like are stronger than women, unlike the phrases from group 1. All of the phrases from group 1 are translated as the Russian chelovek, and the ones from group 2 as muzhchina. So if we label the rule (1.1) as (1.9), and devise a set of rules in (1.11), we get high probability scores for all of them and thus bias the decoding system to choose these rules for decoding Man is the crown of creation. Analogously, we can devise rules using labeling and grouping paraphrases meaning `male' under the same label (1.12), in order to bias the system in favor of translating Men are stronger than women correctly.
XL1 → hhuman, cheloveki | hindividual, cheloveki | hman, cheloveki | ... (1.11)
XL1 → h guy, muzhchinai | hmale, muzhchinai | hman, muzhchinai | ... (1.12) The implementation of this idea, which we are going to dene in chapter 3, comes with some important limitations that we decided to make for the initial stage of the research for the sake of simplicity. First, ideally we want to cluster translation pairs to form classes of semantically equivalent units. However, the models that we dene actually start by clustering phrases of one language and then transforming the cluster into a set of all phrase pairs containing the phrases from the cluster. Second, we do not estimate for each occurrence of a nonterminal the optimal set of translation units that are generated from it. Instead we use a uniform clustering procedure by dening a function that selects n best paraphrases given a list of all paraphrases and their similarity
scores.
1.4 Outline
The rest of the thesis is structured as follows.
Chapter 2 contains the necessary background information about the elds that are involved in this thesis: statistical machine translation, semantic mod-eling, in particular denitions of semantic similarity, and literature on para-phrases and paraphrase extraction.
In chapter 3 we dene a procedure for generating labels for an hierarchical translation grammar.
In chapter 4 we describe the experiments that we ran to test the perfor-mance of a labeled translations system. We discuss the experimental results and outline possible questions and modeling ideas for further research.
Chapter 2
Background
The purpose of this chapter is to provide the reader with the background information related to and necessary for understanding the original research presented in the subsequent chapters. Our major task in this thesis is to dene a procedure for generating labels for nonterminals of a hierarchical translation grammar.
Since the thesis about machine translation, we introduce the most impor-tant notions of the eld in section 2.1. Paraphrases are informally dened as phrases conveying the same information [BannardCallison-Burch05], therefore more precise instantiations of this general concept has to rely on some no-tion of semantic similarity. In secno-tion 2.2 we make a brief overview of some important state-of-the-art models in computational semantics and the deni-tions of semantic similarity. In section 2.3 we concentrate on the denideni-tions of paraphrases proposed in the literature and the corresponding methods for extracting them from textual data.
2.1 Statistical Machine Translation (SMT)
The goal of machine translation (MT) is, given a text in one language, to automatically produce a syntactically well-formed text conveying the same information in another language. Currently most MT systems translate a text sentence by sentence. An input sentence (the one which is translated) is called source sentence, the output (result of translation) is called target sentence. It is a convention in the state-of-the-art SMT, which we will follow, to use f
(French) for source text and e (English) for target in formal notation.1
The process of translation in MT is called decoding. It relies on a trans-lation grammar a set of productions specifying how to translate a source text to target text. Decoding consists in constructing a space of derivations, each producing some translation given a grammar and an input sentence, and based on that deciding on the best translation. Thus, a complete translation system must consist of at least two following elements: a translation grammar (supplied with a method to learn it, in case it is not manually created) and a decoding algorithm.
As a eld MT originated with a rule-based approach, whereby the de-coding algorithm was based on explicitly dened syntactic-oriented rules and manually-created dictionaries. A new stage of development of MT came with example-based MT [Nagao84]. Its innovation was the source of learning the translation correspondences a bilingual parallel corpus. A parallel corpus is a collection of parallel texts: texts the building units of which are aligned to each other. Alignment is a relation between linguistic units (paragraphs, sentences, phrases, words) that are approximately semantically equivalent or, more specically in our case, translationally equivalent. A bilingual parallel corpus consists of texts in two dierent languages aligned between each other. Input data in example-based MT (which is usually called translation mem-ory) is sentence-aligned: sub-sentential alignment is achieved by inspecting minimal pairs of sentences in one language (sentences that dier only with respect to some proper substring of words) and comparing their translations. As a simplied example, consider sentences He loves summer, He hates sum-mer and their respective French translations Il aime l`ètè, Il déteste l`ètè: it can be inferred that he is translated as il and summer as l`ètè, because these French words repeat in both sentences, just as the English originals; and, cor-respondingly, loves is aime and hates is déteste because these French words are the substring that make the two sentences dierent, just as the English originals. The decoding process consists in nding the most similar sentence from the memory for the source sentence and the most similar strings for the parts that do not match, and then using the translations of the matching strings to produce the target sentence (with an algorithm sketched above). Importantly, the choice of matching parts in the memory is based only on string similarity (typically, an optimal translation is the one that requires the least number of string combinations).
includes language model probability, which is usually dened as an n-gram model.
The task of an SMT system is to nd the best translation based on set of possible derivations and their scores2.
Although we have stated that a SMT system consists of several components (translation grammar, probabilistic model, decoding system), we will not re-view them separately, since throughout the development of SMT models, the evolution of one of the components was always to some extent motivated by the others. That is why we will try to keep the chronological order as the major one in our overview. It is common to single out three major evolution-ary stages of SMT models: word-based, phrase-based, hierarchical. We will review them one by one and give for each of them a description of their main components. However, the scope of this thesis is the structure of a translation grammar, so we will give more attention to this aspect.
2.1.1 Word-based models
Word-based SMT is represented by ve IBM models [Brownetal1993]: they all share the same modeling idea and increase in complexity from rst to fth. We will focus our attention on the rst model, since it is enough to illustrate the most important concepts of word-based SMT.
In [Brownetal1993] the authors dene a generative probabilistic model in which translation from English to French is seen as a basic process. On an intuitive level, the authors take a semantic perspective: the model they build up can be seen as a process of a native French speaker generating sentences in French so that his mental representations are English sentences. They introduce a semantically inspired notion of cept: a subset of positions in an English string together with a sense or concept that they reect. A cept gener-ates French words. On a surface level cepts are realized as word alignments connections between a pair of source and target words which translate (parts) of each other. Word alignment as a relation is not one-to-one: it is often the case that more than two words on one side correspond to a single word on the other side, and even more complex patterns occur. However, the IBM model narrows down its attention to individual words and connections between them and makes an assumption that each English word is aligned to at most one French word.
By transformingP r(e|f)intoP r(P re)P r(f()f|e), we get a decision rule for choosing
a target ebest given source f is dened as in (2.1), which is referred to as the
noisy channel model. The benets of reformulating the decision rule problem from P r(e|f) to P r(e)·P r(f|e) is that it allows to separate the problem of
nding an English string that is conveys the meaning of, or generates, a source French string (P r(f|e) translation model probability) and the problem of
making sure that the target English string is syntactically well-formed (P r(e)
language model probability). We will focus on the former.
ebest=argmaxeP r(e|f) = argmaxeP r(e)·P r(f|e) (2.1)
P r(f|e) can be rewritten as P
fP r(f,a|e), where f = f1m = f1...fm, e =
el1 = e1...el, a =am1 = a1...am, and ai stands for a position in a target string
to whichfm is aligned. The authors decompose the latter formula as in (2.2):
P r(f,a|e) =P r(m|e)
m
Y
j=1
P r(aj|a j−1 1 , f
j−1
1 , m,e)P r(fj|a j
1, f
j−1
1 , m,e) (2.2)
In the rst IBM model it is assumed that P r(m|e) is constant (equal to ),
and P r(aj|aj
−1 1 , f
j−1
1 , m,e) depends only on l (length of the English string),
and the attention is concentrated on t(fj|eaj)≡P r(fj|a
j
1, f
j−1
1 , m,e), which is
estimated as the relative frequency of (e, f) connections with respect to all e
occurrences. Thus:
P r(f, a|e) = (l+ 1)m
m
Y
j=1
t(fj|eaj) (2.3)
Summing over all possible alignments for the given source and target strings, they get a formula of translation probability. The unsupervised learn-ing of the word alignments is done via the expectation-maximization algorithm [Dempsteretal77], where the maximized function is t.
2.1.2 Phrase-based models
of dierent sizes. From the perspective of PBSMT, source and target strings consist of phrases, and word alignments in a string are taken to be constraints on which phrases it consists of:
Denition 1 (Phrase pairs [Zensetal02]). Given a word alignment A between
two strings fJ
1 andeI1, a phrase pair is a pair of substrings f
j+m j ande
i+n i such
that:
• both have no gaps;
• both are consistent withA: for allfk∈fjj+m all theel such that(fk, el)∈
A are in eii+n; and analogically for all al ∈eii+n.
A minimal phrase pair (e, f) is such that there is no phrase pair (e0, f0) where
either e0 is a substring of e, or f0 a substring of f [SimaanWenniger12].
Denition 2 (Phrases in SMT). A (minimal) phrase is an element of a (min-imal) phrase pair from denition 1.
Under the noisy-channel model, in order to estimate translation probability of a source phrase given a target phrase it is now marginalized over all possible segmentations (consistent with word alignments)B of the phrases [Zensetal02]:
P r(f1J|eI
1) =
X
B
P r(f1J, B|eI
1) =
X
B
P r(f1J|B, eI
1)P r(B|e
I
1) (2.4)
The probability distribution over possible segmentations may be assumed uniform [Zensetal02], and the probability of a source phrase given a segmen-tation and a target is estimated as ([Koehnetal03]):
P r(fI
1|eI1) =
I
Y
i=1
φ(fi|ei)d(ai−bi−1), (2.5)
where fI1 designates a particular segmentation of a phrase f1J into I phrases
(likewise for eJ
1), fi (analogously ei) stands for i-th phrase in a segmentation,
φ is a phrase translation distribution, andd(ai−bi−1)is distortion probability
where ai is the start position of a given source phrase and bi−1 is the end
2.1.3 Hierarchical models
Translation grammar is further sophisticated by learning correspondences be-tween more abstract patterns than just word strings: hierarchical SMT [Chiang05] does that by explicitly formalizing translation correspondence as a synchronous context-free grammar ([AhoUllman]). For example, if we have a pair of Chinese-English sentences in (2.6 - 2.7), where square brackets designate phrase align-ments3, we might, for example, not only extract a phrase pair (de shaoshu
guojia zhiyi, one of the few countries), but also translation pairs ([1] zhiyi, one of [1]), where [1] is a placeholder for a phrase and 1 is an index and designates alignment (translation correspondence) between two units. This way we can dene a rule X → hX1 zhiyi,one of X1i.
[Aozhou] [shi] [yu] [Bei Han] [you] [bangjiao] 1[de shaoshu guojia zhiyi ](2.6)
[Australia] [is] [dipl. rels.]1 [with] [North Korea] [is] [one of the few countries](2.7)
This model can capture translation correspondence between units of dier-ent levels of syntactic abstraction. Importantly, the introduction of the nonter-minal symbols allows to learn reordering patterns more eectively. Languages typically dier in the relative order of syntactic constituents and not individual words: phrase-based model does not allow to generalize over reordering exam-ples that involve the same type of syntactic constituents, while hierarchical model achieves the generalization by representing the constituents in question with a single non-terminal in question.
The commonly accepted baseline in hierarchical MT is Hiero translation system [Chiang07], we will refer to its translation grammar as the Hiero gram-mar:
Denition 3 (Hiero translation grammar). Hiero translation grammar is a synchronous context-free grammar (SCFG) (V,TS,TT,N, R, S), where V is a
set of terminal symbols, TS = TT = {X1, X2, S1, S2} are sets of source and
target nonterminals (respectively), N ={X, S} is a set of left-hand terminal
symbols, such that S is a starting symbol. R is a set of rules of the form A→ hγ, α,∼i, where γ is a source string, α is a target string, ∼ is a
one-to-one correspondence between non-terminals in γ and α. The rules are of two
types:
1. rules of the formX → hγ, αi, where γ and α are strings of symbols from
V, TS, TT, with the following constraints 4:
• the length of the source string on the right-hand side is at most ve
symbols;
• each side has at most two nonterminals;
• nonterminals cannot be adjacent on the source side; • a rule must have at least one pair of terminal symbols.
2. glue rules of the form S → hγ, αi, where γ and α are strings of symbols
from TS, TT, with the constraint that they contain two symbols.
There are also constraints which are active at the stage of grammar ex-traction. Hiero grammar rules are extracted in two steps: rst, phrase-pairs are extracted according to Def. 1 to form initial rules (with X on the
left-hand side); second, sub-phrase pairs (if any) are substituted withX symbols.
[Chiang07] imposes a constraint that initial rules have at most ten words on either side. Another constraint is that unaligned words are not included into initial rules.
In sections 2.1.1-2.1.3 we have seen that how the structural complexity of translation correspondence increases: the result is that there are more struc-tural aspects that we might want to include into the probabilistic model. The noisy-channel model takes care only of the translation probability and of the language model probability, while for the expressive power of the Hiero gram-mar we could take into account more than that. [OchNey02] propose to inter-polate an arbitrary number of properties, or feature functions, of an object in one model. They propose to directly model the probability P r(eI
1|f1J) within
the maximum entropy framework [Bergeretal96]: in the framework, one spec-ies a set of M feature functions hm(eI1, f1J), m ∈ {1, ..., M}. Each feature
function is accompanied with a weighting parameter λm and is interpolated
into the whole model as follows:
P r(e|f) =pλM
1 (e|f) =
exp(PM
m=1λmhm(e, f))
P
eexp(
PM
m=1λmhm(e, f))
. (2.8)
Maximizing the probability with respect to e we get a decision rule:
argmaxeP r(e|f) =argmaxe M
X
m=1
λmhm(e, f). (2.9)
This model allows to take into account dierent weighted functions of source-target pairs (hi), such as lexical probability (how well individual words
are translated), distortion probability (how good the word-order correspon-dence is), rule penalty (a monotonously increasing function of the number of rules used in the CFG derivation), etc.
2.1.4 Extensions of the state-of-the-art SMT systems:
structural perspective
In the previous three sections we have traced down how the complexity and de-tailedness of translation equivalence patterns increased from model to model. An important part of the current research in SMT is to dene a model captur-ing translation equivalence even better. There are at least two ways to proceed with this goal which are taken in the literature:
1) adding additional feature functions to the log-linear model (sometimes based on additional patterns extractable from the data);
2) modifying translation grammar;
The rst option is to directly manipulate the probability of a resulting derivation by taking into account additional structural features of aligned sen-tences. For example, [Lietal12] add an additional reordering model to the sys-tem, [Chiangetal09], [ZollmanVogel11] add distortion features, [GimpelSmith08] discuss dierent positional features.
In this thesis we are interested in the second option. To be more specic, we are interested in a particular kind of modication the one that we call labeling: labeling of a grammar is assigning labels of some kind to rules which can be dened in grammar. As said in the previous section, the latest baseline system in SMT is usually taken to be Hiero (Def. ??). As we stated in the introduction, our goal is to provide a labeling for the Hiero grammar, so here we will review the literature on labellings of this translation grammar.
An important reason why there is a general interest in extending, or more specically, labeling the Hiero grammar is that, as we pointed out in section 1.2, it is able to capture syntactic patterns between linguistic units of dierent levels of abstraction: between terminal nodes (i.e. actual words) and variables standing for phrases. The drawback of the Hiero grammar is that it is too unrestricted: it has only one nonterminal symbol (X) that can appear in a
string with terminals. This implies that in a derivation, when using a rule with X on the right-hand side the next choice of a rule can be any other
decision rule will decide that the best derivation is. Instead, if there were more than one nonterminal symbol, the set of rules that could be employed at a given step of a derivation would be smaller. This idea underlies the labeling models that we will review. Generally, the algorithm for learning a labeling assumes that the baseline grammar is available and should include the following steps:
1. Take as input a parallel text parsed with the baseline grammar.
2. To each nonterminal symbol in the parse forest assign a label according some criterion. Since we are working with parallel texts, there is a choice about the source for labeling information: the labels can be assigned to source side and then copied to the aligned target side (or vice versa). The information for labels can be based on bilingual patterns;
3. Extract the new grammar from the given labeled parse forest.
The labelings proposed in the literature are typically based on the output of some theoretically-motivated tool: phrase-structure parser [MylonakisSimaan11], categorial grammar parser [MylonakisSimaan11], dependency parser [Lietal12], POS-tagger [ZollmanVogel11]. On the other hand, labels can be theory-independent, like, for example, reordering labels [MylonakisSimaan11], [SimaanWenniger12]. The set of labels used to extend the baseline grammar can be predened (for example, labels can be the direct output of a tagging tool), or they can be coined on the y. As an example of the latter approach, [ZollmanVogel11] produce labels from sequences of POS tags that cover the phrase span corre-sponding to the nonterminal. Finally, another important modeling decision is what part of the constituent that a given nonterminal dominates is taken into account for labeling. It can be the whole terminal phrase generated from a given nonterminal, but it can be a part of the phrase: often what is called a dominant head (as dened in dependency grammar) of a constituent is used for labelling [Lietal12].
2.1.5 Evaluation metrics for SMT
The most popular automatic evaluation metrics for SMT is BLEU [Papineni etal02]. It relies on a test data set and a set of its reference translation. The evaluation algorithm compares the translation output by the decoder with the reference translations and outputs a number between 0 and 1.
The algorithm computes modied precision scores for n-grams (1- to 4-grams), denoted by pn, according to a formula:
pn=
#n-grams from candidate set in reference translations
The nal score is:
log BLEU= min(1− r c,0) +
N
X
n=1
wnlogpn, (2.11)
where r is the length of the reference corpus, c is the length of the candidate
set, wn stands for an n-gram.
2.2 Automatic extraction of semantic
informa-tion
Mutual paraphrases being sets conveying the same information [BannardCallison-Burch05], we can regard them as semantic equivalence classes and use them as semantic
information in applications. The amount of research output on paraphrases and on semantic similarity in theoretical and computational linguistics is enor-mous and we will not attempt to give the overview of it here. The scope of the present work is very specic: it is how paraphrases are seen from and can be applied in machine translation. But before we go on to present the background on research on paraphrase extraction and application, we would like to spend some time on some relevant highlights in more theoretically-oriented research in computational linguistics. We concentrate on two notable approaches to unsupervised semantic modeling: distributional modeling and the perspective of multilingual learning.
2.2.1 Distributional semantic models
Inspired by [Harris54], there has been a great deal of research in computa-tional modeling of linguistic meaning in terms of distributions in text. By a distribution of a linguistic unit we understand a set of linguistic units of a specied kind with which the former co-occurs in a corpus. The basic idea is that the structure of a language can be described in terms of distributions: for example, vowels are surrounded by consonants in a word, articles are el-ements that precede nouns, a word dog due to its semantics often co-occurs with words bark. A notorious disadvantage of this approach, which is relevant for paraphrase extraction, is that antonyms usually have close distributions: for example, it is very probable that words open and close co-occur equally frequently with a word door.
a good approximation of the semantics of this unit. [Turney06] takes the dis-tributional idea further to model relational similarity between parts of words. DSM denes semantic space as a vector space where dimensions (set of vec-tor features) are formalized as co-occurrence counts with the most frequent lexical items from the corpus, and vectors themselves correspond to senses of target words words for which we want to put into the semantic space. By co-occurrence we understand that two words co-occur in a context window (a set of words surrounding a word) of a specied size. n most frequent words
(which constitute a basis) are chosen for vectors' features because, rst, the co-occurrence counts with them provide more precise information and, second, because they make it possible to compare between dierent vectors.
[PadoLapata07] abstract over the existing DSMs and develop a general model of semantic space, which can be described in terms of what components are needed to construct it from data:
1. Context of a target word is formalized as a dependency path π, which
is said to be anchored to a target word t if t is the starting vertex of
the path (notation: πt). Note that under this denition a context of one
occurrence oftmay consist of more than one paths (if there are more than
one path available). The intuition is that the dependency formalism can capture rather deep semantic relations (if we have appropriate labels for edges of a dependency graph), unlike a simple bag-of-words approach, and is less oriented at surface syntactic properties. In addition to that one denes a context selection function cont: W →2Πt (whereΠ
tis a set
of allπt) which species what information actually is taken into account.
2. A basis B and a basis mapping function µ: Π→B (where Π is a set of
all dependency paths).
3. A path value function v : Π → R (in the simplest case this function
assigns 1 uniformly). Further, for each target word type (i.e. a set of all occurrences of some word) a global co-occurrence frequency (generalized over the whole corpus) is determined via f :B ×T →R.
[ErkPado10] explore a modication of the above model: they do not in-clude global co-occurrence frequency function into the semantic space. Their idea is to store separately contexts for each token (occurrence) and consider them dierent senses of a word. This way they perform sense disambiguation: for a given context, they activate only those exemplars that are suciently close (greater than some thresholdθ) to the current context with respect to a
of exemplars activated in a current context s act(E, s):
act(E, s) = {e∈E|sim(e, s)> θ(E, s)} (2.12)
2.2.2 Semantics in the framework of multilingual
learn-ing
In the previous subsection we have outlined a class of models that represent semantics of linguistic units by modeling the context in which they appear in a corpus. Another source of information that can be used for approximat-ing semantics of lapproximat-inguistic units are parallel corpora (dened in 2.1). Aligned elements in a parallel text are assumed to be approximately semantically equiv-alent, and so if we are able to trace down the regularities in how words (or other units) align to words on the other side of a parallel text, we might also capture semantic regularities. This idea is used in [DiabResnik02], where the authors use aligned French language as a sense language for English for the task of word sense disambiguation. The authors point out that the method has its problems: the sketched model basically assumes that the words of a sense language are monosemous. Another problem is that their algorithm of semantic disambiguation relies on word alignment, which is generally quite noisy. Interestingly, a similar idea is expressed in [Brownetal1993] (which we have already mentioned), where they also construct a generative model with conditionedeand conditioningf and explain their intuitions as when a native
French speaks French he has a corresponding English translation (being some sort of semantic representation) in mind.
The ideas underlying [DiabResnik02] can be characterized as belonging to a more general eld of multilingual learning, where POS tags, syntactic structure, morphological structure of one language is learned from its corre-spondence to another language ([SnyderBarzilay10]).
2.3 Paraphrase extraction methods
Paraphrase extraction methods are typically based on an idea of semantic similarity, that is why they use essentially the same ways of capturing se-mantic closeness as the models discussed in section 2. There are two major approaches to paraphrase extraction. In one, expressions are considered mu-tual paraphrases if their have similar distributions in a corpus. In the second one, the criterion for paraphrase extraction is whether expressions are aligned to the same expression in the parallel text (an approach based on parallel corpora).
2.3.1 Distributional approach
Under the distributional approach phrases are considered mutual paraphrases if they have similar distributions in a corpus.
The idea of distributional similarity is realized in its simplest form in [PascaDienes05] and [Marton etal09]. The set of potential paraphrases is ob-tained by extractingn-grams from a monolingual corpus. For each occurrence
of each n-gram a so-called lexical anchor (a context window of a xed length,
which is just a set of words). [PascaDienes05] do not compute the global co-occurrence frequency, but devise a metric to compare the set of lexical anchors for each two n-grams (potential paraphrases). [Marton etal09], on the other
hand, use global co-occurrence function to build for each phrase its distribu-tional prole (formalized as a vector). Paraphrase closeness is then estimated as vector similarity.
[LinPantel01] formalize context ,as well as paraphrases themselves as spe-cially transformed dependency paths5.
2.3.2 Approach using parallel corpora
dierent language as a pivot to establish the correspondence between phrases of the same language, as [BannardCallison-Burch05] do.
[BarzilayMcKeown01] use alternative translations of the same foreign text as data. They rst perform sentence alignment between them, and then iden-tify parts of the sentence that are the same and parts that are dierent: for example, two sentences They came home and They arrived home almost co-incide except for the word in the middle this kind of divergent strings are learned as paraphrases. The approach is sound from the conceptual point of view, but it meets a major problem in application scarcity of data: multiple translations of the same text are in general not very frequent, and they mostly are alternative translations of ction literature texts, which are typically avail-able only for a small subset of languages.
A pivot-based approach in [BannardCallison-Burch05] solves the problem of scarcity of data discussed above. It uses phrase-alignments (as dened in phrase-based machine translation) between bilingual corpora to extract mono-lingual paraphrases. The idea is that if two phrases are aligned to the same foreign phrase, then they are mutual paraphrases. An important benet of this approach is that one may use dierent language pairs to extract paraphrases for a given language: for example, if one needs a paraphrase system for Spanish, he may use Spanish-English, Spanish-German, Spanish-French, etc. bitexts. This fact is useful when we consider applications of paraphrasing in machine translation: when a decoder encounters an unknown word in a source sentence (the one that is absent in the translation model), this word may be substituted by its paraphrase, which is present in the translation model. Still considering the same application, the authors dene the best paraphrase ebest which is
to be substituted instead of the original one e1 as:
ebest = argmax(e26=e1)p(e2|e1) (2.13)
= argmax(e
26=e1)
X
f
p(e2|f, e1)·p(f|e1) (2.14)
≈ argmax(e26=e1)
X
f
p(e2|f)·p(f|e1) (2.15)
From this formula we can derive the similarity measure between para-phrases (note that the measure is not symmetric):
p(e2|e1) =
X
f
p(e2|f)·p(f|e1) (2.16)
have only those syntactic labels that the original phrase has in the corpus:
pSyntConstr1(e2|e1) = p(e2|e1, S(e1)) (2.17)
= X
f
p(e2|f, S(e1))·p(f|e1, S(e1)) (2.18)
where S is a function for a phrase to a set of its syntactic labels in a corpus,
and p(f|e1, S(e1)) is obviously equal to p(f|e1).
Alternatively, one can maximize the probability with respect to a syntactic label:
pSyntConstr2(e2|e1) = argmaxsi∈S(e1)p(e2|e1, si) (2.19) Finally, one can require the paraphrases to have the same syntactic label as the given occurrence of the original phrase. If the given syntactic model allows multiple labels for one occurrence, then the idea can be implemented in two ways: one can maximize the conditional probability with respect to a label ((2.20), where CurrS(f1)stands for a set of current syntactic labels off1), or
one can sum over all of the current labels (this will give the same formula as (2.17), with S(f1) interpreted as the set of current syntactic labels).
psyntConstr3(f2|f1) = argmaxsi∈CurrS(f1)p(f2|f1, si) (2.20)
2.4 Summary and outlook
In this chapter we have reviewed the research in machine translation and computational semantics which sets the necessary background for the original research presented in this thesis.
In 2.1 we have reviewed the state-of-the-art models of SMT, with an em-phasis on the translation grammar models, and their evaluation metrics. We also gave a brief overview of the current literature on sophisticating the gram-mar formalisms for translation the line of research represented by this thesis. In section 2.2 we presented a summary of the recent modeling attempts in the eld of computational semantics. We drew a line between two kinds of approaches: the ones that approximate linguistic meaning based on the contexts of the units that carry the meanings, and the ones that capture meaning correlation by capturing how linguistic units of one (target) language are related (via translation correspondence) to other language(s).
divided into two classes: some models take paraphrases to be phrases that appear in similar contexts, and other models dene paraphrases as phrases aligned to the same elements in a parallel corpus.
Chapter 3
Paraphrase clustering for label
generation
The goal of the present thesis is dene a procedure for generating a set of labels for an hierarchical translation grammar (we work with a particular case, the Hiero grammar). We generate a set of labels by considering each occurrence of a nonterminal in a parse forest of an unlabeled grammar and dening a label for this occurrence. The kind of labeling we want to arrive at is semantic-oriented, which in particular implies that the classes denoted by the labels are much narrower than syntactic labels used in the literature (reviewed in chapter 2). We stress that in this chapter we dene label generation, and not the actual labeling algorithm. The latter will be considered in the next chapter, where we consider a number of alternative labeling methods: fully labeling the original grammar, labeling a restricted set of occurrences of nonterminals, and others. We start with an informal overview of all the steps needed to generate the labeling we want to arrive at. The sections that follow will provide a detailed and formal specication of the framework.
To informally describe the procedure of label generation, we need to in-troduce some additional terminology: intuitively, it is reasonable to have a concept of a representative phrase for a nonterminal. In general, labeling is about nding a good representative of the class of phrases that are to be gen-erated from some nonterminal symbol. On the other hand, each X in a tree
dominates a terminal phrase, therefore, it is reasonable to say that for each occurrence ofX there is a phrase that represents it, or characterizes it (it may
be the whole dominated phrase, or its subphrase, or some particular repre-sentation of the phrase) we are going to use this second meaning. The idea behind our labeling procedure is then, given a symbol X in a tree, to cluster
denition of the method to generate our labeling set requires specication of at least the following components:
• what to consider a representative phrase representing for a given
con-stituent;
• how to dene a paraphrase of a given phrase;
• what measure function to use to determine cluster membership;
• what clustering algorithm to employ.
The following sections give a detailed specication of the needed compo-nents. Section 3.1 species the models that we use as an initial step of our clustering: the grammar that we will label we will refer to it as a basis for labeling (the Hiero grammar) and a model for choosing a representative phrase for a non-terminal (CCL model). Section 3.2 denes an algorithm for choosing a cluster of closest paraphrases of a given phrase. First, it gives two general denitions of paraphrase similarity functions and derives from it a paraphrase similarity measure. Second, it denes the clustering algorithm itself. Sec-tions 3.3 gives the simplest implementaSec-tions of the two general deniSec-tions of paraphrases: we call them basic models.
3.1 Determining a starting element of a
para-phrase cluster
In this section we dene formally the models that we will use to produce the rst step of the labeling: a grammar that is a basis for labeling and a model which determines for each occurrence of a nonterminal in the basis grammar a representative phrase (in the sense dened above). We conclude this section with a precise specication of our framework.
3.1.1 The baseline model
As a baseline we use a translation grammar Hiero from [Chiang07], the de-nition for which we repeat here:
Denition 4 (Hiero translation grammar). The Hiero translation grammar is a synchronous context-free grammar (SCFG) (V,TS,TT,N, R, S), where V is
a set of terminal symbols,TS =TT ={X1, X2, S1, S2}are sets of of source and
symbols, such that S is starting symbol. R is a set of rules of the form A → hγ, α,∼i, where γ is a source string, α is a target string, ∼ is a one-to-one
correspondence between non-terminals in γ and α. The rules are of two types:
1. rules of the formX → hγ, αi, where γ and α are strings of symbols from V, TS, TT, with the constraints on the form and size of source and target
strings specied in Denition 3;
2. glue rules of the form S → hγ, αi, where γ and α are strings of symbols
from TS, TT, with the constrain that they contain two symbols.
3.1.2 Choosing a representative phrase for a nonterminal
A straightforward solution for choosing a representing phrase would be to take the whole terminal phrase that is generated from a given non-terminal in a tree. Such a model might not work for our purposes (clustering), since the likeliness of a phrase to have a paraphrase decreases with its length. As a result, the implemented model will be sparse and might not work well. Also, such a solution neglects one of the major generalizations in linguistics the recursiveness of language, since it treats each phrase as dening its own separate category. Projected onto the setting we are working in (hierarchical translation grammar), the idea of recursiveness of language can be spelled out as: each phrase has a subphrase that captures its essential properties. If this hypothesis is correct, then, given a selection criterion for a dominant element, it can be a rst step of generalizing the set of categories. The second step would be (in our case) clustering of phrases, to reduce the set of categories even further.
For the purpose of selecting a dominant element, we might preprocess the data with some existing tool, such as a dependency parser, in order to choose a dominant subphrase for each phrase. However, existing tools typically work with syntactic information, while in the introduction chapter we showed that our idea is capture is more narrow semantic regularities. Thus, a syntactic tool might be too crude for our goals. Another problem is that not all tools are available for any language.
Core context labels (CCL): an information-theoretic approach to labeling of synchronous hierarchical representations
The CCL model takes an information-theoretic perspective and chooses a phrase with the most unpredictable translation to label a rule with it. The labels that are produced by the model are called core context labels, since, supposedly, they capture the core of the phrase for which the labeling is done. In order to single out a phrase with the most unpredictable translation, a probabilistic model is dened.
The extensions of the Hiero grammar that we reviewed in the background chapter are typically based on the structural properties of one of the languages (source or target). This approach is well-motivated, since interlingual dier-ences are often quite systematic. On the other hand, the translation grammars that we reviewed (from word-based to hierarchical) have bilingual structures (word pairs, phrase pairs, etc.) as their basic units. The CCL model continues this latter approach of directly formalizing the interlingual correspondence.
The CCL model formalizes an event space as consisting of events of a particular source phrase having a particular corresponding target unit. A random variable is formalized as a source phrase ranging between dierent translations:
Denition 5 (Random variable represented by a phrase). A random variable varf represented by a source phrase f ranges between translation pairs in which
the rst element is f and the second element is any e phrase. The probability distribution of varf is dened as the conditional probability of a target phrase
e given f:
P r(varf) ={P r(e|f)|e∈set of target phrases in a corpus}
As an example, suppose a French word f1 is aligned toe1, e2,e3 in a
bilin-gual corpus: the values of the corresponding variable are{hf1, eii|ei is in the corpus},
and their probabilities are #(f1↔e1)
#f1 ,
#(f1↔e2)
#f1 ,
#(f1↔e3)
#f1 and zero for the rest of the values.
The CCL model restricts the set of variables to those which are represented by minimal phrases. We repeat the denition from the background chapter: a minimal phrase does not contain a proper subphrase. Non-minimal phrases are taken to be complex objects whose probability distribution is a function of the distributions of the minimal phrases of which it consists: it is dened to be the distribution of the minimal phrase with the highest entropy.
Denition 7 (Probability distribution of phrase in CCL). A probability dis-tribution of a phrase f is equal to the one of the minimal phrases constituting it with the highest entropy, dened as:
H(varf) = −
X
xi∈varf
P r(xi) logP r(xi) =−
X
hf,eii
P r(ei|f) logP r(ei|f),
where ei ∈ {set of target phrases in a corpus} and the random variable varf is
dened in Def.5.
The basic idea behind the CCL-based labeling is to choose a minimal phrase which has the highest entropy among the minimal subphrases constituting a phrase dominated by a given nonterminal. However, we might want to exclude cases where a very rare phrase pair is chosen, even though formally it has a higher entropy than the rest of the minimal subphrases. For that reason the entropy measure is weighted with a probability of the given minimal phrase pair.
Denition 8 (CCL-based labeling for the Hiero tree representation). Every non-terminal in a Hiero tree is labeled with a phrase pair hf, ei which is
de-rived from this non-terminal and has the highest rank among the rest of the minimal subphrases derived from this nonterminal with respect to the follow-ing measure: ambigf(hf, ei)×prob(hf, ei), where ambigf is the entropy of a
variable represented by a source (French) part of the phrase pair and prob is the probability of hf, ei.
On the other hand, such a local approach to labeling has a disadvantage:1
a core context label derived from a terminal node and its translation may go up high in the derivation tree, if the minimal phrase has a high ambiguity score. As a result, we get rules with labels consisting of phrases which are actually generated much later, in a dierent context. It follows then that at decoding we have to choose a translation of a very ambiguous word just based on the probabilistic model, without any context (words or phrases that are adjacent to the phrase in question in the derived sentence). On the other hand, in the eld of statistical parsing lexicalization (labeling of nonterminal with some lexical information from the constituents) has been shown to be useful. However, in parsing lexical labels are additional to syntactic ones, i.e. they they further categorize already existing classes of rules, which is not the case in our setting.
We use the CCL-based labeling as a rst step of our own labeling procedure. The hypothesis behind CCL labeling is that a minimal phrase pair chosen to represent a constituent is its good characterization in the sense that it splits the rules in such a way that the resulting best translations increase in quality. In our model we x the source phrase of the chosen minimal phrase pair and then group together its closest paraphrases. Thus, what we are assuming in addition to the above is: close paraphrases characterize approximately the same sets of constituents.
3.1.3 Summary: a paraphrase-based framework of
ex-tension of Hiero translation grammar
Now we can describe in more detail the learning method for our paraphrase-based extension. We specify it in form of a high-level algorithm:
1. The input is a set of aligned sentences parsed with a trained Hiero gram-mar.
2. Each occurrence ofX in the forest receives a CCL label as prescribed in
Def.8. We only need the source-side element of a label.
3. For each occurrence of a CCL label, we take its source-side part f1, and
compute its similarity to the rest of the source phrases sim(fi|f1). We
then choose the subset of all the paraphrases based on their rankings to get a cluster label.
c
par1 par2
Figure 3.1: A model of type 1: c stands for a conditioning (causing), pari
stand for conditioned paraphrase variables
3.2 Paraphrase clustering: choosing a paraphrase
model and a clustering method
Clustering can be described as grouping of elements of the same nature into classes. Clustering typically relies on some measure that characterizes some property of all the elements in the dataset, and based on it the closeness be-tween elements with respect to this measure is determined. Dierent clustering methods dene algorithms that determine for every pair of elements in a set whether they should be included in the same class. Thus, in order to specify our clustering method, we have to x a measure function of closeness between phrases and also choose a clustering algorithm. The two following subsections do that.
3.2.1 Two types of models for a measure function
In this subsection we describe two dierent approaches that we take to derive a similarity measure between phrases.
Model of type 1
Under the rst approach, which we will call model of type 1, a measure func-tion is derived directly from a probabilistic model: it is condifunc-tional probabil-ity P r(par2|par1)(where par stands for paraphrase). We dene a generative
we can extract from data (i.e. something on which a denition of paraphrases is based).
We can make a choice whether in our model the conditioning variable is observed this will inuence the nal formula of conditional probability: (3.1) is a formula for the setting when the conditioning variable is unobserved , (3.2) is for when it is observed. The equality in (3.2) and in the last step in (3.1) is due to the principle of D-separation in a directed graphical model.
P(par2|par1) =X
c
P(par2, c|par1) =X
c
P(par2|c,par1)·P(c|par1) =X
c
P(par2|c)·P(c|par1)
(3.1)
P(par2|par1, c) = P(par2|c) (3.2)
In section 3 we describe the simplest model of this type in which a con-ditioned variable is interpreted as a source phrase, and a conditioning as a target phrase.
Model of type 2
Under the second approach, we build up a vector space model in which para-phrases are represented as vectors. We then use a standard measure of vector similarity to determine paraphrase closeness. In this work we chose the cosine measure, which is a dot product of two vectors divided by their lengths:
cos(v1, v2) =
v1·v2
|v1||v2| (3.3)
The feature space should somehow characterize the distribution of a given phrase: for example, with respect to translations or contextual information. In section 4 we describe a basic model of type 4, where the features of a vector are taken to be conditional probability betweenf (a phrase) and all the
potential translations ei.
An important dierence from the conditional probability measure of type 1 model is that the current similarity measure is normalized with respect vector lengths. Another dierence is that the cosine measure is symmetric, unlike the conditional probability.
3.2.2 Clustering algorithm
Clustering can be characterized as global or local [ManningSchuetze02]. Clustering is global if it estimates classes by taking into account the whole data set. Local clustering methods estimate cluster with respect to a particular data point or data region. It is clear from the informal description of our method that we need local clustering: since we estimate for each data point (phrase) a set of its closest points.
A typical disadvantage of local clustering is that the nal result might largely depend on the starting point of the clustering procedure. However in this paper, when estimating a cluster for a particular phrase, we disregard the clusters that have been already computed: each cluster is computed separately for each nonterminal occurrence. This entails that the clusters that we get in the end might intersect to a large extent.
We work with two similar clustering methods: the rst one is based on k nearest neighbors (kNN), and the second one is a modication of the rst one. kNN is an algorithm for estimating density of unknown data distribution
[Bishop06]. Its assumption is that there is a (quite small) number k such
for each data point x its nearest (according to some similarity measure) k
elements are get the same value in some probability distribution. The most popular version of the kNN algorithm is used in supervised classication, in which an unlabeled element gets a label of the majority of elements among its
k nearest neighbors. We use the general idea dierently: from the spelled out
assumption it follows that all the k+ 1 elements are of essentially the same
nature and therefore form a natural class.
Thus, the kNN algorithm for computing a cluster in our model is:
1. given data point (phrase) f1, compute similarity sim(fi, f1) for each fi
in the corpus;
2. order the phrases with respect to similarity score they got;
3. choosek best phrases with the respect to the ordering. In the case where
the least score that gets into a cluster is such that if we do not add it, then there will be less than k elements, and if we add all the phrases
with this score, then there will be more than k elements, we choose
the second option.
We modify the described method by lettingkbe function of the distribution of
the paraphrase similarity measure. We set a lower bound p on the score that
fi
e
[image:39.612.273.331.106.219.2]N
Figure 3.2: Basic model of type 1
The dened clustering algorithm has a variable parameter k or p, and
there is no a priori value for it. The optimization measure that we use is the nal BLEU score that the whole translation system gets.
3.3 Basic models for type 1 and 2
The models of type 1 and 2 were dened abstractly: we did not specify what kind of data to use for the parameters of the models. In this section we give a simple instantiation of the two models based on the idea proposed in [DiabResnik02] and [BannardCallison-Burch05], and call them basic models. The idea behind the basic models is to use the denition of paraphrases from [BannardCallison-Burch05]: two phrases are paraphrases if they have common translations.
3.3.1 Basic model of type 1
Given that we will cluster phrases of a source language, as a basic model of type 1 we will assume the one that has a conditioning evariable and conditionedf
variables. An e variable ranges between all the possible English phrases, and f variables between French phrases (Fig. 3.2). This basic model relies on
the already discussed hypothesis that dierent translations of a word or phrase correspond to a certain extent to its dierent senses [DiabResnik02].
referred paper, now we have both intuitive and formal motivations for it.2.
p(f2|f1) =
X
e
p(f2|e)·p(e|f1) (3.4)
p(f2|f1, e) = p(f2|e) (3.5)
3.3.2 Basic model of type 2
As it was said in section 2.1.2, the features of a vector should somehow char-acterize the corresponding paraphrase. We make the same assumption as for basic model of type 1: we assume that a translation of a phrase provides sucient characterization of a phrase.
Unlike for the model of type 1, we do not have to make any assumptions about the causal relation between English and French phrases: all we need is to reect the distribution of French phrases with respect to English phrases. Therefore, we can take a vector to be a list of probabilitiesP(ei|f), wheref is
xed, and ei ∈ {all English phrases in a corpus}. We use v(f) to denote such
vector for a phrase f. Then for every f1 and f2 their cosine score will be:
cos(v(f1), v(f2)) =
P
ep(e|f1)p(e|f2)
|v(f1)|
pP
ep(e|f2)2
(3.6)
=
P
e
#(f1,e)#(f2,e)
#f1#f2
|v(f1)|
pP
ep(e|f2)2
(3.7)
= 1
#f1·#f2· |v(f1)|
P
e#(f1, e)#(f2, e)
pP
e#(e, f2)2
(3.8)
For the application that we are considering here estimating closeness to phrasef1, the value of the initial factor in (3.8) is constant for all paraphrases.
Therefore, for this application, the model under consideration is equivalent to 2Actually in the original paper the equality is approximate:
p(f2|f1) =
X
e
p(f2|e, f1)·p(e|f1)
≈ X
e
p(f2|e)·p(e|f1)
the one in which vector features are dened as joint counts of a phrase with all possible translation.
Alternatively, we may dene a model where feature values are conditional probabilities of a given phrase given an English translation, for all English translations P(f|ei). We designate by v0(f) a vector representing phrase f
in this model. The cosine measure will look like:
cos(v0(f1), v0(f2)) =
P
ep(f1|e)p(f2|e)
|v0(f
1)|
pP
ep(f2|e)2
(3.9)
= 1
|v0(f
1)|
P
e
#(f1,e)#(f2,e)
#e2
q P
e
#(f2,e)2
#e2
(3.10)
For the same reason as above we disregard the initial factor. The major dierence between the two presented versions of type 2 models is the fact that in the second one the joint counts are normalized with respect to English phrases. So, it is supposed to be more rened than the previous one.
3.4 Assumptions and predictions associated with
the dened procedure
The ultimate goal we are pursuing in this thesis is to improve the performance of the Hiero translation system, as evaluated by the BLEU metrics. Therefore, the dened procedure for label generation has a number of implicit assumptions about how the the design choices inuence the performance of the resulting grammar (some of which have been mentioned before). These assumptions lead to certain predictions and intuitions about how the dened models will perform. We list and discuss them here.
1. Close paraphrases are representatives of the same class of phrase pairs
2. All paraphrase classes can be characterized with the same prob-ability distribution
Our clustering algorithm is based on an assumption which could be a serious limitation of our whole framework. Our clustering algorithm, k nearest neighbors, assumes that there is a xedkand a probability
distri-bution such that for each data point itsk nearest neighbors are assigned
the same probability value. Applied to the case, it means that each para-phrase equivalence class is characterized by the same distribution of its elements.
3. A minimal phrase with the highest ambiguity is a good repre-sentative phrase for a constituent
We made a choice to use the CCL model for dening a representing phrase. It is a very specic model and it is not clear whether it works well. In the next chapter we test also some alternative denitions. 4. Dierences between alternative design choices
There are some design choices that are not nal, and we explore a num-ber of alternatives, since we do not know in advance which is the best: language side based on which clustering is done, paraphrase similarity function model, number of members in a cluster (k). However, we do
have some intuitions and predictions about how these parameters will work. If the CCL model is a reasonable model for choosing a represent-ing phrase and our intuitions about it are correct, then we expect the model of Type 1 with both source and target states observed to per-form better than the other two models, since the per-former perper-forms sense disambiguation of a highly ambiguous phrase.
