Inference

The objective function of the lattice parser is

arg max

y∈Y,p∈P

fT(y) + fP(p) (7.10)

subject to the two agreement constraints in Equations (7.8) and (7.9).

We use Alternating Directions Dual Decomposition or AD3 (Martins et al. 2011a)5 to find the optimal solution to this constrained optimization problem. AD3 is an approximate algorithm that finds the optimal solution in an iterative fashion. If the algorithm converges, the solution is guaranteed to be optimal. AD3 can efficiently handle first-order logic constraints, which we use to implement the agreement constraints. Similar approaches, where such constraints are used to ensure certain properties in the output structures, have been used, e.g., in semantic parsing (Das et al. 2012), compressive summarization (Almeida and Martins 2013), and joint quotation attribution and coreference resolution (Almeida et al. 2014).

CLE can be implemented such that its worst case complexity is O(T2), while the Viterbi

4_{The lattice structure ensures that always only one of the bigrams with the same token in second position} can be part of a path.

7.1 Lattice Parsing 121

algorithm needed to find the path is of worst case complexity O(QT2), where Q is the number of states in the lattice. Instead of combining these two problems directly, which would multiply their complexity, AD3combines them additively, such that the complexity of the parser is O(k(T2+ QT2))with k being the number of iterations that AD3is run.

7.1.3 Second-Order Parsing

The formulation of the parser given so far describes a first-order dependency parser, where features are extracted on single arcs only. However, higher-order models that consider sibling or grandparent information are known to be superior to first-order models (McDonald and Pereira 2006, Carreras 2007, Koo and Collins 2010).

Koo et al. (2010) present a formulation for a dependency parser that uses head automata (Alshawi 1996, Eisner 2000) to model second-order factors, i.e., sibling and grandparent features (see also Martins et al. 2013). In this model, a dependency structure is decomposed into smaller graphs, two for each token in the sentence. The smaller graphs model the left and right direct dependents for each token, independent of each other and, crucially, independent of the other tokens in the sentence.

Finding the optimal left and right dependents for each token independently of the other tokens does however not guarantee that the partial solutions agree with each other and form a coherent dependency tree. Therefore, Koo et al. (2010) use CLE to ensure proper spanning trees in the output of the parser. They use dual decomposition to find the optimal dependency tree defined by the head automata and CLE under the constraint that both agree on the structure.

We adopt the formalization given in Martins et al. (2013) extending it slightly to include arc labels. Let

Ain_h := { hg, h, li | g ∈ T, l ∈ L, g 6= h } (7.11) Aout_h := { hh, d, li | d ∈ T − {ROOT}, l ∈ L, h 6= d } (7.12)

122 7 Graph-based Lattice Dependency Parsing

is further split into the set of outgoing arcs to the right and to the left of h.

Aout_h,→:= { hh, d, li | d ∈ T − {ROOT}, l ∈ L, h < d } (7.13)

Aout_h,←:= { hh, d, li | d ∈ T − {ROOT}, l ∈ L, h > d } (7.14)

The set of candidate arcs for a right (left) grandparent-sibling head automaton (Koo et al. 2010) is then

AGSIB_h,→= Aout_h,→∪ Ain_h (7.15) AGSIB

h,←= Aouth,←∪ Ainh (7.16)

Note that the terms right and left are a bit fuzzy when it comes to lattices, and the definition in Equations (7.13) and (7.14) depends on the numbering of tokens in the lattice. In our lattices, tokens (i.e., transitions) are numbered such that a higher index means that the token is either on a competing path or it is further to the right on the same path. This way, the indices of tokens on the same path increase from left to right.

For each head token h in the sentence, let hd0, d1, . . . , dm+1i be the sequence of right

dependents of h (the left case works analogously). d0and dm+1 are special symbols for

the begin and end of the sequence. Furthermore, let g := head(h) be the head of h (the grandparent). The scoring function for a given grandparent-sibling head automaton is then defined as fGSIB h,→(y|AGSIB h,→) = m+1 X k=1 w · (φSIB(h, dk−1, dk, lk) + φGP(g, h, dk, lk)), (7.17) where y|AGSIB

h,→ is the subvector of y that contains all indicator variables indexed by A

GSIB

h,→.

φSIBand φGPare feature functions that extract features from the head, the dependent, its immediate next sibling, and the grandparent.

Sibling head automata (without the grandparent part) can be solved in quadratic time using dynamic programming (Koo et al. 2010, Martins et al. 2013) to find the set of right (left) dependents of h with the highest score. To include information about grandparents, this algorithm is run once for each possible grandparent yielding a cubic complexity. As there are two grandparent-sibling head automata per token in a sentence (one for right and one for left dependents), the complexity of solving the head automata for all tokens in the sentence is O(T4_L).

7.1 Lattice Parsing 123

Head automata can simply be added to the set of subproblems defined in Section 7.1.1. The objective function for the parser changes accordingly to

arg max y∈Y,p∈P fT(y) + fP(p) + T X h=0 fGSIB h,→(y|AGSIB h,→) + T X h=1 fGSIB h,←(y|AGSIB h,←)

subject to the two agreement constraints in Equations (7.8) and (7.9). The head automata subproblem gives the parser access to second-order features, but it also increases the complexity of the parser significantly.

7.1.4 Learning

We use passive-aggressive online learning (Crammer et al. 2003) with parameter averaging (Freund and Schapire 1999). We train only one parameter vector that includes features from the tree and from the path. The parser thus always performs full decoding during training. We map features into vector space using a hash kernel as described in Bohnet (2010). It gives a boost in speed and allows the model to learn weights for structures that do not occur in the training data but may be predicted during online training.

The loss function computes Hamming loss between the gold standard tree y and the predicted tree ˆy.

loss(y, ˆy) = X

ahh,d,li∈A

y(h, d, l)(1 − ˆy(h, d, l)) (7.18)

Loss is computed for the full spanning tree over the lattice including theNORELarcs. One

could compute loss only on the actual tokens being selected by the parser, but this could not be done with Hamming loss since the number of tokens in gold and prediction would not necessarily coincide.

However, while theNORELarcs are considered for computing the loss, they are not scored by the statistical model and thus do not contribute to the score of the actual solution. This is done because the only interpretation of aNORELarc is that this token is not correct. Preliminary experiments showed that scoring them indeed yields worse models.

124 7 Graph-based Lattice Dependency Parsing

Feature Model

The feature model for the parser was developed semi-automatically on the Turkish de- velopment set (see Section 7.3.1) by performing a few rounds of leave-one-out feature selection. The main purpose of the feature selection was to make the feature set smaller, in particular for the second-order features. No feature selection or adaptation was done with respect to the Hebrew data. We did not systematically evaluate the importance of individual features. The feature set is an obvious place to improve the parser and to explore the possibilities and restrictions that a lattice imposes on the type of features that are available.

The feature set comprises mostly standard features like surface form, lemmas, part-of- speech tag, morphological features and combinations thereof. The full feature set of the lattice parser is described in Appendix B. Here, we will only discuss a few interesting cases that arise due to the lattice structure.

Distance Features. In standard dependency parsing, the distance between the head and the dependent is computed based on the total ordering of the tokens in the input sentence. When parsing lattices, there is no total order over the tokens anymore, as some tokens are alternatives for each other. To determine the distance between two tokens, we therefore compute the length of the shortest path between the two tokens in the lattice. There is no shortest path for tokens that can never be on the same path, but we do not need to cover this case as the parser does not need to consider arcs between such tokens anyway (see also Section 7.1.5). Note, however, that the shortest distance between two tokens in the lattice may not coincide with the distance of the two tokens in the path that is chosen by the parser. While we do not do this, in-between features (McDonald et al. 2005) could be computed based on the shortest path as normal.

Space-delimited Words. Since for Turkish and Hebrew, tokens in the lattice are segments of larger units, i.e., space-delimited words, features can also be extracted on these words. We extract the surface form of the space-delimited word that the current token is part of and whether the token is a suffix of a space-delimited word. Furthermore, if two tokens are to be related by a dependency arc, features are added that state whether the two tokens

7.1 Lattice Parsing 125

are part of the same space-delimited word and whether they are adjacent to each other. These features are helpful for Turkish because the relations inside a space-delimited word always follow the same pattern.

Latent Context Features. One problem observed for transition-based parsing by Hatori et al. (2011) and Bohnet and Nivre (2012) is that joint models that predict, e.g., syntax and part-of-speech tags together cannot use part-of-speech tags as features on tokens that have not yet received a part-of-speech tag.6 Hatori et al. (2011) solve this problem by what they call delayed features, i.e., they delay the evaluation of features involving part-of-speech tags until the token is assigned a part-of-speech tag. Bohnet and Nivre (2012) stack their joint model on a part-of-speech tagger and features extracted from buffer tokens thus use the top-scoring part-of-speech tag from the tagger even though it might not be the one assigned by the joint model later on.

A similar problem as for the transition-based parsers arises for linear context features in the lattice parser. Due to the lattice structure, it is generally not possible to find a unique left and right neighbor of a token. We define the immediate left neighbor of a token as a token whose target state coincides with the source state of current token. Analogously, the right neighbor token’s source state coincides with the target state of the current token. However, as there are multiple paths in a lattice, there may be more than one immediate left/right neighbor for each token. We first tried to compute context features on all of them, but this made the model learn the specific lattice structure and led to strong overfitting effects. We therefore use the path features of the parsing model (see Table B.1 in Appendix B) to determine the context token with the highest score according to the current weight vector and extract context features only on these latent neighbors. The latent neighbors are computed only once and their scores are not influenced by the penalties from the dual decomposition algorithm.

6_{In transition-based parsing, the tokens that are still on the buffer have usually not yet received a part-of-} speech tag.

126 7 Graph-based Lattice Dependency Parsing

Fractional Solutions

AD3solves a relaxation by treating the binary indicator variables (Equations (7.3) and (7.6)) as continuous variables ranging from 0 to 1. This means that in general, the solution produced by the parser is not a binary indicator vector (y) but a vector of real numbers. In most of the cases, the actual solution of the algorithm will however converge to an integer solution once the model is trained. During training, when the model is not finished, fractional solutions happen more frequently. We address this issue differently during testing and during training.

During testing, we follow Martins et al. (2009) and project fractional solutions to the nearest integer solution. They run CLE on the sentence using the fractional values of the solution as arc weights rather than the actual scores from the model. In the lattice parser, we first run the Viterbi algorithm with the fractional bigram values (p) to find the best path through the lattice. Afterwards, we run CLE on the tokens selected by the first step weighted with the fractional arc values from y.7 In the experiments described in Section 7.3, fractional solutions occur in about 9% of the sentences in the Turkish development set during testing.

During training, we skip the projection step and instead perform updates with the frac- tional solution. If AD3 _{finds an integer solution, the learning algorithm performs the}

normal update. If it finds a fractional solution, features are weighted by the fractions that were assigned to the factor they were computed on. Fractional solutions also lead to fractional loss.

7.1.5 Pruning

The search space of the parser is defined as the set of well-formed non-projective depen- dency trees (Y). It is already restricted to those trees that form a coherent path through

7_{The second step can potentially fail if the pruning step (see Section 7.1.5) has pruned off all arcs that} would link a token to the other tokens on the path selected by the first step. In this case, all arcs from such a token to any other token on this path are allowed. As there is no weight for these arcs, the parser will make a very poor choice attaching this token. This solution is motivated solely from the need for a proper spanning tree, it does not give the parser means to make a sensible attachment. Fortunately, this case occurs only very rarely.

7.1 Lattice Parsing 127

the lattice by the formulation of the parser. Reducing this search space even further can lower parsing time considerably. It is commonly done and often necessary in higher-order graph-based dependency parsers (Martins et al. 2009, Koo and Collins 2010, Koo et al. 2010, Rush and Petrov 2012, Zhang and McDonald 2012, Martins et al. 2013).

Excluding Arcs between Competing Paths

First of all, we can exploit the formal properties of the lattice itself to reduce the number of arcs that the parser needs to consider. Some tokens in the lattice can never end up on the same path because they are parts of competing analyses of a word. The parser therefore does not need to consider arcs between these tokens, because they cannot both be part of a dependency tree simultaneously. Arcs between them are thus never allowed and can therefore safely be discarded from the beginning.

0 1 2

a

b

c

Figure 7.5:Pruning arcs between edges in the lattice that can never be on the same path. Token a can never be on the same path as token b and thus dependency arcs between a and b can be excluded beforehand. Both a and b can be on the same path as c.

To find such arcs we define Ctas the set of tokens that can be connected with t by a

dependency arc. Ctcontains all tokens in the lattice that can be part of a path through the

lattice that also contains t. Let source(t) be the source state of a token (recall that a token is represented as an edge in the lattice) and let target(t) be the target state of that token. The set Ctis defined recursively as

1. if target(t0) =source(t) then t0 ∈ C_t

2. if t0∈ Ctthen Ct= Ct∪ Ct0

3. if t ∈ Ct0 then t0 ∈ C_t

The first condition states that two adjacent tokens can always be part of the same path. The second condition defines transitivity, i.e., that all tokens that can be on the same path

128 7 Graph-based Lattice Dependency Parsing

as token t0 can also be part of the same path as token t if t0 and t can be on the same path. The third condition defines symmetry. If t0 can be on the same path as t, then t can be on the same path as t0.

All tokens t0 ∈ C/ tcan neither be dependents nor heads of t. The corresponding arcs can

therefore be removed from the set of candidate arcs.

Excluding Arcs Based on the Annotation Scheme

The second formal property we can use is the annotation scheme of the treebank. For example, the Turkish treebank only allows the last segments of a word to have their head outside of this word. Inner segments of a word have by definition their head immediately to the right. Since word boundaries (not token/segment boundaries) are known before parsing, tokens can be classified beforehand for whether they can be word-final segments or not. For tokens that cannot be word-final segments, we can reduce the set of candidate heads to the ones immediately following this token.

Figure 7.6 shows a schematic lattice. Word boundaries are marked with double circles and edges leading into double circles therefore represent word-final segments. Word internal segments must have their head immediately to the right, which reduces the head options for these tokens considerably. The head options for the word internal segments in Figure 7.6 would be a : {b, c}, b : {d}, c : {d}, e : {g}, f : {g}. 0 a 1 2 3 4 5 b c d e f g

Figure 7.6:Using the annotation scheme of the Turkish treebank to prune dependency arcs. Word boundaries are marked by double circles. Edges leading into double circles represent word-final segments (marked red). All other segments must have their head immediately to the right.

Arc Pruning

A heuristic pruning step is applied after the two pruning steps described above. During heuristic pruning, the number of candidate heads for each token is reduced to a maximum

7.1 Lattice Parsing 129

of ten.

The heuristic pruner is a simple classifier that is trained on the treebank and uses the first-order features of the parser’s feature model (φARC). It ranks all potential heads for each token and keeps only the top ten heads on the list. It does not enforce a tree structure