A Note on Sequential Rule Based POS Tagging

A Note on Sequential Rule-Based POS Tagging

Sylvain Schmitz

LSV, ENS Cachan & CNRS, Cachan, France

[email protected]

Abstract

Brill’s part-of-speech tagger is defined through a cascade of leftmost rewrite rules. We revisit the compilation of such rules into a single sequential transducer given by Roche and Schabes (Comput. Ling. 1995) and provide a direct construction of the minimal sequential transducer for each individual rule.

Keywords. Brill Tagger; Sequential Trans-ducer; POS Tagging

1 Introduction

Part-of-speech (POS) tagging consists in assigning the appropriate POS tag to a word in the context of its sentence. The program that performs this task, the POS tagger, can be learned from an annotated corpus in case of supervised learning, typically us-ing hidden Markov model-based or rule-based tech-niques. The most famous rule-based POS tagging technique is due to Brill (1992). He introduced a three-parts technique comprising:

1. alexical tagger, which associates a unique POS tag to each word from an annotated training corpus. This lexical tagger simply associates to each known word its most probable tag ac-cording to the training corpus annotation, i.e. a unigram maximum likelihood estimation; 2. anunknown word tagger, which attempts to tag

unknown words based on suffix or capitaliza-tion features. It works like the contextual tag-ger, using the presence of a capital letter and bounded sized suffixes in its rules: for instance in English, a-ablesuffix usually denotes an ad-jective;

3. acontextual tagger, on which we focus in this paper. It consists of a cascade of string rewrite rules, calledcontextual rules, which correct tag assignments based on some surrounding con-texts.

In this note, we revisit the proof that contextual rules can be translated into sequential transducers1 proposed by Roche and Schabes (1995): whereas Roche and Schabes give a separate proof of sequen-tiality and exercise it to show that their constructed non-sequential transducer can be determinized (at the expense of a worst-case exponential blow-up), we give a direct translation of a contextual rule into the minimal normalized sequential transducer, by adapting Simon (1994)’s string matching automa-ton to the transducer case. Our resulting sequential transducers are of linear size (before their composi-tion). A similar construction can be found in (Mi-hov and Schultz, 2007), but no claim of minimality is made there.

2 Contextual Rules

We start with an example by Roche and Schabes (1995): Let us suppose the following sentences were tagged by the lexical tagger (using the Penn Tree-bank tagset):

∗_{Chapman/NNP killed/VBN John/NNP Lennon/NNP} ∗_{John/NNP Lennon/NNP was/VBD shot/VBD by/IN}

Chapman/NNP

He/PRP witnessed/VBD Lennon/NNP killed/VBN by/IN Chapman/NNP

1_{Historically, what we call here “sequential” used to be} called “subsequential” (Sch¨utzenberger, 1977), but we follow the more recent practice initiated by Sakarovitch (2009).

There are mistakes in the first two sentences: killed

should be tagged as a past tense form “VBD”, and

shotas a past participle form “VBN”.

The contextual tagger learns contextual rules over some tagsetΣof formuav→ubv(ora→b / u v using phonological rule notations (Kaplan and Kay, 1994)), meaning that the tag arewrites to b in the context of u v, where the context is of length

|uv| bounded by some fixed k + 1; in practice, k = 2ork = 3(Brill (1992) and Roche and Sch-abes (1995) use slightly differenttemplatesthan the one parametrized by k we present here). For in-stance, a first contextual rule could be “nnp vbn→

nnp vbd” resulting in a new tagging

Chapman/NNP killed/VBD John/NNP Lennon/NNP ∗_{John/NNP Lennon/NNP was/VBD shot/VBD by/IN}

Chapman/NNP

∗_{He/PRP witnessed/VBD Lennon/NNP killed/VBD} by/IN Chapman/NNP

A second contextual rule could be “vbd in →

vbn in” resulting in the correct tagging

Chapman/NNP killed/VBD John/NNP Lennon/NNP John/NNP Lennon/NNP was/VBD shot/VBN by/IN

Chapman/NNP

He/PRP witnessed/VBD Lennon/NNP killed/VBN by/IN Chapman/NNP

As stated before, our goal is to compile the entire sequence of contextual rules learned from a corpus into a single sequential function.

Let us first formalize the semantics we will em-ploy in this note for Brill’s contextual rules.2 _Let

C=r1r2· · ·rnbe a finite sequence of string rewrite rules inΣ∗_×Σ∗ _{withΣa POS tagset offixed size.}

In practice the rules constructed in Brill’s contextual tagger arelength-preservingand1-change-bounded, i.e. they modify a single letter, but this is not a useful consideration for our transducer construction. Each rule_ri ri =ui →videfines aleftmost rewrite relation

=⇒

lm defined by

w=ri⇒

lm w

0 _{iff∃x, y∈Σ}∗_{, w=xu}

iy∧w0 =xviy

∧ ∀z, z0_∈Σ∗_{, w6=zu}

iz0∨x≤pref z 2_{This is not exactly the semantics assumed by either Brill} nor Roche and Schabes, who used iterated-application seman-tics, resp. contextual and non contextual, instead of the single-application semantics we use here. This has little practical con-sequence.

wherex≤pref zdenotes thatxis a prefix ofz. Note that the domain of=ri⇒

lm isΣ

∗_·ui·Σ∗_{. Thebehavior}

of a single rule is then the relationJriKincluded in

Σ∗_×Σ∗ _{defined byJr}

iK = =_lmri⇒ ∪ IdΣ∗_\(Σ∗_·ui·Σ∗₎,

i.e. it applies=ri⇒

lm onΣ

∗_·ui·Σ∗ _{and the identity on}

its complementΣ∗_\(Σ∗_·ui·Σ∗_{). The behavior of}

C is then the composition JCK = Jr1K # Jr2K #· · ·#

JrnK. Note that this behavior doesnot employ the transitive closure of the rewriting rules.

A naive implementation ofCwould try to match each ui at every position of the input string w in

Σ∗_{, resulting in an overall complexity of O(|w| ·}

P

i|ui|). One often faces the problem of tag-ging asetof sentences{w1, . . . , wm}, which yields O((P_i|ui|)·(Pj|wj|)). As shown in Roche and Schabes’ experiments, compilingCinto a single se-quential transducerT results in practice in huge sav-ings, with overall complexities inO(|w|+|T |)and O(|T |+P_j|wj|)respectively.

EachJriKis a rational function, being the union of two rational functions over disjoint domains. Let|ri| be the length|uivi| ≤k. Roche and Schabes (1995, Sec. 8.2) provide a construction of an exponential-sized transducer Tri for each JriK, and compute their compositionTC of size|TC| = O(Qn_i=12|ri|).

As they show that each JriK is actually a sequen-tial function, their composition JCKis also sequen-tial, and TC can be determinized to yield a

se-quential transducer T of size doubly exponential inPn_i=1|ri| ≤ nk (see Roche and Schabes, 1995, Sec. 9.3). By contrast, our construction directly yields linear-sized minimal sequential transducers for each JriK, resulting in a final sequential trans-ducer of sizeO(Qn_i=1|ri|) =O(2nlogk).

3 Sequential Transducer of a Rule

Intuitively, the sequential transducer for JriKis re-lated to the string matching automaton (Simon, 1994; Crochemore and Hancart, 1997) for ui, i.e. the automaton for the languageΣ∗_u

i. This insight yields adirectconstruction of the minimal sequen-tial transducer of a contextual rule, with at most

3.1 Preliminaries

Overlaps, Borders (see e.g. Crochemore and Hancart, 1997, Sec. 6.2). The overlap ov(u, v) of two wordsuandvis the longest suffix ofuwhich is simultaneously a prefix ofv. A worduis aborder

of a wordvif it is both a prefix and a suffix ofv, i.e. if there existv1,v2inΣ∗ such thatv =uv1 =v2u. Forv 6= ε, the longest border ofvdifferent fromv itself is denotedbord(v).

Fact 1. For allu, vinΣ∗ _{andainΣ,ov(ua, v) =} ov(u, v)·aifov(u, v)·a≤pref vandov(ua, v) =

bord(ov(u, v)·a)otherwise.

Sequential Transducers (see e.g. Sakarovitch, 2009, Sec.V.1.2). Formally, a sequential transducer fromΣto∆is a tupleT = hQ,Σ,∆, q0, δ, η, ι, ρi

whereδ :Q×Σ→Qis a partial transition function, η:Q×Σ→∆∗_{a partial transition output function}

with the same domain asδ, i.e.dom(δ) = dom(η), ι ∈ ∆∗ _{is an initial output, and ρ : Q → ∆}∗

is a partial final output function. T defines a par-tial sequential function JTK : Σ∗ _{→ ∆}∗ _with

JTK(w) =ι·η(q0, w)·ρ(δ(q0, w))for allwinΣ∗for which δ(q0, w) andρ(δ(q0, w))are defined, where η(q, ε) = εandη(q, wa) = η(q, w)·η(δ(q, w), a)

for allwinΣ∗_andainΣ.

Let us noteT_(q)for the sequential transducer with qfor initial state. We writeu∧vfor the longest com-mon prefix of stringsu andv; the longest common prefix of all the outputs from state q can be writ-ten formally asV_v∈Σ∗JT_(q)K(v). A sequential

trans-ducer is normalizedif this value isεfor allq ∈ Q such that dom(JT_(q)K) 6= ∅, i.e. if the transducer outputs symbols as soon as possible; any sequen-tial transducer can be normalized. The translation

of a sequential function f by a word w in Σ∗ _is

the sequential function w−1_{f withdom(w}−1_{f) =} w−1_{dom(f) and w}−1_{f(u) = (}V

v∈Σ∗f(wv))−1 ·

f(wu) for all u in dom(w−1_{f). As in the finite} automata case where minimal automata are isomor-phic with residual automata, the minimal sequen-tial transducer for a sequensequen-tial functionf is defined as thetranslation transducerhQ,Σ,∆, q0, δ, η, ι, ρi, where Q = {w−1_{f | w ∈ Σ}∗_{} (which is finite),}

q0 =ε−1f,ι=Vv∈Σ∗f(v)ifdom(f)6=∅andι=

εotherwise,δ(w−1_{f, a) = (wa)}−1_f,η(w−1_{f, a) =} V

v∈Σ∗(w−1f)(av) if dom((wa)−1f) 6= ∅ and η(w−1_{f, a) = ε otherwise, and ρ(w}−1_{f) =}

(w−1_{f)(ε)ifε∈dom(w}−1_{f), and is otherwise} un-defined.

3.2 Main Construction

Here is the definition of our transducer for a contex-tual rule (see Fig.1):

Definition 2(Transducer of a Contextual Rule). The sequential transducer Tr associated with a contex-tual rule r = u → v with u 6= ε is defined as

Tr=hpref(u),Σ,Σ, ε, δ, η, ε, ρiwith the set of pre-fixes of u as state set, ε as initial state and initial output, and for allainΣandwinpref(u),

δ(w, a)=

    

wa ifwa≤prefu

w ifw=u

bord(wa) otherwise

ρ(w)=

    

ε ifw≤pref (u∧v)

(u∧v)−1_{w if(u∧v)<pref w <pref u} ε otherwise, i.e. ifw=u

η(w, a)=

                    

a ifwa≤pref (u∧v) ε if(u∧v)<prefwa<pref u

(u∧v)−1_{v ifwa=u}

a ifw=u

ρ(w)a·ρ(bord(wa))−1 otherwise.

It remains to show that this sequential transducer is indeed the minimal normalized sequential trans-ducer forJrK.

Proposition 3(Correctness). Letr = u → v with

u6=ε. ThenJTrK=JrK.

Proof. Let us first consider the case of input words inΣ∗_\(Σ∗_·u·Σ∗_):

Claim3.1. For allwinΣ∗_\(Σ∗_·u·Σ∗_{),δ(ε, w) =} ov(w, u)andη(ε, w) =w·ρ(ov(w, u))−1_.

By induction onw: sinceu 6= ε, the base case is w = εwithδ(ε, ε) = ε = ov(ε, u) andη(ε, ε) =

ε ε

ε

a

ε

ab

ε

aba

a

abab

ab

ababb

ε b:b

a:a a:a

b:b b:b

a:ε a:aa

b:ε b:bbb a:ab

a:a, b:b

Figure 1: The sequential transducer constructed forababb→abbbb.

andainΣ:

δ(ε, wa) =δ(δ(ε, w), a)i.h.= δ(ov(w, u), a)

Fact 1_{= ov(wa, u)}

η(ε, wa) =η(ε, w)·η(δ(ε, w), a)

i.h._{= w·ρ(δ(ε, w))}−1_{·η(δ(ε, w), a)}

=w·ρ(w0₎−1_·η(w0_{, a) ;}

(by settingw0 _{=δ(ε, w))}

we need to do a case analysis for this last equation:

Casew0_a6≤

pref u Then η(w0, a) = ρ(w0) · a ·

ρ(border(w0_a))−1_{, which yields η(ε, wa) =} w·ρ(w0₎−1_·ρ(w0_{)·a·ρ(δ(ε, wa))}−1 _=wa· ρ(δ(ε, wa))−1_.

Casew0_{a <}

pref u Then δ(ε, wa) = w0a, and

we need to further distinguish between several cases:

w0_a≤

pref (u∧v) then ρ(w0) = ε,

η(w0_{, a) = a, and ρ(w}0_{a) = ε,}

thus η(ε, wa) = wa = wa · ε−1 ₌ wa·ρ(w0_a)−1_,

w0 _{= (u∧v) thenρ(w0) = ε,η(w0, a) =ε,}

and ρ(w0_{a) = (u ∧ v)}−1 _{· w}0_{a = a,}

η(ε, wa) = w = wa · a−1 _{= wa ·} ρ(w0_a)−1_,

(u∧v) <pref w0 thenρ(w0) = (u∧v)−1 ·

w0_{, η(w}0_{, a) = ε, and ρ(w}0_{a) = (u ∧}

v)−1_·w0_{a, thusη(ε, wa) =w·((u∧v)}−1_· w0₎−1 _=wa·a−1_·((u∧v)−1_·w0₎−1 ₌ wa·ρ(w0_a)−1_.

The claim yields thatJTrKcoincides with JrKon words in Σ∗_\(Σ∗ _·u·Σ∗_{), i.e. is the identity over} Σ∗_\(Σ∗ _{·u ·Σ}∗_{). Then, since u 6= ε, a word in} Σ∗_·u·Σ∗_{can be written aswaw}0_withwinΣ∗_\(Σ∗_·

u·Σ∗_{),ainΣwithwainΣ}∗_{·u, andw}0_inΣ∗_{. Let}

u = u0_{a; the claim implies that δ(ε, w) = u}0 _and

η(ε, w) = w·ρ(u0₎−1_{. Thus, by definition ofT} r, δ(ε, wa) =u0_{a= uand thusη(ε, wa) = η(ε, w)·}

η(u0_{, a) =w·ρ(u}0₎−1_·(u∧v)−1_·v;

if(u∧v) <pref u0 η(ε, wa) = w·((u∧v)−1·

u0₎−1_·(u∧v)−1_·v=w·u0−1_·v=wa·u−1_·v;

otherwise i.e. ifu0 _{= (u∧v):η(ε, wa) =w·u}0−1_· v=wa·u−1_·v.

Thus in all casesJTrK(wa) =JrK(wa), and sinceTr starting in stateu(i.e.Tr(u)) implements the identity overΣ∗_{, we have more generallyJT}

rK=JrK.

Lemma 4(Normality). Letr =u→ v. ThenTris

normalized.

Proof. Let w ∈ Prefix(u) be a state of Tr; let us show thatVJT_r(w)K(Σ∗_{) =ε.}

If(u∧v) <pref w <pref u let u0 =

w−1_{u ∈ Σ}+_{, and consider the two} out-putsJT_r(w)K(u0_{) =η(w, u}0_{)ρ(u) = (u∧v)}−1_v and JTr(w)K(ε) = ρ(w) = (u ∧ v)−1w. Since (u ∧ v) <pref u we can write u as

(u ∧v)au00_u0_{, and either v = (u ∧v)bv}0 _or

v = u ∧v, for some a 6= b inΣ and u00_{, v}0

inΣ∗_{; this yields w = (u ∧v)au}00 _{and thus}

JT_r(w)K(u0_)∧JT

r(w)K(ε) =ε.

otherwise ρ(w) =ε, which yields the lemma.

Proposition 5 (Minimality). Letr = u → v with

u6=εandu6=v. ThenTris the minimal sequential

transducer forJrK.

Proof. Let w <pref w0 be two different states in

Prefix(u); we proceed to prove that Jw−1_T rK 6=

Jw0−1_{TrK, hence that no two states of Tr can} be merged. By Thm. 4 it suffices to prove that

such thatJTr(w)K(x) =6 JTr(w0₎K(x). We perform a

case analysis:

ifw0 _≤

pref (u∧v) then w <pref (u ∧ v) thus

JT_r(w)K(x) = xfor all x 6∈ w−1_·Σ∗_·u·Σ∗_;

considerJT_r(w)K(w0−1_{u) =w}0−1_u6=w0−1_v=

JTr(w0₎K(w0−1u);

ifw≤pref (u∧v)andw0 =u then

JTr(w0₎K(x) = x for all x and we

con-sider JT_r(w)K(w−1_{u) = w}−1_{v 6= w}−1_{v =}

JT_r(w0₎K(w−1u);

otherwise that is ifw≤pref (u∧v)and(u∧v)<pref w0 _{<pref u, or(u∧v)<pref w <pref w}0 _≤pref

u, we have ρ(w) 6= ρ(w0_{) thus JT}

r(w)K(ε) 6=

JT_r(w0₎K(ε).

4 Conclusion

The results of the previous section yield (the cases u=εandu=vare trivial):

Theorem 6. Given a contextual rule r = u → v, one can construct directly the minimal normalized sequential transducerTrof sizeO(|r|)forJrK.

The remaining question is whether we can ob-tain better upper bounds on the size of the sequen-tial transducer TC for a cascade C = r1· · ·rn than O(2nlogk_{). It turns out that there are cascades of} length n for which no sequential transducer with a subexponential (inn) number of states can exist, thus our construction is close to optimal.

References

Eric Brill. 1992. A simple rule-based part of speech tag-ger. InANLP ’92, pages 152–155. ACL Press. Maxime Crochemore and Christophe Hancart. 1997.

Au-tomata for matching patterns. In Grzegorz Rozenberg and Arto Salomaa, editors,Handbook of Formal Lan-guages, volume 2. Linear Modeling: Background and Application, chapter 9, pages 399–462. Springer. Ronald M. Kaplan and Martin Kay. 1994. Regular

mod-els of phonological rule systems. Comput. Linguist., 20(3):331–378.

Stoyan Mihov and Klaus U. Schultz. 2007. Effi-cient dictionary-based text rewriting using subsequen-tial transducers.Nat. Lang. Eng., 13(4):353–381. Emmanuel Roche and Yves Schabes. 1995.

Determinis-tic part-of-speech tagging with finite-state transducers.

Comput. Linguist., 21(2):227–253.

Jacques Sakarovitch. 2009. Elements of Automata The-ory. Cambridge University Press.

Marcel-Paul Sch¨utzenberger. 1977. Sur une variante des fonctions s´equentielles. Theor. Comput. Sci., 4(1):47– 57.