Motivation

Parsing regular expressions. _{We recall the traditional construction of NFAs from regular}

expressions, and specifically the case of concatenation, r “ r1 ¨r2, in figure 6.1. After constructing this NFA, the process to match an input stringwagainst a regular expressionris like playing a board game: A token‚(pronounced “bullet”) is first inserted into the machine in stateq0. After reading each input charactera, for each token-containing stateq, a new token is placed in every stateq1 such thatqÑa q1

, and the original token is taken away fromq. Ultimately,wis accepted iff there is a token in some accepting stateqf.

The process we just described typically only returns a boolean value indicating whether the string matches the pattern under consideration. It can, however, be extended to return the parse tree of the string, and this more involved problem is conceptually closer to the problem of evaluating QREs.

Each token represents a potential (and stillincomplete) parse tree of the entire stringw, as shown in figure 6.2. The edges of the NFA can then be labelled withactionsto update the incomplete parse trees corresponding to each token.

Recall that we inductively build accepting NFAs from regular expressions: Let r be a sub-expression of a larger regular expressionr1, and consider the NFAA, corresponding to

r_{, constructed as a component of the NFA}A1

which acceptsr1. Let the entire input string bew1 “w_pre¨w¨w_{post, where}wPJrK. Say also that processing the prefixwpre results in a token‚being dropped in the initial stateq0,A of the component machineA:

q_0,A1 Ñwpre q_0,A.

Finally, say that processingwbyAresults in the input token‚moving to some accepting stateqf,A:

‚@q0,AÑw‚1@qf,A.

Recall that the input token‚contains the partial parse tree for the prefixwpre, and that we resolved to annotate the transitions of the NFA with actions to update these partial parse trees. Here ‚1 is the modified token reaching qf,A. Therefore, to compute parse trees in

addition to the boolean accept / reject value, the idea is to include the parse treetwofw according to the sub-expressionras part of the output token‚1:

‚1 “ pt_w,‚q.

When‚1 re-enters the outer machineA1, we can perform appropriate actions to incorporate

twinto the larger parse tree under construction.

More concretely, a token (which represents an incomplete parse tree) is encoded as a tuple

pt₁,t₂, . . . ,t_kqof fully-specified parse trees for various sub-expressions of the complete regular expression. Consider, for example, the partial tree of figure 6.2b. This can be represented by the tuplepp˚bbq,p˚aaaqq, indicating the maximal fully-specified subtrees of the tree under consideration.

InAr1¨r2 from figure 6.1, we label theq0 Ñ q10andq1f Ñ q20edges with the empty

action. The incoming token ‚ is therefore passed unchanged to the state q10 of A1. On matching a prefixwpreof the input, the token reaching stateq1fis of the formpτpre,‚q, where

τ_{pre is a parse tree of}w_{prefor the sub-expression}r_{1. This token is then placed in the starting}

stateq20ofA2. On reading a suffixwpostwhich matchesr2, the token reachingq2f is of the formpτ_post,τ_pre,‚q, whereτpostis a parse tree ofwpost. The entire stringw“wprewpostcan then be described by the parse treeτ“concatpτ_pre,τ_postq. It is then sufficient to modify the original automatonAr1¨r2 as shown in figure 6.3.

Evaluating QREs. The main problem with this approach is the product construction required for oppe,fq, as repeated invocations of the product construction will yield a machine whose size is exponential in the given QRE. Given a consistent, single-use QRE e, we therefore instead construct an “evaluator” Mwhich computesJeKpwq by lazily traversing the state

space.Mis a stateful object with two actions:

start ‚1 ‚2 ‚3 ‚4 ‚5 a b a b _b (a) ¨ ˚ a a a `l ¨ ˚ b b ? (b)

Figure 6.2:Visualizing incomplete parse trees. The regular expression of interest isr“r₁¨r_2,

where r1 “ a˚ and r2 “ b˚a`b˚b. The string read so far is w “ a3_b2

. We show the incomplete parse tree corresponding to the token‚2 in figure 6.2b, where incomplete nodes are circled.

q₀

start q10 A1 q1f q20 A2 q2f qf

Emit‚ Emit‚ Let‚ “ pt_post,t_pre,‚1q.

Definetw“concatptpre,tpostq. Emitptw,‚1q.

Figure 6.3: _{Extending the NFA}Ar1¨r2 of figure 6.1 with actions to construct parse trees. On

all three marked transitions,‚refers to the incoming token. Theq0 Ñq10andq1f Ñq20

transitions simply emit the incoming token unchanged. The final q2f Ñ qf transition

M 1 2 i j n e_defined e_defined f_defined M_e M_f ‚ ‚ ptei,‚q pt_ei,‚q pt_ej,‚q ptej,‚q pt_fj,t_ej,‚q pt,‚q

Figure 6.4: Example run of the evaluatorM for splitpe Ñp fq over a streamw. The sub- expressioneis defined for two prefixesw1,w2, . . . ,wiandw1,w2, . . . ,wjof the input stream, andfis only defined forwj`1, . . . ,wn. The token at the top-left corner initiates the causal sequence of starts and results. At the end of the string, when the parent evaluatorMreceives the resultpt_fj,t_ej,‚qfromMf, it computest “t_frp:“t_es, and itself produces the result

pt,‚q.

2. Mpaq, for eachaPΣ_{, corresponding to the machine reading the symbol}a_{from the}

input stream.

In response to each action,Mmay optionally produce a result of the formpt_e,‚q, where‚is some token introduced in the past, andte is the result ofeon all input characters received after receiving‚.

We illustrate the input-output behavior of the evaluatorMfor splitpeÑpfqin figure 6.4.

M_{maintains two sub-evaluators,}M_{e and}M_{f, corresponding to each sub-expression.}M_is

given the input sequencepstart,‚q,w1,w2, . . . ,wn. The first sub-expressioneis defined on two prefixes of the input:

JeKpw1w2¨ ¨ ¨wiq “tei, and JeKpw1w2¨ ¨ ¨wjq “tej.

The second sub-expressionfis defined on the suffixwj`1wj`2¨ ¨ ¨wn:

JfKpwj`1wj`2¨ ¨ ¨wnq “tfj.

Similar to the NFA of figure 6.1,Meis given the input sequence:pstart,‚q,w1,w2, . . . ,wn, corresponding to a token being injected in the start state at the beginning of the string. Me

responds by declaringptei,‚qandptej,‚qafter readingwiandwjrespectively. In turn,Mfis given the input sequencew1,w2, . . . ,wi,pstart,ptei,‚qq,wi`1,wi`2, . . . ,wj,pstart,ptej,‚qq,

Algorithm 6.1M_splitpeÑp_fq.

1. Initialize sub-evaluatorsMe andMf foreandfrespectively. 2. On receivingpstart,‚q, invokereÐMepstart,‚q, and letrf “none. 3. On receivingaPΣ_{, invoke}r_e ÐM_epaq, andrfÐMfpaq.

4. Ifre ‰none, invokerf1 ÐMfpstart,req. Otherwise, letrf1 “none.

5. Assertrf “none_rf1 “none. At least one of the calls toMf must have returned none.

6. Ifrf“ ptf,te,‚q(for somete,tf, and‚) andrf1 “none, definet“tfrp:“tes. Output

pt,‚q.

7. Otherwise, ifrf “none andrf1 “ ptf,te,‚q(for somete,tf,‚), definet“tfrp:“tes.

Outputpt,‚q.

8. Otherwise, ifrf “rf1 “none, output none.

wj`1,wj`2, . . . ,wn. After reading wn, Mf produces the output ptfj,tej,‚q. The parent evaluatorMprocesses this result and itself produces the outputpt,‚q, wheret“t_fjrp:“t_ejs. Algorithm 6.1 describes the final construction ofM. We will prove its correctness in section 6.4.

Bounding memory usage and collision-freedom. So far, we have ignored the memory

requirements of the constructed evaluators. In the regular expression matching problem, tokens are values without structure. If two tokens reach the same state, either of them can be silently dropped without affecting algorithmic correctness (see figure 6.5). There are therefore at most Op|r|qtokens, independent of the string length.

When we generalize the problem to construct parse trees, tokens have internal structure: each token is a tuple of parse trees, with total size at most Op|r||w|q. If we are only interested in computingsomeparse tree, rather than findingallparse trees, we can still drop co-located tokens. The total memory usage of parse tree computation is therefore Op|r|2|w|q.

We present the evaluatorMoppe,fqfor oppe,fqin algorithm 6.2. The assertion on line 4 follows from the consistency requirements: eand fare mandated to have equal domains. Observe that the assertion on line 5a is crucial to correctness: otherwise, if Me and Mf

report results corresponding to different threads, ‚ ‰ ‚1, there is no meaningful way to combine them. However, the co-located-token-dropping optimization weakens the guarantee to generate some, rather than all, parse trees, and is therefore erroneous.

Consider an input stringwwhich results in the presence of co-located tokens, and a suffix

w1

such thatww1 is accepted by the machine. It follows that there exists a stringww1 which admits two different parse trees. Recall that consistency also requires the underlying regular expressions to be unambiguous. For the top-level evaluator, it follows that there arenever any co-located tokens. During our inductive construction, we will require and ensure that the

q ‚ ‚1 a a w_post w1 post

Figure 6.5: _{Dropping co-located tokens while pattern matching. Once both tokens}‚and‚1

reach the same stateq, they have equal destinies for all suffixeswpost,wpost1 . Either token can be safely dropped without sacrificing correctness. The number of live tokens is therefore bounded by|r|.

Algorithm 6.2M_oppe,fq.

1. Create sub-evaluatorsMe,Mf fore,frespectively.

2. On receivingpstart,‚q, invokereÐMepstart,‚q, andrfÐMfpstart,‚q. 3. On receivingaPΣ_{, invoke}r_e ÐM_epaqandrf ÐMfpaq.

4. Assertpre ‰none^r_f‰noneq _ pre “none^r_f“noneq. Either both sub-evaluators simultaneously report results, or neither does.

5. Ifre “ pte,‚qandrf “ ptf,‚1q(for somete,tf,‚, and‚1):

(a) Assert‚ “ ‚1.

(b) Definet“oppt_e,t_fqand returnpt,‚q. 6. Otherwise, output none.

input to each sub-evaluator be “collision-free”, i.e. that the process of inserting tokens into the start state itself does not cause token co-location. The assertion on line 5a ofMoppe,fq follows from this consideration, as token-dropping is unnecessary for collision-free inputs.

In section 6.4, we will constructMby induction on e. The most important part of the correctness proof will be showing that if a parent evaluator receives a collision-free input sequence, then the children will also receive collision-free input sequences. We will formally define collision-freedom in section 6.3.