Witten-Bell Smoothing

The Witten-Bell (WB) technique is another technique to solve the problem of zero probability. Here, the count of unseen events is estimated by calculating the occurrence of events which we are seeing for the first time (Witten and Bell, 1991)2. We can assume

that a training corpus is a number of events. T is a count of the number of times a new event type is seen in the corpus - i.e. a count of the number of event types in the corpus, and N is how many event tokens occur in the corpus. Let me give an example. In a corpus consisting of seven sentences:

(30) a. You are kind. b. I am happy.

c. I am happy. d. He is happy.

e. I am unhappy. f. You are kind. g. I am happy.

Each sentence in the corpus (30) is counted as an event token, so there are 7 event tokens, N. Then each different sentence is considered as an event type. Since there are 4 different sentences which are You are kind, I am happy, He is happy, and I am unhappy, so the event types, T is 4.

Now let’s focus on how the WB formula is defined. The WB probability for an event, ei is calculated by normalising the count of ei with the sum of all possible event

types, T and event tokens. Hence the equation is Pwb(ei) =

cnt(ei)

T + N, if cnt(ei) > 0 (4.10)

The normalisation by (T + N) is applied in Equation 4.10 because T and N are treated as two different events. What may happen if cnt(ei) = 0? Surely we know that

if that case occurs, it is considered as a zero probability in MLE. In WB smoothing, the zero count event is estimated from the count of event types, T. We can estimate a total probability mass of all zero counts by dividing T by the sum of T and N. Hence the equation of the total probability mass of zero-counts is formulated as:

X

ei:cnt(ei)=0

Pwb(ei) =

T

T + N (4.11)

Then how do we specify a probability of each zero count event? This can be done by dividing the probability mass by the total number of zero count events. We define Z to

be the total number of zero count events, therefore the probability of each zero count event is

Pwb(ei) =

T

Z(T + N), if cnt(ei) = 0 (4.12)

For some given w1. . . wn−1, the total space of events (ei) of the form w1. . . wn−1wn

will be of size V, as wn ∈ V. Z is therefore this value less the events from that space

that have already been seen, i.e. T. A summary of Equation 4.10 and 4.12 is depicted below, Pwb(ei) =          cnt(ei) T+N , if cnt(ei) > 0 T Z(T+N), if cnt(ei) = 0 (4.13)

Equation 4.13 is applied to unigram models. How do we generalise it for n-gram models, for example a bigram model, a trigram model and so forth? In order to calculate WB probability for n-gram (here n is greater than one), information about its (n-1 )-gram is required. When calculating a probability of non-zero count n-grams, information of its (n-1 )-gram types, T and tokens, N is used. Similarly, a probability of zero count n-gram is calculated based on T and N as well. Equation 4.14 below represents the WB formula for n-gram models.

Pwb(wn|w1. . . wn−1) =          cnt(w1...wn−1wn) T_{(w1...wn−1)}+N_{(w1...wn−1)}, if cnt(w1. . . wn−1wn) > 0 T_{(w1...wn−1)} Z_{(w1...wn−1)}(T_{(w1...wn−1)}+N_{(w1...wn−1)}), if cnt(w1. . . wn−1wn) = 0 (4.14) Let’s start with a simple example. Suppose in corpus (30), we want to calculate a probability of a trigram I am happy. An (n-1 )-gram for I am happy is a bigram I am. There are three occurrences of I am in the corpus, so N(I am) = 4. Yet, N(I am) can also be represented as cnt(I am). T(I am) means how many different types of n-gram start with I am. Here, there are two types that starts with I am which are I am happy and I am not happy. As such T(I am) = 2. If cnt(I am happy) > 0, then we can just apply those values to an appropriate equation in 4.14.

On the other hand, what if cnt(I am happy) = 0? Referring to Equation 4.14 for a case of zero counts, there is a variable called Z. Hence, Z(I am) represents a total

Table 4.2: Comparison of each probability value among MLE, Add- One and WB. MLE Add-One WB P(I |<sos>) 0.5 0.16 0.29 P(<eos>|kind ) 1.0 0.13 0.5 P(girl |happy) 0.33 0.11 0.2 P(man|happy) 0 0.06 0.03

number of zero count (unseen) n-grams, which means, here, how many unseen n-grams start with I am. This variable Z(I am) is calculated by subtracting T(I am) from the vocabulary size of the corpus, V. In a mathematical formula, Z(I am) = V−T(I am). The next paragraph provides an example of WB calculation I have previously performed in MLE and Add-One sections.

These examples refer to the corpus (27) on page 133 and Example (28) on page 133. Details of computation and results are outlined in (31). A comparison among each probability value computed from MLE, Add-One and WB techniques is depicted in Table 4.2. (31) a. Pwb(I|<sos>) = cnt(<sos> I) T(<sos>)+N(<sos>) = 2 3+4 = 2 7 = 0.29

b. Pwb(<eos>|kind) = cnt(kind <eos>)_T

(kind)+N(kind) = 1 1+1 = 1 2 = 0.5 c. Pwb(girl|happy) = cnt(happy girl) T(happy)+N(happy) = 1 2+3 = 1 5 = 0.2 d. Pwb(man |happy) = T(happy)

Z(happy)∗(T(happy)+N(happy) =

2

(15−2)∗(2+3) = 2 65 =

0.03

As depicted in Table 4.2, the difference between WB and MLE probability val- ues is not so great. For instance, Pwb(I|<sos>) reduces approximately a half from

Pmle(I|<sos>). Similarly in Pwb(<eos>|kind), the probability drops half from

Pmle(<eos>|kind). Therefore, we can say here that the WB technique yields the

Jurafsky and Martin (2009) view a discounting technique as an alternative to a smoothing algorithm. This technique is applied in order to get a probability mass by decreasing non-zero counts n-grams and assigns the probability mass to zero counts n-grams. A discount value, discnt is generated from the discounting technique and is defined as

discnt(wi) =

cnt∗(wi)

cnt(wi)

(4.15)

If we would like to see how far an original count is adjusted to a new count, an adjusted count for WB, cnt∗_wb(wi) is computed by multiplying each sub-equation in Equation

4.13 by N, cnt∗_wb(wi) =          cnt(wi) × _T+NN , if cnt(wi) > 0 T Z × N T+N, if cnt(wi) = 0 (4.16)

In the following section, I’ll describe a technique that applied a discounting tech- nique and also uses information of (n-1 )-grams if there is no occurrence of its n-grams. The technique is called Backoff.