Data processing - Measuring, refining and calibrating speaker and language information extracte

In the previous section we showed that the best recognizer, from the point of view of the observer, is the one that computes the observer’s posterior OX. We

also showed that OX is useful in the sense that the expected cost of its decisions

is no greater than the expected cost of using the prior to make decisions. Here we show that other useful recognizers also exist. The search for other useful recognizers is important, because the ideal recognizer OX is in general

not available, for two reasons:

• Usually P (x|θ, O) (which defines OX) is not available.

• Even if P (x|θ, O) were available, the computational complexity to im- plement OX may be prohibitive.

We therefore examine here the consequences of processing the input data, x, not via OX(x, π), but via some other function of x.

The results of this section will further provide us with the tools to analyse a secondary evaluation criterion, namely calibration, that will be introduced in the next and final section of this chapter.

2.3.1 Statistics of the data

A function S(x) that processes x in order to infer the value of θ is called a statistic of x, for θ. The key to analyse the effect of data processing with S(x) is the observer’s posterior, given the output of S(x).

Posteriors given statistics

We have already defined the observer’s posterior OX(x, π), which can be

viewed as a function of the identity statistic, x = X(x).

We now generalize this to other statistics. Let s = S(x), where S is some statistic of x. Then the observer’s conditional distributions for s may be written in terms of the Dirac delta, δ, as:

P (x, s|θi, S, O) = P (s|S, x)P (x|θi, O) = δ s − S(x)P (x|θi, O), (2.25)

P (s|θi, S, O) =

P (x, s|θi, S, O) dx (2.26)

which in turn defines the observer’s posterior, given s as: P (θi|s, π, S, O) =

πiP (s|θi, S, O)

j=1πjP (s|θj, S, O)

. (2.27)

In analogy to OX, we now define the function OS(s, π) as one that outputs

Note on expectation

Let f (s) be some function of s, then (see e.g. [56]) for s = S(x): D f (s)E s= D f S(x)E x. (2.28)

(To see this, define the LHS in terms of (2.26) and integrate out s, which is trivial because of the Dirac delta.) We shall make use of this fact several times in this work—the first two applications follow in the next two equations below. Posterior entropy

In analogy to the posterior entropy ¯C(OX), we now define the posterior entropy

of OS, for which we use short-hand ¯C(OS):

C(OS) = ¯C(OS|O) =

D C∗ OS(s, π) E s|,O =DC∗ OS(S(x), π) E x|,O . (2.29)

As we show below, ¯C(OS) has properties analogous to ¯C(OX).

Minimum posterior cross-entropy is posterior entropy

Consider a recognizer, R, which processes the data only via some statistic s = S(x), so that R(x, π) = RS S(x), π. The posterior Bayes risk, or

posterior cross-entropy, for this recognizer is: ¯ C(RS|O) = D C∗RS S(x), π θ E x,θ|,O =DC∗ RS(s, π) θE s,θ|,O =DC∗ RS(s, π) OS(s, π) E s|,O . (2.30)

As before, applying (2.13) for every s, we find: ¯

C(RS|O) ≥ ¯C(OS|O) = ¯C(OS) . (2.31)

Since RS is an arbitrary function of s, this shows that the best recognizer of

the form RS S(x), π, is the observer’s own posterior, OS S(x), π.

Maximum posterior entropy is prior entropy Here we show that OS is also a useful recognizer:

C(OS) ≤ C∗(π) . (2.32)

The benefit of a statistic

In section 2.2 we demonstrated the existence of at least one useful recognizer, OX(x, π). Here we have just shown that ‘pre-processing’ the input x with

some statistic, s = S(x), can give another useful recognizer, OS(s, π). This

can be exploited by designing the statistic, S, to give its output in a form that is more easily modelled than the original x. Then it may be easier for the recognizer, RS S(x), π, to approximate OS S(x), π. In speech processing,

the feature extractor may be understood in this way.

However, there may be a downside to using S(x), rather than the original x. By computing S(x), some relevant information that was present in x may be irretrievably lost. This is made explicit in the form of the data processing inequality.

2.3.2 The generalized data processing inequality

The data processing inequality is traditionally stated in terms of mutual information, see e.g. Cover and Thomas [28]. As mentioned above, mutual information is just prior entropy minus posterior entropy and we prefer to give an equivalent statement in terms of posterior entropy.

The generalized data processing inequality states, when s = S(x) and OX(x, π) and OS(s, π) are the observer’s posteriors given respectively x and

s, then the posterior entropies are related as: ¯

C(OX) ≤ ¯C(OS) . (2.33)

This means that the best decisions the observer could make using s as input cannot have better expected cost than the best decisions the observer could make by using x as input. Equivalently, the relevant information in s cannot exceed that in x.

Proof. This proof of (2.33) is based on [57]. Since s is a function of x, θ is conditionally independent of s, given x, or P (θi|x, s, π, O) = P (θi|x, π, O).

Again, we use Jensen’s inequality and the concavity of C∗(q): D C∗(OX(x, π)) E x|s,,O ≤ C∗ D OX(x, π) E x|s,,O = C∗(OS(s, π)) (2.36) and then take the expectation w.r.t. s on both sides of the inequality:

D C∗(OX(x, π)) E x|,O ≤ D C∗(OS(s, π)) E s|,O (2.37)

which by definitions (2.20) and (2.29) proves (2.33).

Finally, we ask the devil’s advocate question: If processing the data cannot create information, and has the potential to destroy information, why do we process the data? The answer is that we need to compute the posterior OX(x, π), or at least some approximation R(x, π), in order to be able to uti-

lize the information in the data. In fact, as we show below, OX(x, π) is the

most processed form of the relevant information in x.

2.3.3 The posterior is minimal sufficient

Any function S(x) can be denoted a statistic of x. Here we give the conditions for a statistic to be a sufficient statistic and a minimal sufficient statistic [58, 59].

As before, let s = S(x), and let respectively OX(x, π) and OS(s, π) denote

the observer’s posteriors for θ, given x and given s. The statistic S is a sufficient statistic[56] of x for θ, if for every x and π:

OX(x, π) = OS(S(x), π) . (2.38)

A sufficient statistic can therefore be used to make decisions which are as good as those made via the original data. Trivially, OX(x, π) is a sufficient statistic

of x, for θ, because it is the posterior that defines sufficiency.

A sufficient statistic is minimal sufficient if it is a function of every sufficient statistic. That is, a sufficient statistic M (x) is minimal sufficient, if for every sufficient statistic s = S(x) there exists some function g(s), so that M (x) = g(S(x)).

We now show that OX(x, π) is a minimal sufficient statistic of x for θ, by

choosing the function g, such that g(s) = OS(s, π). Definition (2.38) shows

that if S(x) is sufficient, then OX(x, π) = g(S(x)). Since S(x) is arbitrary,

the posterior OX(x, π) is a function of every sufficient statistic S(x) and is

therefore minimal sufficient.

Our point is that since OX(x, π) is a function of every sufficient statistic,

it is at least as much processed as any other sufficient statistic and is therefore the most processed form of the relevant information in x.

2.3.4 Invertible functions preserve information

We conclude this section by pointing out that statistics can be partitioned into equivalence classes, where all statistics in an equivalence class extract the same amount of relevant information. Two statistics are equivalent if one can be derived from the other via an invertible transformation.

Let s = S(x) and z = Z(x) be two statistics of x, with associated posteriors OS(s, π) and OZ(z, π). Further, let there be a bijection6 f from the range of

S to the range of Z, such that f (S(x)) = Z(x) and f−1(Z(x)) = S(x), for every x ∈ X . Then the posteriors are equal and so is the relevant information about θ extracted by each statistic:

OS S(x), π = OZ Z(x), π (2.39) and ¯ C(OS) = ¯C(OZ) . (2.40)

2.3.5 Data processing summary

In summary of this section, suppose that x = X(x), s = S(x) = f−1(Z(x)) and z = Z(x) = f (S(x)), then the best possible recognizers (according to the observer), that take respectively x, s and z as their inputs are OX, OS and

OZ. The expected costs of using these recognizers are related as:

C(OX) ≤ ¯C(OS) = ¯C(OZ) ≤ C∗(π) (2.41)

where C∗(π) is the expected cost of decisions made with the prior alone. Thus all of the observer’s posteriors, OX, OS and OZ are useful recognizers in the

sense that they make decisions with lower expected cost than decisions that can be made with the prior.

Processing with non-invertible functions can reduce the relevant information content (equivalently the decision making ability). Processing with invertible functions preserves the information content.

Even though processing can destroy relevant information, processing is unavoidable. In order to optimally use the information in data, the data has to be processed to compute the posterior. In fact we showed that the posterior is the most processed form of the relevant information.

In document Measuring, refining and calibrating speaker and language information extracted from speech (Page 46-50)