One criticism that is often made of neural networks—especially the MLP—is that it is not clear exactly what it is doing: while we can go and have a look at the activations of the neurons and the weights, they don’t tell us much. We’ve already seen some methods that don’t have this problem, principally the decision tree in Chapter 12. In this chapter we are going to look at methods that are based on statistics, and that are therefore more transparent, in that we can always extract and look at the probabilities and see what they are, rather than having to worry about weights that have no obvious meaning.
We will look at how to perform classification by using the frequency with which examples appear in the training data, and then we will see how we can deal with our first example of unsupervised learning, when the labels are not present for the training examples. If the data comes from known probability distributions, then we will see that it is possible to solve this problem with a very neat algorithm, the EM algorithm, which we will also see in other guises in later chapters. Finally, we will have a look at a rather different way of using the dataset when we look at nearest neighbour methods.
7.1 GAUSSIAN MIXTURE MODELS
For the Bayes’ classifier that we saw in Section 2.3.2 the data had target labels, and so we could do supervised learning, learning the probabilities from the labelled data. However, suppose that we have the same data, but without target labels. This requires unsupervised learning, and we will see lots of ways to deal with this in Chapters 14 and 6, but here we will look at one special case. Suppose that the different classes each come from their own Gaussian distribution. This is known as multi-modal data, since there is one distribution (mode) for each different class. We can’t fit one Gaussian to the data, because it doesn’t look Gaussian overall.
There is, however, something we can do. If we know how many classes there are in the data, then we can try to estimate the parameters for that many Gaussians, all at once. If we don’t know, then we can try different numbers and see which one works best. We will talk about this issue more for a different method (the k-means algorithm) in Section 14.1.
It is perfectly possible to use any other probability distribution instead of a Gaussian, but Gaussians are by far the most common choice. Then the output for any particular datapoint that is input to the algorithm will be the sum of the values expected by all of the M Gaussians:
f (x) =
M
X
m=1
αmφ(x; µm, Σm), (7.1)
153
FIGURE 7.1
Histograms of training data from a mixture of two Gaussians and two fitted models, shown as the line plot. The model shown on the left fits well, but the one on the right produces two Gaussians right on top of each other that do not fit the data well.
where φ(x; µm, Σm) is a Gaussian function with mean µmand covariance matrix Σm, and the αm are weights with the constraint that
M
P
m=1
αm= 1.
Figure 7.1 shows two examples, where the data (shown by the histograms) comes from two different Gaussians, and the model is computed as a sum or mixture of the two Gaussians together. The figure also gives you some idea of how to use the mixture model once it has been created. The probability that input xi belongs to class m can be written as (where a hat on a variable (ˆ·) means that we are estimating the value of that variable):
p(xi∈ cm) = αˆmφ(xi; ˆµm; ˆΣm)
M
P
k=1
ˆ
αmφ(xi; ˆµk; ˆΣk)
. (7.2)
The problem is how to choose the weights αm. The common approach is to aim for the maximum likelihood solution (the likelihood is the conditional probability of the data given the model, and the maximum likelihood solution varies the model to maximise this conditional probability). In fact, it is common to compute the log likelihood and then to maximise that; it is guaranteed to be negative, since probabilities are all less than 1, and the logarithm spreads out the values, making the optimisation more effective. The algorithm that is used is an example of a very general one known as the expectation-maximisation (or more compactly, EM) algorithm. The reason for the name will become clearer below. We will see another example of an EM algorithm in Section 16.3.3, but here we see how to use it for fitting Gaussian mixtures, and get a very approximate idea of how the algorithm works for more general examples. For more details, see the Further Reading section.
7.1.1 The Expectation-Maximisation (EM) Algorithm
The basic idea of the EM algorithm is that sometimes it is easier to add extra variables that are not actually known (called hidden or latent variables) and then to maximise the function over those variables. This might seem to be making a problem much more complicated than it needs to be, but it turns out for many problems that it makes finding the solution significantly easier.
In order to see how it works, we will consider the simplest interesting case of the Gaussian mixture model: a combination of just two Gaussian mixtures. The assumption now is that
data were created by randomly choosing one of two possible Gaussians, and then creating a sample from that Gaussian. If the probability of picking Gaussian one is p, then the entire model looks like this (where N (µ, σ2) specifies a Gaussian distribution with mean µ and standard deviation σ):
G1= N (µ1, σ21) G2= N (µ2, σ22)
y = pG1+ (1 − p)G2. (7.3)
If the probability distribution of p is written as π, then the probability density is:
P (y) = πφ(y; µ1, σ1) + (1 − π)φ(y; µ2, σ2). (7.4) Finding the maximum likelihood solution (actually the maximum log likelihood) to this problem is then a case of computing the sum of the logarithm of Equation (7.4) over all of the training data, and differentiating it, which would be rather difficult. Fortunately, there is a way around it. The key insight that we need is that if we knew which of the two Gaussian components the datapoint came from, then the computation would be easy. The mean and standard deviation for each component could be computed from the datapoints that belong to that component, and there would not be a problem. Although we don’t know which component each datapoint came from, we can pretend we do, by introducing a new variable f . If f = 0 then the data came from Gaussian one, if f = 1 then it came from Gaussian two.
This is the typical initial step of an EM algorithm: adding latent variables. Now we just need to work out how to optimise over them. This is the time when the reason for the algorithm being called expectation-maximisation becomes clear. We don’t know much about variable f (hardly surprising, since we invented it), but we can compute its expectation (that is, the value that we ‘expect’ to see, which is the mean average) from the data:
γi( ˆµ1, ˆµ2, ˆσ1, ˆσ2, ˆπ) = E(f | ˆµ1, ˆµ2, ˆσ1, ˆσ2, ˆπ, D)
= P (f = 1| ˆµ1, ˆµ2, ˆσ1, ˆσ2, ˆπ, D), (7.5) where D denotes the data. Note that since we have set f = 1 this means that we are choosing Gaussian two.
Computing the value of this expectation is known as the E-step. Then this estimate of the expectation is maximised over the model parameters (the parameters of the two Gaussians and the mixing parameter π), the M-step. This requires differentiating the expectation with respect to each of the model parameters. These two steps are simply iterated until the algorithm converges. Note that the estimate never gets any smaller, and it turns out that EM algorithms are guaranteed to reach a local maxima.
To see how this looks for the two-component Gaussian mixture, we’ll take a closer look at the algorithm:
The Gaussian Mixture Model EM Algorithm
Turning this into Python code does not require any new techniques:
while count<nits:
count = count + 1
# E-step
for i in range(N):
gamma[i] = pi*np.exp(-(y[i]-mu1)**2/(2*s1))/ (pi * np.exp(-(y[i]-' mu1)**2/(2*s1)) + (1-pi)* np.exp(-(y[i]-mu2)**2/2*s2))
# M-step
mu1 = np.sum((1-gamma)*y)/np.sum(1-gamma) mu2 = np.sum(gamma*y)/np.sum(gamma)
s1 = np.sum((1-gamma)*(y-mu1)**2)/np.sum(1-gamma)
FIGURE 7.2