by Douglas G McIlwraith, Haralambos Marmanis, and Dmitry Babenko
x 1 x 2 theta weighted sum sign of weighted sum AND x
6.5 Going deeper: from multilayer neural networks to deep learning
6.5.3 RBMS in action
In this section we’ll use a modified version of the logistic classification problem pre- sented in the scikit-learn documentation.2 Full credit must be given to Dauphin, 1 Kevin Swersky, Bo Chen, Ben Marlin, and N de Freitas, “A Tutorial on Stochastic Approximation Algorithms for Training Restricted Boltzmann Machines and Deep Belief Nets,” Information Theory and Applications Workshop (ITA), 2010 (San Diego: ITA, 2010) 1–10.
2 scikit-learn, “Restricted Boltzmann Machine features for digit classification,” http://scikit-learn.org/ stable/auto_examples/neural_networks/plot_rbm_logistic_classification.html#example-neural-networks-plot- rbm-logistic-classification-py (accessed December 22, 2015).
E V H( , ) = –
i
j(wi, , ,j vi hj)+
i bivi+
j,cjhjP V H( , ) = e–E(V H, )⁄Z
v h,
( ) e–E(v h, )
Log(P V H( , )) = Log(e–(V H, )–LogZ W( ) )
Log(P V( )) =
HLog(e–(V H, )–LogZ W( ) )v h,
Niculae, and Synnaeve for this illustrative example, and we won’t stray far from their material here. As in previous examples, we’ll omit the import block and concentrate on the code. The full listing can be found in the supporting content.
digits = datasets.load_digits() X = np.asarray(digits.data, 'float32') X, Y = nudge_dataset(X, digits.target)
X = (X - np.min(X, 0)) / (np.max(X, 0) + 0.0001) # 0-1 scaling X_train, X_test, Y_train, Y_test = train_test_split(X,
Y,test_size=0.2,random_state=0)
The first thing we do is to load in the dataset, but we actually do more than this. From the original dataset, we generate further artificial samples by nudging the dataset with linear shifts of one pixel, normalizing each so that each pixel value is between 0 and 1. So, for every labeled image, a further four images are generated—shifted up, down, right, and left respectively—each with the same label, that is, which number the image represents. This allows training to learn better representations of the data using such a small data set, specifically representations that are less dependent on the script being centralized within the image. This is achieved using the nudge_dataset func- tion, defined in the following listing.
def nudge_dataset(X, Y): """
This produces a dataset 5 times bigger than the original one, by moving the 8x8 images in X around by 1px to left, right, down, up """ direction_vectors = [[[0, 1, 0],[0, 0, 0],[0, 0, 0]], [[0, 0, 0],[1, 0, 0],[0, 0, 0]], [[0, 0, 0],[0, 0, 1],[0, 0, 0]], [[0, 0, 0],[0, 0, 0],[0, 1, 0]]] shift = \
lambda x, w: convolve(x.reshape((8, 8)), mode='constant',\ weights=w).ravel()
X = np.concatenate([X] + \
[np.apply_along_axis(shift, 1, X, vector) for vector in \ direction_vectors])
Y = np.concatenate([Y for _ in range(5)], axis=0) return X, Y
Given this data, it’s now simple to create a decision pipeline consisting of an RBM, fol- lowed by logistic regression. The next listing presents the code to both set up this pipeline and train the model.
Listing 6.11 Creating your dataset
91
Going deeper: from multilayer neural networks to deep learning
# Models we will use
logistic = linear_model.LogisticRegression() rbm = BernoulliRBM(random_state=0, verbose=True)
classifier = Pipeline(steps=[('rbm', rbm), ('logistic', logistic)]) ####################################################################### # Training
# Hyper-parameters. These were set by cross-validation,
# using a GridSearchCV. Here we are not performing cross-validation to # save time.
rbm.learning_rate = 0.06 rbm.n_iter = 20
# More components tend to give better prediction performance, but larger # fitting time
rbm.n_components = 100 logistic.C = 6000.0
# Training RBM-Logistic Pipeline classifier.fit(X_train, Y_train) # Training Logistic regression
logistic_classifier = linear_model.LogisticRegression(C=100.0) logistic_classifier.fit(X_train, Y_train)
This code is taken directly from Scikit-learn,1 and there are some important things to
note. The hyper-parameters, that is, the parameters of the RBM, have been selected specially for the dataset in order to provide an illustrative example that we can discuss. More details can be found in the original documentation.
You’ll see that beyond this, the code does very little. A classifier pipeline is set up consisting of RBM followed by a logistic regression classifier, as well as a standalone logistic regression classifier for comparison. In a minute you’ll see how these two approaches perform. The following listing provides the code to do this.
print("Logistic regression using RBM features:\n%s\n" % ( metrics.classification_report(
Y_test,
classifier.predict(X_test))))
print("Logistic regression using raw pixel features:\n%s\n" % ( metrics.classification_report(
Y_test,
logistic_classifier.predict(X_test))))
The output of this provides a detailed summary of the two approaches, and you should see that the RBM/LR pipeline far outstrips the basic LR approach in precision,
Listing 6.13 Setting up and training an RBM/LR pipeline
1 Scikit-learn, “Bernoulli RBM.”
recall, and f1 score. But why is this? If we plot the hidden components of the RBM, we should start to understand why. The next listing provides the code to do this, and fig- ure 6.14 provides a graphical overview of the hidden components of our RBM.
plt.figure(figsize=(4.2, 4))
for i, comp in enumerate(rbm.components_): #print(i) #print(comp) plt.subplot(10, 10, i + 1) plt.imshow(comp.reshape((8, 8)), cmap=plt.cm.gray_r,interpolation='nearest') plt.xticks(()) plt.yticks(())
plt.suptitle('100 components extracted by RBM', fontsize=16) plt.subplots_adjust(0.08, 0.02, 0.92, 0.85, 0.08, 0.23) plt.show()
In our RBM, we have 100 hidden nodes and 64 visible units, because this is the size of the images being used. Each square in figure 6.14 is a grayscale interpretation of the weights between that hidden component and each visible unit. In a sense, each hid- den component can be thought of as recognizing the image as given previously. In the pipeline, the logistic regression model then uses the 100 activation probabilities
( for each i) as its input; thus, instead of doing logistic regres-
sion over 64 raw pixels, it’s performed over 100 inputs, each having a high value when the input looks close to the ones provided in figure 6.14. Going back to the first sec- tion in this chapter, you should now be able to see that we’ve created a network that has automatically learned some intermediate representation of the numbers, using an RBM. In essence, we’ve created a single layer of a deep network! Imagine now what could be achieved with deeper networks and multiple layers of RBMs to create more intermediate representations!
Listing 6.15 Representing the hidden units graphically
100 components extracted by RBM
Figure 6.14 A graphical representation of the weights between the hidden and visible units in our RBM. Each square represents a single hidden unit, and the 64 grayscale values within represent the weights from that hidden value to all the visible units. In a sense this dictates how well that hidden variable is able to recognize images like the one presented.
93
Conclusions
6.6
Conclusions
In this chapter
We provided you a whistle-stop tour of neural networks and their relationship to deep learning. Starting with the simplest neural network model, the MCP model, we moved on to the perceptron and discussed its relationship with logis- tic regression.
We found that it’s not possible to represent non-linear functions using a single perceptron, but that it is possible if we create multilayer perceptrons (MLP).
We discussed how MLPs are trained through backpropagation—and the adop- tion of differentiable activation functions—and provided you with an example whereby a non-linear function is learned using backpropagation in PyBrain.
We discussed the recent advances in deep learning, specifically, building multi- ple layers of networks that can learn intermediate representations of the data.
We concentrated on one such network known as a Restricted Boltzmann Machine, and we showed how you can construct the simplest deep network over the digits dataset, using a single RBM and a logistic regression classifier.
web users leave behind as they interact with your pages and applications. You can unlock those insights by using intelligent algorithms like the ones that have earned Facebook, Google, Twitter, and Microsoft a place among the giants of web data pattern extraction. Improved search, data classification, and other smart pattern matching techniques can give you an enormous advantage when you need to understand and interact with your users.
Algorithms of the Intelligent Web, Second Edition teaches the most important approaches to algorithmic web data analysis, enabling you to create your own machine learning applications that crunch, munge, and wrangle data collected from users, web applications, sensors, and website logs. In this totally-revised edition, you’ll look at intelligent algorithms through the lens of machine learning, with code examples that show you how to extract value from their data. Key machine learning concepts are explained and introduced with code examples in Python’s scikit-learn. This book guides you through the underlying machinery and intelligent algorithms to capture, store, and structure data streams coming from the web. You’ll explore recommenda- tion engines from the example of Netflix movie recommendations and dive into classi- fication via statistical algorithms, neural networks, and deep learning. You’ll also consider the ins and outs of ranking and how to test applications based on intelligent algorithms.
What’s inside
Machine learning for newbies
Algorithms for IoT, public health, and other areas
An introduction to deep learning and neural networks