Parametrizing a Learned Preditor

There are many ways to define the family of input-output functions, loss function, regularizer and optimization procedure, and the most common ones are described below, while more advanced ones are left to later chapters, in particular Chapters 10 and 13.

6.2.1 Family of Functions

The general idea of the family of functions defined by multi-layer neural networks is to compose affine transformations and non-linearities, where the choice of parameter values controls the degree and location of these non-linearities. As discussed in Section 6.4

below, with the appropriate choice of parameters, multi-layer neural networks can in principle approximate smoooth functions, with more hidden units allowing one to achieve better approximations

A multi-layer neural network with more than one hidden layer can be defined by generalizing the above structure, e.g., as follows, where we chose to use hyperbolic tangent² activation functions instead of sigmoid activation functions:

h^k= tanh(b^k + W^kh^k−1)

where h⁰ =x is the input of the neural net, h^k (for k > 0) is the output of the -thk hidden layer, which has weight matrix W^k and offset (or bias) vector b^k. If we want the output f_θ( ) to lie in some desired range, then we typically define anx output non-linearity on top of a definition such as the above Example 6.1.1. The non-non-linearity for the output layer is of course generally different from the tanh, depending on the type of output to be predicted and the associated loss function (see below).

As discussed in Chapter 19 (Section 19.5) and TODO there are several other non-linearities besides the sigmoid and the hyperbolic tangent which have been successfully used with neural networks. In particular, TODO discusses alternative non-linear units, such as the rectified linear unit (max(0 +, b w x· )) and the maxout unit (maxi(b_i+W_i·x)) which have been particularly successful in the case of deep feedforward or convolutional networks. These and other non-linearities or neural network computation commonly found in the literature are summarized below.

• Rectifierorrectified linear unit (RELU)orpositive part: ( ) = max(0 +h x , b w x· ).

• Hyperbolic tangent: ( ) = tanh( +h x b w x· ).

• Sigmoid: ( ) = 1 (1 +h x / e^{−( +b w x)·} ).

• Softmax: ( ) = softmax( +h x b W x). It is mostly used as output non-linearity for predicting discrete probabilities. See definition and discussion below, Eq. 6.1.

• Radial basis function orRBFunit: h x( ) =e^{−|| − ||w x 2/σ2}. This is heavily used in kernel SVMs (Boser et al., 1992; Sch¨olkopf et al., 1999) and has the advantage that such units can be easily initialized as a random subset of the input exam-ples (Powell, 1987; Niranjan and Fallside, 1990). The disadvantage of such units is that there is little meaningful generalization that is possible whenx is far from any of the RBF centers wof the hidden units (Bengio et al., 2006a).

• Softplus: this is a smooth version of the rectifier, introduced in Dugaset al. (2001) for function approximation and in Nair and Hinton (2010a) in RBMs. When they introduced the rectifier for deep nets, Glorot et al.(2011) compared the softplus and rectifier and found better results with the latter, in spite of the very similar shape and the differentiability and non-zero derivative of the softplus everywhere.

2which is linearly related to the sigmoid as follows: tanh( ) = 2x ×sigmoid(2 )x −1

• Hard tanh: this is shaped similarly to the tanh and the rectifier but unlike the latter, it is bounded, ( ) = max( 1 min(1h x − , , x)). It was introduced by Collobert (2004).

• Absolute value rectification: ( ) =h x | |x (may be applied on the dot product output or on the output of a tanh unit). It is also related to the rectifier and has been used for object recognition from images (Jarrett et al., 2009), where it makes sense to seek features that are invariant under a polarity reversal of the input illumination.

• Maxout: this is discussed in more detail in TODO It generalizes the rectifier but

introduces multiple weight vectors w_i(called filters) for each unit. ( ) = maxh x _i(b_i+ w_i· x).

This is not an exhaustive list but covers most of the non-linearities and unit computa-tions seen in the deep learning and neural nets literature. Many variants are possible.

As discussed in Section 6.3, the structure (also called architecture) of the family of input-output functions can be varied in many ways, which calls for a generic principle for efficiently computing gradients, described in that section. For example, a common variation is to connect layers that are not adjacent, with so-called skip connections, which are found in the visual cortex (in which the word “layer” should be replaced by the word “area”). Other common variations depart from a full connectivity between adjacent layers. For example, each unit at layer may be connected to only a subset ofk units at layer k−1. A particular case of such form of sparse connectivity is discussed in chapter 12 with convolutional networks. In general, the set of connections between units of the whole network only needs to form a directed acyclic graph in order to define a meaningful computation (see the flow graph formalism below, Section 6.3). When units of the network are connected to themselves through a cycle, one has to properly define what computation is to be done, and this is what is done in the case of recurrent networks, treated in Chapter 13.

6.2.2 Loss Function and Conditional Log-Likelihood

In the 80’s and 90’s the most commonly used loss function was the squared error L f( _θ( ) ) =x , y ||fθ( )x − ||y ². If f was unrestricted (non-parametric), minimizing the expected value of loss function over some data-generating distribution P X, Y( ) would yield f x( ) = E Y X[ | ], the true conditional expectation of Y given X³. This tells us what the neural network is trying to learn. Replacing the squared error by an absolute value makes the neural network try to estimate not the conditional expectation but the conditional median.

However, when yis a discrete label, i.e., for classification problems, other loss func-tions such as the Bernoulli negative log-likelihood⁴ have been found to be more ap-propriate than the squared error. In the case where y ∈ { , }0 1 is binary this gives

3Proving this is not difficult and is very instructive.

4often called cross entropy in the literature, even though the term cross entropy should also apply to many other losses that can be viewed as negative log-likelihood, discussed below in more detail. TODO:

where do we actually discuss this? may as well discuss it in maximum likelihood section

L f( _θ( ) ) = x , y −ylogf_θ( )x −(1− y) log(1− fθ( )) and it can be shown that the optimalx f minimizing this criterion is ( ) =f x P Y ( = 1| ). Note that in order for the abovex expression of the criterion to make sense f_θ( ) must be strictly between 0 and 1 (anx undefined or infinite value would otherwise arise). To achieve this, it is common to use the sigmoid as non-linearity for the output layer, which matches well with the Binomial negative log-likelihood criterion⁵

Learning a Conditional Probability Model

More generally, one can define a loss function as corresponding to a conditional log-likelihood, i.e.,

L f( _θ( ) ) = logx , y − P_θ(Y|X .)

For example, if Y is a continuous random variable and we assume that, given X, it has a Gaussian distribution with mean f_θ( ) and varianceX σ², then

− log Pθ(Y|X) = 1

2(f_θ( )X − Y)²/σ² + log(2πσ²).

Up to an additive and multiplicative constant (which would give the same choice of θ), this is therefore equivalent to the squared error loss. Once we understand this principle, we can readily generalize to other distributions, as appropriate. For example, it is straightforward to generalize the univariate Gaussian to the multivariate case, and under appropriate parametrization consider the variance to be a parameter or even parametrized function of x (for example with output units that are garanteed to be positive, or forming a positive definite matrix).

Similarly, for discrete variables, the Binomial negative log-likelihood criterion corre-sponds to the conditional log-likelihood associated with the Bernoulli distribution with probability p = f_θ( ) of generatingx Y = 1 given X = x (and probablity 1− p of generating Y = 0):

− log Pθ(Y|X) = −1Y =1 log p−1Y =0log(1−p) = −Y logf_θ( )X − −Y(1 ) log(1−fθ( ))X . When Y is discrete (in some finite set, say 1{ , . . . N}) but not binary, the Bernoulli distribution is extended to the Multinoulli (Murphy, 2012)⁶. This distribution is spec-ified by a vector of N probabilities that sum to 1, each element of which provides the probability p_i =P Y( =i X| ). We thus need a vector-valued output non-linearity that produces such a vector of probabilities, and a commonly used non-linearity for this purpose is thesoftmax non-linearity introduced earlier.

p= softmax( )a ⇐⇒ pi= e^ai



je^aj . (6.1)

5In reference to statistical models, this “match” between the loss function and the output non-linearity is similar to the choice of alink functionin generalized linear models (McCullagh and Nelder, 1989).

6The Multinoulli is also known as the categorical or generalized Bernoulli distribution, which is a special case of the Multinomial distribution with one trial, and the term multinomial is often used even in this case

where typically a = b+ W h is the vector of scores whose elements a_i are associated with each category i. The corresponding loss function is therefore L p, y( ) = −logp_y. Note how minimizing this loss will push a_y up (increase the score a_y associated with the correct label ) while pushingy a_i(for = ) down (decreasing the score of the otheriy labels, in the context ). The first effect comes from the numerator of the softmax whilex the second effect comes from the normalizing denominator. These forces cancel on a specific example only if p_y = 1 and they cancel in average over examples (say sharing the same x) if piequals the fraction of times that Y = i for this value . The samex principles can be seen at play in the training of Markov Random Fields, Boltzmann machines and RBMs, in Chapter 9. Note other interesting properties of the softmax.

First of all, the gradient of the Multinoulli negative log-likelihood with respect to the softmax argument ais

∂

∂ai ( log− py) = (pi− δiy)

which does not saturate, i.e, the gradient does not vanish when the output probabilities approach 0 or 1, except when the model is providing the correct answer. On the other hand, other loss functions such as the squared error applied to sigmoid outputs (which was popular in the 80’s and 90’s) will have vanishing gradient when an output unit sat-urates, even if the output is completely wrong. Another property that could potentially be interesting is that the softmax output is invariant under additive changes of its input vector: softmax( ) = softmax( + ) whena a b b is a scalar added to all the elements of vector .a

In general, for any parametric probability distribution (P Y ω| ) with parameters , weω can construct a conditional distribution P Y( |X) by makingω a parametrized function of X, and learning that function:

P Y ω( | = fθ( ))X

where f_θ( ) is theX outputof a predictor, X is its input, and Y can be thought of as a target. This established choice of words comes from the common cases of classification and regression, where f_θ( ) is really a prediction associated withX Y, or its expected value. However, in generalω=f_θ( ) may contain parameters of the distribution ofX Y other than its expected value. For example, it could contain its variance or covariance, in the case where Y is conditionally Gaussian. In the above examples, with the squared error loss, ω is the mean of the Gaussian which captures the conditional distribution of Y (which means that the variance is considered fixed, not a function of X). In the common classification case, ω contains the probabilities associated with the various events of interest. For example, in the case whereY is binary (i.e. Bernoulli distributed), it is enough to specify = (ω P Y = 1|X). In the Multinoulli case, ωis generally specified by a vector of probabilities summing to 1, e.g., via the softmax non-linearity discussed above.

Once we view things in this way, we automatically get as the natural cost function the negative log-likelihood L X, Y( ) = log (− P Y|ω = f_θ( )). Besides , there could beX θ other parameters that control the distribution of . For example, we may wish to learnY

the variance of a conditional Gaussian, even when the latter is not a function ofX. In this case, the solution can be computed analytically because the maximum likelihood estimator of variance is simply the empirical mean of the squared difference between observations Y and their expected value (here f_θ( )). In other words, the conditionalX variance can be estimated from the mean squared error.

Multiple Output Variables

WhenY is actually a tuple formed by multiple random variables Y₁, Y₂, . . . Y_k, then one has to choose an appropriate form for their joint distribution, conditional onX= . Thex simplest and most common choice is to assume that the Y_i are conditionally independent, i.e.,

P Y( 1, Y2, . . . Yk|x) =

k i=1

P Y( i|x .)

This brings us back to the single variable case, especially since the log-likelihood now decomposes into a sum of terms log (P Yi|x). If each (P Yi|x) is separately parametrized (e.g. a different neural network), then we can train these neural networks independently.

However, a more common and powerful choice assumes that the different variables Yi

share some common factors that can be represented in some hidden layer of the network (such as the top hidden layer). See Sections 6.5 and 10.5 for a deeper treatment of the notion of underlying factors of variation and multi-task training.

If the conditional independence assumption is considered too strong, what can we do?

At this point it is useful to step back and consider everything we know about learning a joint probability distribution. Any probability distribution P_ω( ) parametrized byY parameters ω can be turned into a conditional distribution P_θ(Y|x) by making ω a function ω = f_θ( ) parametrized by . This is not only true of the simple parametricx θ distributions we have seen above (Gaussian, Bernoulli, multinoulli) but also of more complex joint distributions typically represented by a graphical model. Much of this book is devoted to such graphical models (see Chapters 9, 15, 16, 17). For more concrete examples of such structured output models with deep learning, see Section 20.4.

6.2.3 Training Criterion and Regularizer

The loss function (often interpretable as a negative log-likelihood) tells us what we would like the learner to capture. Maximizing the conditional log-likelihood over the true distri-bution, i.e., minimizing the expected loss E_X,Y[ log− P_θ(Y|X)], makes Pθ(Y|X) estimate the trueP Y( |X) associated with the unknown data generating distribution, within the boundaries of what the chosen family of functions allows. In practice we cannot min-imize this expectation because we do not know P X, Y( ) and because computing and mininizing this integral exactly would generally be intractable. Instead we are going to approximately minimize a training criterion C θ( ) based on the empirical average of the loss (over the training set). The simplest such criterion is the average training loss

1 n

n

i=1L f( _θ(x_i), y_i), where the training set is a set ofnexamples (x_i, y_i). However, bet-ter results can often be achieved by crippling the learner and preventing it from simply

finding the best that minimizes the average training loss. This means that we combineθ the evidence coming from the data (the training set average loss) with some a priori preference on the different values that θcan take (the regularizer). If this concept (and the related concepts of generalization, overfitting and underfitting) are not clear, please return to Section 5.3 for a refresher.

The most common regularizer is simply an additive term equal to a regularization coefficientλtimes the squared norm of the parameters, || ||θ ²₂. This regularizer is often called the weight decay or L2 weight decay or shrinkage because adding the squared L2 norm to the training criterion pushes the weights towards zero, in proportion to their magnitude. For example, when using stochastic gradient descent, each step of the update with regularizer term λ θ|| ||²would look like

θ← −θ ∂L f( _θ( ) )x , y

∂θ − λθ

where is the learning rate (see Section 4.3). This can be rewritten θ← (1 −θ λ)−∂L f( _θ( ) )x , y

∂θ

where we see that the first term systematically shaves off a small proportion of beforeθ adding the gradient.

Another commonly used regularizer is the so-called L1 regularizer, which adds a term proportional to the L1 (absolute value) norm, |θ|1 =

i|θi|. The L1 regularizer also prefers values that are small, but whereas the L2 weight decay has only a weak preference for 0 rather than small values, the L1 regularizer continues pushing the parameters towards 0 with a constant gradient even when they get very small. As a consequence, it tends to bring some of the parameters to exactly 0 (how many depends on how large the regularization coefficient λis chosen). When optimizing with an approximate iterative and noisy method such as stochastic gradient descent, no actual 0 is obtained. On the other hand, the L1 regularizer tolerates large values of some parameters (only additively removing a small quantity compared to the situation without regularization) whereas the L2 weight decay aggressively punishes and prevents large parameter values. The L1 and L2 regularizers can be combined, as in the so-called elastic net (Zou and Hastie, 2005), and this is commonly done in deep networks ⁷.

Note how in the context of maximum likelihood, regularizers can generally be in-terpreted as Bayesian priors on the parameters P θ) or on the learned function, as( discussed in Section TODO . Later in this book we discuss regularizers that are data-dependent (i.e., cannot be expressed easily as a pure prior), such as the contractive penalty (Section 10.1) as well as regularization methods that are difficult to interpret simply as added terms in the training criterion, such as dropout TODO

7See the Deep Learning Tutorials at http://deeplearning.net/tutorial/gettingstarted.html#

l1-and-l2-regularization

6.2.4 Optimization Procedure

Once a training criterion is defined, we enter the realm of iterative optimization proce-dures, with the slight twist that if possible we want to monitor held-out performance (for example the average loss on a validation set, which usually does not include the regularizing terms) and not just the value of the training criterion. Monitoring vali-dation set performance allows one to stop training at a point where generalization is estimated to be the best among the training iterations. This is called early stopping and is a standard machine learning technique, discussed in Section 19.5.1.

Section 4.3 has already covered the basics of gradient-based optimization. The sim-plest such techique is stochastic gradient descent (with minibatches), in which the pa-rameters are updated after computing the gradient of the average loss over a minibatch of examples (e.g. 128 examples) and making an update in the direction opposite to that gradient (or opposite to some accumulated average of such gradients, i.e., the mo-mentum technique, reviewed in Section 19.5.4). The most commonly used optimization procedures for multi-layer neural networks and deep learning in general are either vari-ants of stochastic gradient descent (typically with minibatches), second-order methods (the most commonly used being LBFGS and nonlinear conjugate gradients) applied on large minibatches (e.g. of size 10000) or on the whole training set at once, or natural gradient methods (Amari, 1997; Park et al., 2000; Pascanu and Bengio, 2013). Exact second-order and natural gradient methods are computationally too expensive for large neural networks because they involve matrices of dimension equal to the square of the number of parameters. Approximate methods are discussed in Section 8.7.

On smaller datasets or when computing can be parallelized, second-order methods have a computational advantage because they are easy to parallelize and can still perform many updates per unit of time. On larger datasets (and in the limit of an infinite dataset, i.e., a stream of training examples) one cannot afford to wait for seeing the whole dataset before making an update, so that favors either stochastic gradient descent variants (possibly with minibatches to take advantage of fast or parallelized implementations of matrix-matrix products) or second-order methods with minibatches.

A more detailed discussion of issues arising with optimization methods in deep learn-ing can be found in Chapter 8. In particular, many design choices in the construction of the family of functions, loss function and regularizer can have a major impact on the difficulty of optimizing the training criterion. Furthermore, instead of using generic optimization techniques, one can design optimization procedures that are specific to the learning problem and chosen architecture of the family of functions, for example by initializing the parameters of the final optimization routine from the result of a differ-ent optimization (or a series of optimizations, each initialized from the previous one).

Because of the non-convexity of the training criterion, the initial conditions can make a very important difference, and this is what is exploited in the various pre-training strategies, described in Sections 8.6 and 10.3.