Unsupervised and Transfer Learning
10.1 Auto-Encoders
The idea of auto-encoders has been part of the historical landscape of neural networks for decades (LeCun, 1987; Bourlard and Kamp, 1988; Hinton and Zemel, 1994) but has really picked up speed in recent years, in part due to what was an incomplete understanding of the mathematical and geometrical underpinnings and interpretation of auto-encoders, which are developed further in Chapters 14 and 17.9.
input!x!
code!h!
reconstruc,on!r!
Decoder.
Encoder.
Figure 10.1: General schema of an auto-encoder, mapping an inputxto an output (called reconstruction) r through an internal representation or code . The auto-encoder hash two components: the encoder (mapping xto ) and the decoder (mappingh h to ).r
An auto-encoder is simply a neural network that tries to copy its input to its output. The architecture of an auto-encoder is typically decomposed into the following parts, illustrated in Figure 10.1:
• an input, x
• an encoder function f
• a “code” or internal representation h= ( )f x
• a decoder function g
• an output, also called “reconstruction” = (r g h) = ( ( ))g f x
• a loss functionL computing a scalar (L r, x) measuring how good of a reconstruc-tion r is of the given input . The objective is to minimize the expected value ofx L over the training set of examples { }x .
.
10.1.1 Regularized Auto-Encoders
Predicting the input may sound useless: what could prevent the auto-encoder from simply copying its input into its output? In the 20th century, this was achieved by constraining the architecture of the auto-encoder to avoid this, by forcing the dimension of the code h to be smaller than the dimension of the input .x
input!x!
Code*bo,leneck!h:!
undercomplete*
representa4on*
reconstruc4on!r!
Decoder*
Encoder*
Decoder*
Encoder*
Code!h:!
overcomplete*
representa4on*
Figure 10.2: Left: undercomplete representation (dimension of code h is less than di-mension of input ). Right: overcomplete representation. Overcomplete auto-encodersx require some other form of regularization (instead of the constraint on the dimension of h) to avoid the trivial solution where r= for all .x x
Figure 10.2 illustrates the two typical cases of auto-encoders: undercomplete vs over-complete, i.e., with the dimension of the representationh respectively smaller vs larger than the input . Whereas early work with auto-encoders uses the undercompletenessx – i.e. a bottleneck in the sequence of layers – to avoid learning the identity function, more recent work allows overcomplete representations. What we have learned in recent years is that it is possible to make the auto-encoder meaningfully capture the structure of the input distribution even if the representation is overcomplete, so as other forms of constraint or regularization are employed. In fact, once you realize that auto-encoders
can capture the input distribution (indirectly, not as a an explicit probability function), you realize that it should need more capacity as one increases the complexity of the distribution to be captured (and the amount of data available): it should not be limited by the input dimension.
Besides thebottleneckconstraint, alternative constraints or regularization methods have been explored and can guarantee that the auto-encoder does something useful and not just learn some trivial identity-like function:
• Sparsity of the representation or of its derivative: even if the intermediate representation has a very high dimensionality, the effective local dimensionality (number of degrees of freedom that capture a coordinate system among the prob-able ’s) could be much smaller if most of the elements ofx h are zero (or any other constant, such that ||∂h∂xi|| is close to zero. When ||∂h∂xi|| is close to zero, hi does not participate in encoding local changes in . There is a geometrical interpretationx of this situation in terms of manifold learning that is discussed in more depth in Chapter 14. That discussion also explains how an auto-encoder naturally tends towards learning a coordinate system for the actual factors of variation in the data. At least four types of “auto-encoders” clearly fall in this category of sparse representation:
– Sparse coding (Olshausen and Field, 1996) has been heavily studied as an unsupervised feature learning and feature inference mechanism. It looks for representations that are both sparse and explain the input through the decoder. In general there is no parametric encoder and the representation is instead considered a free variable that is obtained through an optimization, i.e., a particular form of inference:
h = ( ) = arg minf x
h
L g h , x( ( ) ))
where L is the reconstruction loss, f the (non-parametric) encoder, g the (parametric) decoder, and in practice the minimization can be approximate.
Sparse coding has a manifold or geometric interpretation that is discussed in Section 14.2.2. It also has an interpretation as a directed graphical model, and it is described in more details in Section 16.3. To achieve sparsity, the optimized objective function includes a term that is minimized when the representation has many zero or near-zero values, such as the L1 penalty
| |h1 =
i|hi|.
– An interesting variation of sparse coding combines the freedom to chose the representation through optimization and a parametric encoder. It is called predictive sparse decomposition (PSD) (Kavukcuoglu et al., 2008) and is briefly described in Section 14.2.
– At the other end of the spectrum are simplysparse auto-encoders, which combine with the standard auto-encoder schema a sparsity penalty which encourages the output of the encoder to be sparse. These are also described in
Section 14.2. Besides the L1 penalty, other sparsity penalties that have been explored include the Student-t penalty (Olshausen and Field, 1996; Bergstra,
2011),
i
log(1 + α2h2i)
(i.e. where αhi has a Student-t prior density) and the KL-divergence penalty (Lee et al., 2008; Goodfellowet al., 2009; Larochelle and Bengio, 2008)
−
i
( logt hi+ (1− t) log(1− hi)),
with a target sparsity level t (for hi ∈ (0 1), e.g. through a sigmoid non-, linearity).
– Contractive autoencoders (Rifai et al., 2011), covered in Section 14.4, explicitly penalize ||∂h∂x||2F, i.e., the sum of the squared norm of the vectors
∂hi( )x
∂x (each indicating how much each hidden unit hi responds to changes in x and what direction of change inx that unit is most sensitive to, around a particular ). With such a regularization penalty, the auto-encoder is calledx contractive1 because the mapping from input to representation, h x( ), is encouraged to be contractive, i.e., to have small derivatives in all directions.
Note that a sparsity regularization indirectly leads to a contractive mapping as well, when the non-linearity used happens to have a zero derivative when hi= 0 (which is the case for the sigmoid non-linearity).
• Robustness to injected noise: if noise is injected in inputs or hidden units, or if some inputs are missing, while the neural network is asked to reconstruct the clean and complete input, then it cannot simply learn the identity function. It has to capture the structure of the data distribution in order to optimally perform this reconstruction. Such auto-encoders are called denoising auto-encoders and are discussed in more detail in Section 14.3. There is a tight connection between the denoising auto-encoders and the contractive auto-encoders: it can be shown that in the limit of small Gaussian injected noise, the denoising reconstruction error is equivalent to a contractive penalty on the reconstruction function r = ( ( )).g f x In other words, since both x and x+(where is some small noise vector) must yield the same target output , the reconstruction function is encouraged to bex insensitive to changes in all directions . The only thing that prevents it from simply being a constant (completely insensitive to the input, is that one also has to reconstruct different values around different training examples x. However, the auto-encoder can learn to be approximately constant around training examples x while producing a different answer for different training examples. As discussed in Section 14.1, if the examples are near a low-dimensional manifold, this encour-ages the representation to vary only on the manifold and be locally constant in
1A function ( ) is contractive iff x ||f x − f y ||( ) ( ) < x|| − ||y for nearby xand , or equivalently if itsy derivative ||f( )x|| <1.
directions orthogonal to the manifold, i.e., the representation locally captures a (not necessarily Euclidean, not necessarily orthogonal) coordinate system for the manifold. In addition to the denoising auto-encoder the variational auto-encoder (Section 17.7.1) and the generative stochastic networks (Section 17.9) also involve the injection of noise, but typically in the representation-space itself, thus intro-ducing the notion ofh as alatent variable.
• Pressure of a Prior on the Representation: an interesting way to generalize the notion of regularization applied to the representation is to introduce in the cost function for the auto-encoder a log-prior term
log ( )P h
which captures the assumption that we would like to find a representation that has a simple distribution, or at least one that is simpler than the original data distribution. Among all the encoding functions , we would like to pick one thatf
1. can be inverted (easily), and this is achieved by minimizing some reconstruc-tion loss, and
2. yields representations whose distribution is “simpler”, i.e., can be capturedh with less capacity than the original training distribution itself.
The sparse variants described above clearly fall in that framework. The varia-tional auto-encoder (Section 17.7.1) provides a clean mathematical framework for justifying the above pressure of a top-level prior when the objective is to model the data generating distribution.
10.1.2 Representational Power, Layer Size and Depth
Nothing in the above description of auto-encoders restricts the encoder or decoder to be shallow, but in the literature most trained auto-encoders have had a single hidden layer which is also the representation layer or code.
For one, we know by the usual universal approximator abilities of single hidden-layer neural networks that a sufficiently large hidden hidden-layer can represent any function with a given accuracy. This observation justifies overcomplete auto-encoders: in order to represent a rich enough distribution, one probably needs many hidden units in the inter-mediate representation layer. We also know that Principal Components Analysis (PCA) corresponds to an undercomplete auto-encoder with no intermediate non-linearity, and that PCA can only capture a set of directions of variation that are the same every-where in space. This notion is discussed in more details in Chapter 14 in the context of manifold learning.
For two, it has also been reported many times that training a deep neural network, and in particular a deep auto-encoder (i.e. with a deep encoder and a deep decoder) is more difficult than training a shallow one. This was actually a motivation for the ini-tial work on the greedy layerwise unsupervised pre-training procedure, described below in Section 10.3, by which we only need to train a series of shallow auto-encoders in order to
initialize a deep auto-encoder. It was shown early on (Hinton and Salakhutdinov, 2006) that, if trained properly, such deep auto-encoders could yield much better compression than corresponding shallow or linear auto-encoders. As discussed in Section 11.3, deeper architectures can be in some cases exponentially more efficient (both in terms of com-putation and statistically) than shallow ones. However, if we can usefully pre-train a deep net by training and stacking shallow ones, it is interesting to consider single-layer auto-encoders, as has been done in most of the literature discussed in Chapter 14.
10.1.3 Reconstruction Distribution
The above “parts” (encoder function , decoder function , reconstruction loss ) makef g L sense when the loss L is simply the squared reconstruction error, but there are many cases where this is not appropriate, e.g., when x is a vector of discrete variables or when P x h( | ) is not well approximated by a Gaussian distribution 2. Just like in the case of other types of neural networks (starting with the feedforward neural networks, Section 6.2.2), it is convenient to define the loss as a negative log-likelihood over someL target random variables. This probabilistic interpretation is particularly important for the discussion in Chapter 17.9, where the output of the auto-encoder is interpreted as a probability distribution P x h( | ), for reconstructing , given hidden units . Thisx h distribution captures the uncertainty about the original x (which gave rise to , eitherh deterministically or stochastically, given . In the simplest and most ordinary cases, thish distribution factorizes, i.e., P x h( | ) =
iP x( i|h). This covers the usual cases of xi|h being Gaussian (for unbounded real values) and xi|h having a Bernoulli distribution (for binary values xi), but one can readily generalize this to other distributions, such as mixtures.
Thus we can generalize the notion of decoding functiong h( ) to decoding distribution P x h( | ). Similarly, we can generalize the notion of encoding function f x( ) to encoding distribution Q h x( | ). In that case, we use this to capture the fact that noise is injected at the level of the representation , now considered like a latent variable. This general-h ization is crucial in the development of the variational auto-encoder (Section 17.7.1).
We also find a stochastic encoder and a stochastic decoder in the RBM, described below.
10.2 RBMs
A restricted
2See the link between squared error and normal density in Chapter 5.5