2.7 Summary
3.1.2 Sparse spectral techniques
In this section, some of the available sparse techniques for manifold learning are described. These techniques focus on retaining the local similarities measured in the input space via the solution of a sparse (generalized) eigenproblem.
Locally linear embedding (LLE)
A low-dimensional manifold constructed with LLE [180] aims to preserve the local neigh- borhoods of the high-dimensional data in the low-dimensional learned space. LLE is simi- lar to Isomap considering that both approaches construct a graph representation of the data points. However, Isomap solely attempts to preserve the local properties of the data, as- suming locally linear relationship between neighboring data points. It represents every data point xias a weighted combination of its k nearest neighbors in the high-dimensional space.
This defines a set of weights wi j for the k neighbors of xi and the aim is to find a low-
dimensional representation yi that respects this weighting. The LLE objective function is
φ(Y) =
∑
i yi− k∑
j=1 wi jyi j 2 subject to y (k) 2 = 1 for ∀k , (3.7)where y(k) represents the kth column of the solution matrix Y. The constraint on the co- variance of the columns of Y is required to exclude the trivial solution Y = 0. Using a sparse weight matrix W, it can be shown that the embedding can be obtained from the d eigenvectors corresponding to the smallest nonzero eigenvalues of(I − W)T(I − W) [180].
Hessian LLE
Using the same concept of local linearity, Hessian LLE [57] minimizes the curvature of the high-dimensional manifold when learning the low-dimensional representation. The method constrains the distances in both spaces to be locally isometric. Applying PCA to every data point xi and its k nearest neighbors gives an approximation of the local tangent space at
every data point. The mapping function M obtained from the d principal components at every point xiis then used to obtain an estimator for the Hessian Hiof the manifold at that
data point [57]. From the Hessian estimators in tangent space, a matrix
H
is constructed with entries:H
lm=∑
i∑
j (Hi)jl× (Hi)jm . (3.8)The eigenvectors that correspond to the d smallest eigenvectors of
H
are used to define the low-dimensional embedding Y that minimizes the curvature of the manifold.Laplacian eigenmaps
Laplacian eigenmaps can be used to find a low-dimensional representation of the data while preserving the local geometric properties of the manifold [17]. Laplacian Eigenmaps uses a local neighborhood graph to approximate geodesic distances between data points. This graph is defined by either connecting every data item xito its k closest neighbors or to all
subjects within some fixed radiusε. From these distances a sparse neighborhood graph G
ing points xi and xj in G (zero elsewhere) according to the distance between the points is
computed using a Gaussian heat kernel:
wi, j = K(xi, xj) = exp − xi− xj 2 2σ2 ! . (3.9)
Here σ is the standard deviation of the Gaussian kernel. Laplacian eigenmaps aims to place points xi and xj close together in the low-dimensional space if their weight wi, j is high, e.g. if they are close in the original, high-dimensional space. This is done by means of minimizing the cost function given by
φ(Y) = argmin
∑
i, j
kyi− yjk2wi, j, (3.10)
under the constraint that yTDy= 1 which removes an arbitrary scaling factor in the embed- ding and prevents the trivial solution where all yi are zero. The minimization of Equation (3.10) can be formulated as an eigenproblem [10] through the computation of the degree matrix M and the Laplacian L. The degree matrix M is a diagonal matrix that contains information about the degree of each vertex of W, where mi,i= ∑jwi, j and the Laplacian
L= M − W. Hence the low-dimensional manifold Y that represents all the data points can
be obtained via solving a generalized eigenproblem
Lν = λMν , (3.11)
whereν and λ are the eigenvectors and eigenvalues, and in turn the d eigenvectors ν corre- sponding to the smallest (non-zero) eigenvaluesλ represent the new coordinate system.
3.1.3
Summary
Some of the advantages of the dense spectral techniques is that they can achieve a more faith- ful representation of the data’s global structure and that their metric-preserving properties are better understood theoretically [54]. Sparse spectral techniques have two main advan- tages: Computational efficiency as they involve only sparse matrix computations which may
yield a polynomial speedup and representational capacity as they may give useful results on a broader range of manifolds, whose local geometry is close to Euclidean, but whose global geometry may not be [54]. An additional consideration that must be taken into consider- ation when choosing a dimensionality reduction technique is that some application require the mapping of new points into the learned manifold: Linear dimensionality reduction, such as PCA, provides a projection matrix for exact transformation between the original and the embedded space. This is not the case for most non-linear methods and approximation tech- niques must be used. Bengio et al. [18] proposed an out of sample embedding technique, that employs the Nystr¨om approximation [163], for dimensionality reduction techniques that rely on an eigendecomposition.