Therefore, th e vector of com ponent loadings for th e first com ponent is the solution of th e equations
L i ( q ) = 2 c — 2 A q = 0 ,
and we find th a t th e first p artial least squares com ponent loadings are pro
portional to the sam ple covariance vector c . We have
c
q i = TTTT-
Consider the calculation of th e com ponent loadings q/c+i for th e k +
com ponent, w ith 1 < k < r^- The com ponent m ust m axim ize th e criterion
(c^q)^, subject to th e conditions q ^ q = 1 and q J C q = 0, for j
At this point, we are forced to change the notation for principal com po nents and we will w rite C = V E V ^ for th e principal com ponent decom posi tion of th e com plete d ata, such th a t V is the m atrix of principal com ponent coefficients. Using th e Stone-Brooks approach to p artial least squares de com position, we derive th e p artial least squares com ponent from a ro tatio n of the principal com ponent decom position of th e com plete data. Stone and Brooks give proof th a t th e principal components span th e sam e space as the p artial least squares components. Hence, we m ay w rite
q = Vr,
and we m u st now find th e optim um of th e Lagrangian equations
w here d = V ^ c , = ( a i , . . . , a^) and A = E V ^ Q , w ith Q = ( q i , . . . , q^t)
th e m atrix formed by th e first k vectors of p artial least squares com ponent
loadings. The optim um is the solution of
— = 2d — 2Ar — Ao; = 0.
Hence, absorbing th e constant 2 in th e Lagrange m ultipliers, we find th a t
r oc d — A(x
w ith A ^ r = 0 and therefore,
r oc (I — A ( A ^ A )“ ^A ^)d . We find M d r = ,|M d ||’ w ith M = I - A ( A ^ A ) - 'A ^ .
P a r tia l L ea st S q u a res C o m p u ta tio n s
T he above algebra is inefficient in its present form. This is because th e com
p u ta tio n of M involves th e inverting of a m atrix th a t m ust be recom puted
for each com ponent, in the sequence of p artial least squares com ponents. A naive approach to this com putation would involve, for each p artial least squares com ponent, th e com putation of th e principal com ponent decom po
sition of M . This would render th e cross-validatory algebra useless. T he
problem is th a t th e cross-validatory algebra does not yet take full account of th e leave-one-out n atu re of th e downdating. Stone and Brooks have shown th a t an efficient form ula can be found for the com putation of these inverse m atrices, using th e Sherman-M orisson theorem , which m ay be derived as a special case of theorem 2.2. We m ay employ th e upd atin g form ula
for M at each stage in th e cross-validatory com putations, where a is th e
colum n appended to A at stage k I. The details of this m ay be found in
The singular case m ay be dealt w ith using the same m ethods as explained for principal com ponent decomposition. Indeed, we are deriving th e p artial least squares com ponents from a rotation of th e principal com ponent decom position and hence, we m ay use th e same m ethods as before to derive the in itial principal com ponent decom position in th e singular case.
3.1.2
P a rtia l Least Squares and C ross-valid ation
P a rtia l least squares is more generally known as a class of p a th m odelling techniques in which th e interrelation of blocks of observed variables is m od elled through sets of laten t derived factors. The laten t variables are obtained through a sequence of least squares projections. The technique was origi
n ated by H erm an Wold [Wol66] [Wol85] in th e context of econom etric predic
tion m odelling and inherits from factor analysis techniques in psychom etry. T h e m ethodology has been much applied since, w ith applications in different branches of science.
In chem om etrics, p artial least squares has been successful in th e calibra tion of linear equations for prediction. In these applications, th e num ber of predictor variables is often huge, as in spectroscopic d a ta for exam ple, while th e num ber of observations is small. The usual form ulation of p a rtia l least squares in these applications is as an application of th e NIPALS algorithm , as described by Wold [WL69].
It is recent studies of the p artial least squares approach to regression
by Hoskuldsson [Hos88], Holland [Hel88] and Stone and Brooks [SB90] th a t
have shown th a t th e p artial least squares com ponent coefficient vectors m ay be in terp reted as directions in th e m easurem ent space th a t optim ize th e covariance w ith th e observed response. This strips p artial least squares from its algorithm ic form ulation. The in terp retatio n was used explicitly by Stone and Brooks [SB90] to derive th e p artial least squares com ponents from a ro ta tio n of th e principal components of th e predictor data. There has been no work on th e efficient cross-validation of p artial least squares regression. All im plem entations of cross-validation for p artial least squares are based, essentially, on W old’s approach to th e choice of th e num ber of com ponents in factor and principal com ponent models [Wol78], Thus, th ey consider th e NIPALS algorithm and subsets of d a ta rath er th an a full leave-one-out cross- validation.