Linear Algebra Methods for Data Mining

Linear Algebra Methods

for Data Mining

Saara Hyv¨onen, [email protected]

Spring 2007

QR decomposition

• Any matrix A ∈ _Rm×n, m ≥ n, can be transformed to upper triangular

form by an orthogonal matrix:

A = Q

R 0

How? By using the Givens rotation:

Let x be a vector. The parameters c and s, c2 + s2 = 1, can be chosen

so that multiplication of x by G =      1 0 0 0 0 c 0 s 0 0 1 0 0 −s 0 c     

θ

x

”Skinny” QR decomposition

A = 1 1 1

1 2 4

1 3 9

1 4 16

[Q,R]=qr(A) %the QR decomposition

Q = -0.5000 0.6708 0.5000 0.2236 -0.5000 0.2236 -0.5000 -0.6708 -0.5000 -0.2236 -0.5000 0.6708 -0.5000 -0.6708 0.5000 -0.2236 R = -2.0000 -5.0000 -15.0000 0 -2.2361 -11.1803 0 0 2.0000 0 0 0

[Q,R]=qr(A,0) %the skinny version of QR Q = -0.5000 0.6708 0.5000 -0.5000 0.2236 -0.5000 -0.5000 -0.2236 -0.5000 -0.5000 -0.6708 0.5000 R = -2.0000 -5.0000 -15.0000 0 -2.2361 -11.1803 0 0 2.0000

Uniqueness?

Suppose A ∈ _Rm×n has full column rank (i.e. the column vectors are

linearly independent). The ”skinny” QR factorization

A = Q₁R₁

is unique where Q₁ has orthonormal columns and R₁ is upper triangular

What is QR good for?

• It will come up when we discuss the computation of some matrix

decompositions

• It can also be used for solving the least squares problem.

Least squares for linear regression

• Consider problems of the following form: given a number of

measure-ments X = [x₁...x_n] and an outcome y, build a linear model

ˆ y = b₀ + n X j=1 x_jb_j,

which uses the measurements X to predict the outcome y.

• X = clinical measures of patients, y = level of cancer specific antigen.

• X = atmospheric measurements of each day, y = occurence of

Least squares for linear regression

• The model yˆ = b₀ + Pn_j=1 x_jb_j can be written in the form

ˆ

y = XTb, b = (b₁ ... b_n b₀)T.

To do this, one must append X = [x₁ ... x_n x_n+1], where x_n+1 is a

vector of ones.

• Fitting a linear model to data is usually done using the method of least

squares.

Example

We have measurement data:

x 1 2 3 4 5

0 1 2 3 4 5 6 6 8 10 12 14 16 18 20 22

Example (continued)

x 1 2 3 4 5

y 7.97 10.2 14.2 16.0 21.2

We wish to find α and β such that αx + β = y. Thus

α + β = 7.97 2α + β = 10.2 3α + β = 14.2 4α + β = 16.0 5α + β = 21.2

Example (continued)

Overdetermined! (More equations than unknowns.) Solve using the least squares method.

Example

• X = atmospheric measurements of each day: temperature, UV-A:

X = t₁ t₂ ... t_n r₁ r₂ ... r_n = temperatures UV-A measurements

• y = which month each day belongs to. Choices are:

August (y = 0) or September (y = 1).

• Looking for b such that yˆ = XTb and yˆ ≈ y.

−1.5 −1 −0.5 0 0.5 1 1.5 −1.5 −1 −0.5 0 0.5 1 1.5

August days (b) and September days (r)

UV

−

The least squares problem

• Let A ∈ _Rm×n, m ≥ n. The system

Ax = b

is called overdetermined: more equations than unknown. Usually such

Example

What to do?

• make the residual vector

r = b − Ax

as small as possible. But how?

• Make r orthogonal to the columns of A:

rT a₁ a₂ ... a_n = rTA = 0.

• Now write r = b − Ax to get the normal equations

Example

C=A’*A %Normal equations

C= 55 15

15 5

x=C\(A’*b)

x= 3.2260

• If the column vectors of A are linearly independent, then the normal equations

ATAx = ATb

are non-singular and have a unique solution

• BUT: the normal equations have two significant drawbacks:

– forming ATA leads to loss of information.

– the condition number of ATA is the square of that of A:

Example

Worse example

Solving least squares problem using QR

Partition Q = (Q₁ Q₂), where Q₁ ∈ _Rm×n, and denote

QTb = b₁ b := QT₁ b QTb .

Then krk2 =k b₁ b₂ − R 0 x)k2 = kb₁ − Rxk2 + kb₂k2.

Minimize krk by making the first term equal to zero: i.e. solve

LS by QR

Theorem. Let A ∈ _Rm×n have full column rank and a thin

QR-decomposition A = Q₁R. Then the least squares problem

min

x kAx − bk2

has the unique solution

Example

R = -7.4162 -2.0226

0 -0.9535

x=R\(Q1’*b)

x = 3.2260

Back to this example:

• X = atmospheric measurements of each day: temperature, UV-A:

X = t₁ t₂ ... t_n r₁ r₂ ... r_n = temperatures UV-A measurements

• y = which month each day belongs to. Choices are:

August (y = 0) or September (y = 1).

• Looking for b such that yˆ = XTb and yˆ ≈ y.

−1.5 −1 −0.5 0 0.5 1 1.5 −1.5 −1 −0.5 0 0.5 1 1.5

August days (b) and September days (r)

UV

−

Example

%X (transpose of) data matrix, 1st col: temperature, 2nd col: UV-A %y month vector, y(j)=0 if August and 1 if September

[length(find(y==0)) length(find(y==1))] = 116 99

%number of August and September days

size(X) = 215 2 %size of data matrix

Xap=[X ones(215,1)]; [Q1,R]=qr(Xap,0);

b=R\(Q1’*y) %this is the same as b=inv(R)*(Q1’*y)

b= -0.4355

yhat=Xap*b; %least squares solution

[min(yhat) max(yhat)] = -0.3506 1.2436

alpha=0.5; %day classified as September if x’y>alpha

length(find(1.0*(yhat>alpha)-y)) = 34 %misclassifications

x=-1:0.01:1;db=(alpha-b(3)-b(1)*x)/b(2);%decision boundary x’b=alpha %attn: in reality, choose alpha with more care!

scatter(Xnorm(:,1),Xnorm(:,2),20,y,’filled’) hold;plot(x,db,’k’)

−1.5 −1 −0.5 0 0.5 1 1.5 −1.5 −1 −0.5 0 0.5 1 1.5

August days (b) and September days (r)

UV

−

Updating the solution of the LS problem

Assume we have reduced the matrix and the right hand side

(A b) → QT(A b) = R QT₁ b 0 QT₂ b .

From this the solution of the LS problem is readily available.

Assume we have not saved Q.

We then get a new observation (a b), a ∈ _Rn, b ∈ _R.

Updating the solution of the LS problem

No. Instead, write the reduction on the previous slide as

A b aT b → QT 0 0 1 A b aT b =    R QT₁ b 0 QT₂ b aT b   .

References

[1] Lars Eld´en: Matrix Methods in Data Mining and Pattern Recognition,

SIAM 2007.

[2] G. H. Golub and C. F. Van Loan. Matrix Computations. 3rd ed. Johns Hopkins Press, Baltimore, MD., 1996.

[3] T. Hastie, R. Tibshirani, J. Friedman: The Elements of Statistical Learning. Data mining, Inference and Prediction, Springer Verlag, New York, 2001.