Matrix Decomposition and Its Application in Statistics NK

(1)

1 1

Matrix Decomposition and its

Application in Statistics

Nishith Kumar

Lecturer

Department of Statistics

Begum Rokeya University, Rangpur.

Email

(2)

(3)

2 2

%vervie&

'' (ntro)uction

(ntro)uction

'' LU )ecomposition

LU )ecomposition

'' *R )ecomposition

*R )ecomposition

'' +holesky )ecomposition

+holesky )ecomposition

'' or)an Decomposition

or)an Decomposition

'' Spectral )ecomposition

Spectral )ecomposition

'' Singular value )ecomposition

Singular value )ecomposition

'' -pplications

-pplications

(4)

3 3

Introduction

Some

Some of of most most freuently freuently use) use) )ecompositions )ecompositions are are thethe LULU,, QR QR ,, Cholesky

Cholesky,, JordanJordan,, Spectral decompositionSpectral decomposition an) an) Singular alueSingular alue decompositions

decompositions..

/his Lecture covers relevant matri0 )ecompositions, !asic

numerical metho)s, its computation an) some of its applications.

Decompositions

Decompositions provi)e

provi)e a

a numerically

numerically sta!le

sta!le &ay

&ay to

to solve

solve

a system of linear euations, as sho&n alrea)y in 12ampler,

3#4"5,

3#4"5, an)

an) to

to invert

invert a

a matri0. -))itionally

matri0. -))itionally, they

, they provi)e

provi)e an

an

important tool for analy6ing the numerical sta!ility of a system.

(5)

4 4

Easy to solve system (Cont.)

Some linear system that can be easily solved Some linear system that can be easily solved

The solution: The solution:             nn nn n n aa b b a a b b a a b b 77 77 77 88 88 8 8 33 33 3 3  

(6)

5

Easy to solve system (Cont.)

Lower triangular matrix:

(7)



Easy to solve system (Cont.)

Upper Triangular Matrix:

(8)

!

LU Decomposition

an) 2here,             = mm m m u u u u u u U        " " " ₈₈ ₈ 3 38 33             = mm m m l l l l l l L        8 3 88 83 33 " " " LU A

=

LU )ecomposition &as originally )erive) as a )ecomposition of ua)ratic an) !ilinear forms. Lagrange, in the very first paper in his collecte) &orks9 34:#; )erives the algorithm &e call <aussian elimination. Later /uring intro)uce) the LU )ecomposition of a matri0 in 3#=> that is use) to solve the system of linear euation.

Let A !e a m × m &ith nonsingular suare matri0. /hen there e0ists t&o

matrices L an) U such that, &here L is a lo&er triangular matri0 an) U is an upper triangular matri0.

?L Lagrange

934@A 3>3@;

-. C. /uring 93#38?3#:=;

(9)

" "

A

A ≈≈…… ≈≈U U

9upper triangular;

⇒

⇒ _U_U

_

_E_E_k_k⋅⋅⋅⋅⋅⋅_E_E₃₃_A_A ⇒

⇒_A_A

_{ 9}

_E_E₃₃

_;;

−−₃₃ ⋅⋅⋅⋅⋅⋅

9 9

_E_E

kk

;;

−−33 U U

(f each such elementary matri0

E E _ii

is a lo&er triangular matrices,

it can !e prove) that 9

E E ₃₃

;;

−−33

,,

⋅⋅⋅⋅⋅⋅

, 9

E E _kk

;;

−−33

are lo&er triangular, an)

99

E E ₃₃

;;

−−33 ⋅⋅⋅⋅⋅⋅

9 9

E E _kk

;;

−−33

is a lo&er triangular matri0.

Let

L= L=

99

E E ₃₃

;;

−−33 ⋅⋅⋅⋅⋅⋅

9 9

E E _kk

;;

−−33

then

A=LU. A=LU.

o& to

o& to )ecompose -

)ecompose -LUF

LUF

          − − − − − −           − − − −           − − = =           − − − − − − ⇒ ⇒           − − − − − −           − − − − = =           − − − − − −           − − − − − − = = 88 3@ 3@ @@ AA >> 38 38 88 88 AA 33 "" 88 77 33 "" 33 88 "" "" 33 33 @@ "" "" 33 "" "" "" 33 :: "" "" 88 == "" 88 88 AA 88 3@ 3@ @@ AA >> 38 38 88 88 AA 33 "" 88 77 33 "" 33 88 "" "" 33 33 38 38 "" 88 == "" 88 88 AA No&, No&, 88 3@ 3@ @@ AA >> 38 38 88 88 AA A A U

(10)

# #

Calculation o!

Calculation o! L

L and

and U (cont.)

U (cont.)

No& re)ucing

No& re)ucing the first column &the first column &e havee have

          −− −− −− == 8 8 3@ 3@ @ @ A A > > 38 38 8 8 8 8 A A A A           −− −− −−           8 8 3@ 3@ @ @ A A > > 38 38 8 8 8 8 A A 3 3 " " " " " " 3 3 " " " " " " 3 3           −− −− −−           −− −−           −− ==           −− −− −− ⇒ ⇒           −− −− −−           −− −− ==           −− −− −− 8 8 3@ 3@ @ @ A A > > 38 38 8 8 8 8 A A 3 3 " " 8 8 77 3 3 " " 3 3 8 8 " " " " 3 3 3 3 @ @ " " " " 3 3 " " " " " " 3 3 : : " " " " 8 8 = = " " 8 8 8 8 A A 8 8 3@ 3@ @ @ A A > > 38 38 8 8 8 8 A A 3 3 " " 8 8 77 3 3 " " 3 3 8 8 " " " " 3 3 3 3 38 38 " " 8 8 = = " " 8 8 8 8 A A $ $

(11)

1% 1%

(f

(f - - is is a a Non Non singular singular matri0 matri0 then then for for eacheach L L 9lo&er triangular matri0; the upper 9lo&er triangular matri0; the upper

triangular matri0 is uniue !ut an

triangular matri0 is uniue !ut an LU LU )ecomposition is not uniue. /here can )ecomposition is not uniue. /here can

!e more than one

!e more than one suchsuch LU LU )ecomposition for a matri0. Such as )ecomposition for a matri0. Such as

Calculation o!

Calculation o! L

L and

and U (cont.)

U (cont.)

          ==                     ==           −−           −− −− −− −− 3 3 @ @ 8 8 77 3 3 " " 3 3 8 8 " " " " 3 3 3 3 @ @ " " " " 3 3 " " " " " " 3 3 3 3 " " 8 8 77 3 3 " " 3 3 8 8 " " " " 3 3 3 3 @ @ " " " " 3 3 " " " " " " 3 3 3 3 " " 8 8 77 3 3 " " 3 3 8 8 " " " " 3 3 33 33           −− −− −− == 8 8 3@ 3@ @ @ A A > > 38 38 8 8 8 8 A A A A           33 @@ 88 77 33 "" 33 88 "" "" 33           −− −− −− : : " " " " 8 8 = = " " 8 8 8 8 A A         − − − − = = 88 3@ 3@ @@ AA >> 38 38 88 88 AA A A           33 @@ @@ "" 33 38 38 "" "" AA         − − − − :: "" "" 88 == "" AA 77 88 AA 77 88 33 No& No& /herefore, /herefore, $ $ =LU =LU $ $ =LU =LU

(12)

11 11

alculation of

alculation of L

L and

and U (cont.)

U (cont.)

/hus

LU LU

)ecomposition is not uniue. Since &e compute

LU LU

)ecomposition !y elementary transformation so if &e change

L then U &ill !e change)

L then U &ill !e change) such that -LU

such that -LU

/o fin) out the uniue

LU LU

)ecomposition, it is necessary to

put some restriction on

L L

an)

U U

matrices. Gor e0ample, &e can

reuire the lo&er triangular matri0

L L

to !e a unit one 9i.e. set

all the entries of its main )iagonal to ones;.

LU

LU Decomposition Decomposition in in R"R"

'' li!rary9Catri0;li!rary9Catri0;

'' 0H?matri09c9@,8,3, #,@,=,=,8,: ;,ncol@,nro&@;0H?matri09c9@,8,3, #,@,=,=,8,: ;,ncol@,nro&@; '' e0pan)9lu90;;e0pan)9lu90;;

Calculation o!

(13)

12 ' #ote there are also generali6ations of LU to non?suare an) singular

matrices, such as rank revealing LU factori6ation.

' 1Ian, +./. 98""";. %n the e0istence an) computation of rank revealing LU factori6ations. Linear Algebra and its Applications, @3A 3##?888.

' Ciranian, L. an) <u, C. 98""@;. Strong rank revealing LU factori6ations. Linear Algebra and its Applications, @A4 3?3A.5

'

Uses" /he LU )ecomposition is most commonly use) in the solution of

systems of simultaneous linear euations. 2e can also fin) )eterminant easily !y using LU )ecomposition 9Iro)uct of the )iagonal element of upper an) lo&er triangular matri0;.

(14)

13

Soling system o! linear e$uation

using LU decomposition

Suppose &e &oul) like to solve a mJm system -  !. /hen &e can fin) a LU?)ecomposition for -, then to solve A X =b, it is enough to solve the

systems

/hus the system LY = b can !e solve) !y the metho) of for&ar) su!stitution an) the system U X  Y can !e solve) !y the metho) of

!ack&ar) su!stitution. /o illustrate, &e give some e0amples +onsi)er the given system A X  b, &here

an)           − − − = 8 3@ @ A > 38 8 8 A A           − = 34 3= > b

(15)

14 2e have seen A = LU , &here

/hus, to solve A X = b, &e first solve LY = b !y for&ar) su!stitution

/hen

Soling system o! linear e$uation

using LU decomposition

          = 3 @ 8 7 3 " 3 8 " " 3 L           − − − = : " " 8 = " 8 8 A U           − =                     34 3= > 3 @ 8 7 3 " 3 8 " " 3 @ 8 3 y y y           − − =           = 3: 8 > @ 8 3 y y y Y

(16)

15 No&, &e solve U X =Y !y !ack&ar) su!stitution

then

Soling system o! linear e$uation

using LU decomposition

          − − =                     − − − 3: 8 > : " " 8 = " 8 8 A @ 8 3 x x x           =           @ 8 3 @ 8 3 x x x

(17)

1

QR Decomposition

(f A is a m×n matri0 &ith linearly in)epen)ent columns, then A can !e

)ecompose) as , &here * is a m×n matri0 &hose columns form an orthonormal !asis for the column space of Aan) R is an

nonsingular upper triangular matri0.

! A =

J%rgen &edersen 'ram

93>:" 3#3A; (rhard Schmidt_93>4A?3#:#;

Girstly *R )ecomposition originate) &ith <ram93>>@;. Later Erhar) Schmi)t 93#"4; prove) the *R Decomposition

(18)

1!

*R?Decomposition 9+ont.;

/heorem  (f A is a m×n matri0 &ith linearly in)epen)ent columns, then

A can !e )ecompose) as , &here * is a m×n matri0 &hose

columns form an orthonormal !asis for the column space of A an) R is an

nonsingular upper triangular matri0.

Iroof Suppose -1u3  u8 . . .  un5 an) rank 9 A;  n.

-pply the <ram?Schmi)t process to Mu3, u8, . . . ,un an) the

orthogonal vectors v3, v8, . . . ,vnare

Let for i=1,2,. . ., n. /hus ₃, ₈, . . . ,_n form a orthonormal

!asis for the column space of A.

! A = 3 8 3 3 8 8 8 8 3 8 3 3 , , , − − −

−

=

_i i i i i i i i " " " u " " " u " " " u u "  i i i " " #

=

(19)

1"

*R?Decomposition 9+ont.;

No&,

i.e.,

/hus ui is orthogonal to # $ for $%i&

3 8 3 3 8 8 8 8 3 8 3 3 , , , − − −

+

=

i i i i i i i i " " " u " " " u " " " u " u  3 3 8 8 3 3 , , ,

+

₋ ₋

+

=

⇒

u_i "_i #_i u_i # # u_i # #  u_i #_i #_i N , , M N , , , M ₃ ₈ _i _i ₈ _i i span " " " span # # # u

∈



=

 3 3 8 8 3 3 8 8 @ 3 3 @ @ @ @ 3 3 8 8 8 8 3 3 3 , , , , , , − − + + + + = + + = + = = n n n n n n n n " # u # # u # # u # # u # # u # # u # " u # # u # " u # " u  

(20)

1#

Let = 1#1 #2 . . . #n5 , so  is a m×n matri0 &hose columns form an

orthonormal !asis for the column space of A .

No&,

i.e., A=!.

2here,

/hus - can !e )ecompose) as A=! ' &here R is an upper triangular an)

nonsingular matri0.

*R?Decomposition 9+ont.;

[

]

[

]













=

n n n n n n " # u " # u # u " # u # u # u " # # # u u u A " " " " , " " , , " , , , @ @ 8 8 @ 8 3 3 @ 3 8 3 8 3 8 3                           = n n n n " # u " # u # u " # u # u # u " ! " " " " , " " , , " , , , @ @ 8 8 @ 8 3 3 @ 3 8 3        

(21)

2%

QR Decomposition

Example:

Gin) the *R )ecomposition of













−

=

3 "

"

3

3 "

"

3

3 A

(22)

21 -pplying <ram?Schmi)t process of computing *R )ecomposition

)st Step" *nd Step" +rd Step"

Calculation o! QR Decomposition

              = = = = " @ 3 @ 3 @ 3 3 @ 3 3 3 3 33 a a # a r @ 8 8 3 38 = # a = − r (               − − = = = =              − − =               − −              − − = − = − = " A 7 3 @ 8 A 7 3 O O 3 @ 8 O " @ 7 3 @ 7 8 @ 7 3 " @ 3 @ 3 @ 3 ; @ 7 8 9 " 3 " 3 O 8 8 8 8 88 38 3 8 8 3 3 8 8 # # # # r r # a a # # a # (

(23)

22 ,th Step" -th Step" .th Step"

Calculation o! QR Decomposition

@ 3 @ 3 3@ = # a = − r ( A 3 @ 8 8@ = # a = r (               − − = = = =              − − = − − = − − = A 7 8 A 7 3 " A 7 3 O O 3 8 7 A O 3 8 7 3 " 8 7 3 O @ @ @ @ @@ 8 8@ 3 3@ @ @ 8 8 @ 3 3 @ @ # # # # r # r # r a a # # a # # a # ( (

(24)

23 /herefore, A=!

R code !or QR Decomposition"

0H?matri09c93,8,@, 8,:,=, @,=,#;,ncol@,nro&@; rstr H? r90;

*H?r.*9rstr; RH?r.R9rstr;

Uses *R )ecomposition is &i)ely use) in computer co)es to fin) the

eigenvalues of a matri0, to solve linear systems, an) to fin) least suares appro0imations.

Calculation o! QR Decomposition

          ₋ ₋               − − − − =             − − − − 8 7 A " " A 7 3 A 7 8 " @ 7 3 @ 7 8 @ A 7 8 " " A 7 3 A 7 3 @ 7 3 " A 7 8 @ 7 3 A 7 3 A 7 3 @ 7 3 3 " " " 3 3 " " 3 3 3 3

(25)

24

Least s$uare solution using QR

Decomposition

/he least suare solution of

b

is

Let

)=!.

/hen

/herefore,

(

)

t

)

b

=

)

t

Y

( )

! ! !b

( )

! !  Y !b  Y * Y  ! !b !t

=

t t

⇔

t −3 t

=

t −3 t t

⇔

=

t

=

(

)

( ) ( )

Y



!

Y

)

!b

!

!b



!

b

!

b

)

t t t t t t t t

=

(26)

25

Cholesky Decomposition

+holesky )ie) from &oun)s receive) on the !attle fiel) on @3 -ugust 3#3> at : oPclock in the morning in the North of Grance. -fter his )eath one of his fello& officers, +omman)ant Benoit, pu!lishe) +holeskyPs metho) of computing solutions to the normal euations for some least suares )ata fitting pro!lems pu!lishe) in the +ulletin g,odesi#ue in 3#8=. 2hich is kno&n as +holesky Decomposition

+holesky Decomposition (f - is a real, symmetric an) positive )efinite matri0 then there e0ists a uniue lo&er triangular matri0 L &ith positive )iagonal element such that ( .

LL A

=

(27)

2

+holesky Decomposition

/heorem (f - is a n×n real, symmetric an) positive )efinite matri0 then there e0ists a uniue lo&er triangular matri0 - &ith positive )iagonal

element such that .

Iroof Since - is a n×n real an) positive )efinite so it has a LU

)ecomposition, A=LU . -lso let the lo&er triangular matri0 L to !e a unit one 9i.e. set all the entries of its main )iagonal to ones;. So in that case LU )ecomposition is uniue. Let us suppose

o!serve that . is a unit upper triangular matri0.

/hus, A=L/ ( .Since - is Symmetric so, A=A( . i.e., L/ ( =/L( . Grom

the uniueness &e have L=/ . So, A=LL( . Since - is positive )efinite so

all )iagonal elements of D are positive. Let then &e can &rite A=--( .

( -- A = ; , , , 9u₃₃ u₈₈ unn diag  =  U . / ( = −3 ; , , , 9 d 33 d 88 d nn diag L -= 

(28)

2!

Cholesky Decomposition 0Cont12

&rocedure 3o !ind out the cholesky decomposition Suppose

2e nee) to solve

the euation

            = nn n n n n a a a a a a a a a A        8 3 8 88 83 3 38 33                                        ( L nn n n L nn n n nn n n n n l l l l l l l l l l l l a a a a a a a a a A                         =             = " " " " " " 8 88 3 83 33 8 3 88 83 33 8 3 8 88 83 3 38 33

(29)

2"

(xample o! Cholesky Decomposition

Suppose

/hen +holesky Decomposition

No&, 8 7 3 3 3 8       − =

∑

− = k s ks kk kk a l l           − − = : 8 8 8 3" 8 8 8 = A           = @ 3 3 " @ 3 " " 8 L Gor k from 3 to n

Gor $ from k01 to n kk

k s ks $s $k $k a l l l l       ₋ =

∑

− = 3 3

(30)

2#

R code !or Cholesky Decomposition

' x4/matrix0c0,5*5/*5 *5)65*5 /*5*5-25ncol7+5nro87+2 ' cl4/chol0x2

'

I! 8e Decompose A as LDLTthen

an)

          − = 3 @ 7 3 8 7 3 " 3 8 7 3 " " 3 L           = @ " " " # " " " = .

(31)

3%

Application o! Cholesky

Decomposition

+holesky Decomposition is use) to solve the system

of linear euation

Ax=b'

&here - is real symmetric

an) positive )efinite.

(n regression analysis it coul) !e use) to estimate the

parameter if

) ( )

is positive )efinite

.

(n Kernel principal component analysis, +holesky

)ecomposition is also use) 92eiya ShiQ ue?Gei

<uoQ 8"3";

(32)

31

Characteristic Roots and

Characteristics 9ectors

-ny non6ero vector x is sai) to !e a characteristic vector of a matri0 A, (f there e0ist a num!er  such that Ax= xQ

2here A is a suare matri0, also then  is sai) to !e a characteristic root of the matri0 A correspon)ing to the characteristic vector x.

+haracteristic root is uniue !ut characteristic vector is not uniue.

2e calculate characteristics root  from the characteristic euation -? 3=4 Gor = _i the characteristics vector is the solution of x from the follo&ing

homogeneous system of linear euation 9-? i(;0"

/heorem (f A is a real symmetric matri0 an) _i an) _T are t&o )istinct latent root of - then the correspon)ing latent vector 0_ian) 0_T are orthogonal.

(33)

32

Multiplicity

Alge:raic Multiplicity"

/he num!er of repetitions of a certain

eigenvalue. (f, for a certain matri0, M@,@,=, then the

alge!raic multiplicity of @ &oul) !e 8 9as it appears t&ice; an)

the alge!raic multiplicity of = &oul) !e 3 9as it appears once;.

/his type of multiplicity is normally represente) !y the <reek

letter , &here 9i; represents the alge!raic multiplicity of i.

'eometric Multiplicity"

the

geometric multiplicity

of an

eigenvalue is the num!er of linearly in)epen)ent eigenvectors

associate) &ith it.

(34)

33

Jordan Decomposition

Camille &ordan (1"!%)

' Let - !e any nJn matri0 then there e0ists a nonsingular matri0 I an) 5 6 78

a kJk matri0 form Such that             = λ λ λ λ         " " " " 3 " " " 3 ; 9 k 5













=

−

;

9 "

"

;

9 "

"

;

9

8 3 3 8 3 r k k k r 5 5 5 A9 9 λ λ λ      

&here k ₁0k ₂0 … 0 k _r =n. -lso _i ' i=1'2'. . .' r are the characteristic roots -n) k _i are the alge!raic multiplicity of i '

or)an Decomposition is use) in Differential euation an) time series analysis.

+amille or)an 93>@>?3#83;

(35)

34

Spectral Decomposition

Let

A

!e a

m × m

real symmetric matri0. /hen

there e0ists an orthogonal matri0

9 such that

or

, &here

:

is a )iagonal

matri0.

9 A9

=

Λ

( ( 9 9 A

=

Λ

CAUC;<5 A1L10)=>?/)>-=2

-. L. +auchy esta!lishe) the Spectral

Decomposition in 3>8#.

(36)

35

Spectral Decomposition and

&rincipal component Analysis 0Cont12

By using spectral )ecomposition &e can &rite

(n multivariate analysis our )ata is a matri0. Suppose our )ata is

)

matri0. Suppose  is mean centere) i.e.,

an) the variance covariance matri0 is V. /he variance covariance

matri0 V is real an) symmetric.

Using spectral )ecomposition &e can &rite

;=9:9 (.

2here

:

is

a )iagonal matri0.

-lso

tr9

;

;  /otal variation of Data tr9

:

;

( 9 9 A

=

Λ

; 9

−

µ

→

) ) ; , , , 9 ₃ ₈ _n diag λ λ  λ = Λ n λ λ λ _≥ _≥ _≥



8 3

(37)

3

/he Irincipal component transformation is the transformation

9?W;

9

2here,



E9

_i

;"



X9

_i

;

_i 

+ov9

_i

,

_$

;" if

_i

Y

_$ 

X9

₃

; Z X9

₈

; Z . . . Z X9

_n

;

 

Spectral Decomposition and

&rincipal component Analysis 0Cont12

∑

= Σ = n i i tr Y < 3 ; 9 ; 9

∏

= Σ = n i i Y < 3 ; 9

(38)

3!

R code !or Spectral Decomposition

0H?matri09c93,8,@, 8,:,=, @,=,#;,ncol@,nro&@; eigen90;

Application"

 Gor Data Re)uction.

 _{(mage Irocessing an) +ompression.}  K?Selection for K?means clustering  Cultivariate %utliers Detection

 Noise Giltering

(39)

3"

/here are five mathematicians &ho &ere responsi!le for esta!lishing the e0istence of the singular value )ecomposition an) )eveloping its theory.

istorical !ackgroun) of SXD

Eugenio Beltrami 93>@:?3>##; +amille or)an 93>@>?3#83; ames oseph Sylvester 93>3=?3>#4; Erhar) Schmi)t 93>4A?3#:#; ermann 2eyl 93>>:?3#::;

/he Singular Xalue Decomposition &as originally )evelope) !y t&o mathematician in the mi) to late 3>""[s

3. Eugenio Beltrami , 8.+amille or)an

Several other mathematicians took part in the final )evelopments of the SXD inclu)ing ames oseph Sylvester, Erhar) Schmi)t an) ermann 2eyl &ho stu)ie) the SXD into the mi)?3#""[s.

+.Eckart an) <. oung prove lo& rank appro0imation of SXD 93#@A;.

(40)

3#

2hat is SXDF

Any real 0m@n2 matrix X 5 8here 0n≤ m25 can :e decomposed5

X = U! T

 _{U is a 0m"n2 column orthonormal matrix 0U}T U7I25

containing the eigenectors o! the symmetric matrix XX T 1

_{ is a 0n"n 2 diagonal matrix5 containing the singular}

alues o! matrix X 1 3he num:er o! non ero diagonal elements o!  corresponds to the rank o! X 1

 _!T is a 0n"n 2 ro8 orthonormal matrix 0! T ! 7I25

containing the eigenectors o! the symmetric matrix X T X 1

(41)

4%

/heorem 9Singular Xalue Decomposition;  Let

)

!e

m×n

of rank

r'

r  n  m

. /hen there e0ist matrices

U

,

<

an) a )iagonal matri0



, &ith positive )iagonal elements such that,

&roo!"

Since

)

is

m × n

of rank

r' r  n  m

. So

)) (

an)

) ( )

!oth

of rank

r

9 !y using the concept of <rammian matri0 ; an) of

)imension

m × m

an)

n × n

respectively. Since

)) (

is real

symmetric matri0 so &e can &rite !y spectral )ecomposition,

2here



an)



are respectively, the matrices of characteristic

vectors an) correspon)ing characteristic roots of

)) (

.

-gain since

) ( )

is real symmetric matri0 so &e can &rite !y

spectral )ecomposition,

Singular 9alue Decomposition 0Cont12

( < U ) = Λ ( (  )) = ( ( !/! ) )

=

(42)

41

2here R is the 9orthogonal; matri0 of characteristic vectors an) C

is )iagonal matri0 of the correspon)ing characteristic roots.

Since

)) (

an)

) ( )

are !oth of rank

r

, only

r

of their characteristic

roots are positive, the remaining !eing 6ero. ence &e can

&rite,

-lso &e can &rite,

Singular 9alue Decomposition 0Cont12













=

" " " r . .













=

" " " r / /

(43)

42

2e kno& that the non6ero characteristic roots of

)) (

an)

) ( )

are

eual so

Iartition



,

!

conforma!ly &ith



an)

/

, respectively

i.e.,

Q

such that

r

is

m × r

,

!r

is

n × r

an)

correspon) respectively to the non6ero characteristic roots of

)) (

an)

) ( )

. No& take

2here

are the positive characteristic roots of

)) (

an) hence those of

) ( )

as &ell 9!y using the concept of

grammian matri0.;

Singular 9alue Decomposition 0Cont12

r r / . = ; , 9 _\  = _r ! = 9 !_r, R _\; r r ! <  U = = ; , , , 9 ₃378 ₈378 378 8 7 3 r r diag d d d .

=

Λ

=

 r i d _i, =3,8,_,

(44)

43

No& )efine,

No& &e shall sho& that

>=)

thus completing the proof.

Similarly,

Grom the first relation a!ove &e conclu)e that for an ar!itrary orthogonal matri0, say I3 ,

2hile from the secon) &e conclu)e that for an ar!itrary orthogonal matri0, say I 8

2e must have

Singular 9alue Decomposition 0Cont12

( r r r . !  > = 378 ) ) !/! ! / ! ! . ! ! .   . ! ! .  ! .  > > ( ( ( r r r ( r r r ( r r r ( r r r ( r r r ( ( r r r ( = = = = = = 9 ; 8 7 3 8 7 3 8 7 3 8 7 3 ( ( )) >> = ) 9 >

=

₃ )9 >

(45)

44

/he prece)ing, ho&ever, implies that for ar!itrary orthogonal

matrices

9 1 ' 9 2

the matri0

)

satisfies

2hich in turn implies that,

/hus

Singular 9alue Decomposition 0Cont12

8 8 3 3 )) 9 , ) ) 9 ) )9 9 )) (

=

( ( (

=

( ( n m 9 2 2 9 = = 8 3 , ( ( r r r . ! U <  > )

=

378

=

Λ

(46)

45

R +o)e for Singular Xalue Decomposition

0H?matri09c93,8,@, 8,:,=, @,=,#;,ncol@,nro&@; svH?sv)90;

DH?sv]) UH?sv]u XH?sv]v

(47)

4

Decomposition in Diagram

Catri0 -Lu )ecomposition

Not al&ays uniue *R Decomposition

Gull column rank

Suare Rectangular SXD Symmetric -symmetric ID +holesky Decomposition Spectral Decomposition -C^<C or)an Decomposition -C<C Similar Diagonali6ation 9 ?1_A9=:

(48)

4!

&roperties B! S9D

Re8riting the S9D

&here

r =

rank of

-

i

 the i?th )iagonal element of _.

u

i

an) v

i

are the i?th columns of U an) X

respectively.

( i r i i i ( " u < U A

∑

=

Λ

=

3 λ

(49)

4"

&roprieties o! S9D

Lo8 rank Approximation

/heorem (f

A=U:< (

is the SXD of

A

an) the

singular values are sorte) as

,

then for any

l @r

, the !est rank?

l

appro0imation

to

A

is

Q

Lo& rank appro0imation techniue is very much

important for )ata compression

.

n λ λ λ

≥



8 3 ( i l i i i

"

u

A

∑

=

3

`

λ

∑

+ =

=

−

r l i i A A 3 8 8 ` λ

(50)

4#

' SXD can !e use) to compute optimal

lo8/rank

approximations

.

' -ppro0imation of - is  of rank

#

such that

(f are the characteristics roots of A( A then

 an)

)

are !oth

m

×

n

matrices.

Lo&?rank -ppro0imation

 k ) rank )

/in

A

)

A

=

−

= ; 9 

`

Gro!enius norm

∑∑

= = = m i n $ i$ a A 3 8 3 n d d d ₃, ₈,,

∑

= = n i i d A 3 8

(51)

5%

Lo&?rank -ppro0imation

' Solution via SXD

set smallest r-k

singular values to zero

                    ( < U )                     • • •                 =                 Λ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ 6=2 ( k

<

U

A

`

=

)iag

9

λ ₃

,...,

λ

,

"

,...,

"

;

column notation: sum of rank 1 matrices ( i i k i iu " A

=

∑

₌ 3 ` _λ

(52)

51

-ppro0imation error

' o& goo) 9!a); is this appro0imationF

' (t[s the !est possi!le, measure) !y the Gro!enius norm of the

error

' &here the 

i

are or)ere) such that 

i ≥



ib3

.

∑

+ = =

=

−

=

−

r k i i   k ) rank )

A

)

A

3 8 8 8 ; 9 

`

min

λ

8 `  A A− No&

(53)

52

Ro& appro0imation an) column

appro0imation

Suppose !i an) c $ represent the i?th ro& an) T?th column of A. /he SXD

of A an) is

/he SXD euation for !iis

2e can appro0imate !_i !y  l@r

BCere i  3,,m.

∑

= = r k k k $k $ " u D 3 λ A` ( k l k k k ( l l l < u " U A

∑

=

Λ

=

3 ` λ ( k r k k k ( " u < U A

∑

= = Λ = 3 λ

∑

=

r k k k ik i u " ! 3 λ

∑

= = l k k k ik l i u " ! 3 λ -lso the SXD euation for D$ is,

&here T  3, 8, , n

2e can also appro0imate D$ !y =

∑

Q l@r

l k k $k l $ " u D λ

(54)

53

Least suare solution in inconsistent

system

By using SXD &e can solve the inconsistent system./his gives the

least suare solution.

/he least suare solution

&here

A g

!e the CI inverse of

A.

8

min

b Ax

(55)

54

/he SXD of

A g

is

/his can !e &ritten as

(56)

55 Basic Results of SXD

(57)

5

S9D :ased &CA

(f &e re)uce) varia!le !y using SXD then it performs like I+-. Suppose ) is a mean centere) )ata matri0, /hen

) using SXD, )=U:< (

&e can &rite? )< = U:

Suppose Y = )< = U:

/hen the first columns of Y represents the first

principal component score an) so on.

o SXD Base) I+ is more Numerically Sta!le.

o (f no. of varia!les is greater than no. of o!servations then SXD !ase) I+- &ill give efficient result9-ntti Niemistd, Statistical -nalysis of <ene E0pression Cicroarray Data,8"":;

(58)

5!

 Data Re)uction !oth varia!les an) o!servations.  _{Solving linear least suare Iro!lems}

 (mage Irocessing an) +ompression.  _{K?Selection for K?means clustering}  Cultivariate %utliers Detection

 Noise Giltering

 /ren) )etection in the o!servations an) the varia!les.

(59)

5"

%rigin of !iplot

 <a!riel 93#43;

 _{%ne of the most}

important a)vances in )ata analysis in recent )eca)es  _+urrently  _{^ :",""" &e! pages}  _{Numerous aca)emic} pu!lications  _{(nclu)e) in most} statistical analysis packages

 Still a very ne&

techniue to most scientists

f. Ruben Gabriel, “The founder of biplot”

+ourtesy of Irof. Purifcación Galindo

(60)

5#

2hat is a !iplotF

' ilot+ $ bi+ , lot+

- *lot+

' scatter *lot o t/o ro/s 0 o t/o columns or

' scatter *lot summariin the ro/s OR the columns

- bi+

' 0T ro/s 678 columns

(61)

%

Iractical )efinition of a !iplot

Any tBo?Bay table can be analyFed using a 2?biplot as soon as it can be suGGiciently approximated by a rank?2 matrix.H 7-abriel' 1IJ18

!"by"# table

Matrix decomposition

P(4, 3) G(3, ) !(, 3)

("o# 3$%&iplots are also possi&le')

          − − ×                 − − →                  − − − − − 8 3 = @ @ 8 @ 8 3 " = = @ 3 @ @ @ 8 @ = 3 38 38 > = # A 3" @ 3: 38 A 8 A # 8" 3 @ 8 3 y x e e e g g g g y x g g g g e e e <3 <8 <@ <= E3 E8 E@

(62)

1

Singular Xalue Decomposition 9SXD; 

Singular Xalue Iartitioning 9SXI;

SXD

SXI

iplot Plot Plot

∑

= − =



→



→



r k k$ G k G k ik ><9 r k k$ k ik ><. i$

"

u

"

u

)

3 3 3

;

;9

9

λ λ λ

*e +ran - o ., i/e/, t*e minimum num&er o P0 re1uired to ully represent . Matrix c*aracterisi n2 t*e ro#s Sin2ular values Matrix c*aracterisi n2 t*e columns 5o#s scores 0olumn scores f3 f" f378

+ommon uses value of f

(63)

2

Biplot

 _{The simplest biplot is to show the first two PCs together with the}

projections of the axes of the original variables

 _{x-axis represents the scores for the first principal component}

 _{Y-axis the scores for the second principal component.}

 _{The original variables are represented by arrows which}

graphically indicate the proportion of the original variance explained by the first two principal components.

 _{The direction of the arrows indicates the relative loadings on}

the first and second principal components.

 _{Biplot analysis can help to understand the multivariate data}

i !raphically ii "ffectively iii Conveniently.

(64)

3

Biplot of (ris Data

C o m * . 2 :%.2 :%.1 %.% %.1 %.2 : % . 2 : % . 1 % . % % . 1 % . 2 1 1 1 1 1 1 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 333 3 3 3 3 3 3 3 3 :1% :5 % 5 1% : 1 % : 5 % 5 1 % Se*al ;. Se*al <. =etal ;. =etal <. 3 Setosa 8 Xersicolor @ Xirginica

(65)

4

(mage +ompression E0ample

Iansy Glo&er image, collecte) from

http77&&&.ats.ucla.e)u7stat7r7co)e7pansy.Tpg

/his image is A""J=A: pi0els

(66)

5

Singular values of flo&ers image

(67)



Lo& rank -ppro0imation to flo&ers image

Rank?3 appro0imation

Rank? :

(68)

!

Rank?8" appro0imation

Lo& rank -ppro0imation to flo&ers image

(69)

" Rank?:" appro0imation

Lo& rank -ppro0imation to flo&ers image

(70)

# Rank?3"" appro0imation

Lo& rank -ppro0imation to flo&ers image

(71)

!%

an>15% a**ro?imation /rue (mage

(72)

!1

Butlier Detection Using S9D

Nishith an) Nasser 98""4,CSc. /hesis; propose a graphical

metho) of outliers )etection using SXD.

(t is suita!le for !oth general multivariate )ata an) regression

)ata. Gor this &e construct the scatter plots of first t&o I+[s,

an) first I+ an) thir) I+. 2e also make a !o0 in the scatter

plot &hose range lies

median

93

st9D

;

K

@

× mad

93

st9D

; in the

)

?a0is an)

median

98

nd9D

@

rd9D

;

K

@

× mad

98

nd9D

@

rd9D

; in the

Y?

a0is.

2here

mad

 me)ian a!solute )eviation.

/he points that are outsi)e the !o0 can !e consi)ere) as

e0treme outliers. /he points outsi)e one si)e of the !o0 is

terme) as outliers. -long &ith the !o0 &e may construct

another smaller !o0 !oun)e) !y 8.:78 C-D line

(73)

!2

Butlier Detection Using S9D 0Cont12

Scatter plot of a&kins, Bra)u an) kass )ata 9a; scatter plot of first t&o I+[s an) 9!; scatter plot of first an) thir) I+.

-2K(NS?BR-DU?K-SS 93#>=;

D-/-Data set containing 4: o!servations &ith 3= influential o!servations. -mong them there are ten high leverage outliers 9cases 3?3"; an) for high leverage points 9cases 33?3=; ?(mon 98"":;.

(74)

!3

Butlier Detection Using S9D 0Cont12

Scatter plot of mo)ifie) Bro&n )ata 9a; scatter plot of first t&o I+[s an) 9!; scatter plot of first an) thir) I+.

C%D(G(ED BR%2N D-/-Data set given !y Bro&n 93#>";.

Ryan 93##4; pointe) out that the original )ata on the :@ patients &hich contains 3 outlier

9o!servation num!er 8=;.

(mon an) a)i98"":; mo)ifie) this )ata set !y putting t&o more outliers as cases := an) ::.

-lso they sho&e) that o!servation 8=, := an) :: are outliers !y using generali6e) stan)ar)i6e)

(75)

!4

+luster Detection Using SXD

Singular Xalue Decomposition is also use) for cluster

)etection 9Nishith, Nasser an) Su!oron, 8"33;.

/he metho)s for clustering )ata using first three

I+[s are given !elo&,

median 71st 9D8 K k × mad 71st 9D8

in the ?a0is an)

median 72nd 9DMrd 9D8 K k × mad 72nd 9DMrd 9D8

in the ?a0is.

2here

mad

 me)ian a!solute )eviation. /he value of

(76)

(77)

!

(78)

!!

+limatic Xaria!les

/he climatic varia!les are, 3. Rainfall 9RG; mm

8. Daily mean temperature 9/?CE-N;"+ @. Ca0imum temperature 9/?C-;"+ =. Cinimum temperature 9/?C(N;"+ :. Day?time temperature 9/?D-;"+

A. Night?time temperature 9/?N(</;"+

4. Daily mean &ater vapor pressure 9XI; CB-R >. Daily mean &in) spee) 92S; m7sec

#. ours of !right sunshine as percentage of ma0imum possi!le sunshine hours 9CIS;

(79)

!"

+onseuences of SXD

<enerally many missing values may present in the )ata. (t may also contain unusual o!servations. Both types of pro!lem can not han)le +lassical singular value )ecomposition.

Ro!ust singular value )ecomposition can solve !oth types of pro!lems. Ro!ust singular value )ecomposition can !e o!taine) !y alternating L3

regression approach 9Douglas C. a&kins, Li Liu, an) S. Stanley oung, 98""3;;.

(80)

!#

(nitiali6e the lea)ing

left singular vector u3

/here is no o!vious choice of the initial values of u3

Git the L3 regression coefficientc _j !y

minimi6ing @ $=1'2' …'p ∑ = − n i i $ i$ c u x 3 3

+alculate right singular vector $₃c7c

, &here . refers to Eucli)ean norm.

-gain fit the L3 regression coefficient

di!y minimi6ing ∑ Q i=1'2'….'n = − p $ $ i i$ d " x 3 3

+alculate the resulting estimate of the left eigenvector u_i=d % d 

(terate this process untill it converge.

The 6lternatin ;1 eression 6lorithm or obust Sinular Aalue 8ecom*osition.

(81)

"%

+lustering &eather stations on Cap

Using RSXD

(82)

"1

Re!erences

' Bro&n B.2., r. 93#>";. Ire)iction analysis for !inary )ata. in

+iostatistics Dasebook' R.<. Ciller, r., B. Efron, B. 2. Bro&n, r., L.E. Coses 9E)s.;, Ne& ork 2iley.

' Dhrymes, Ihoe!us . 93#>=;, /atCematics Gor Econometrics' 8n) e). Springer Xerlag, Ne& ork.

' a&kins D. C., Bra)u D. an) Kass <.X.93#>=;,Location of several

outliers in multiple regression )ata using elemental sets. (ecCnometrics, 8", 3#4?8">.

' (mon -. . C. R. 98"":;. ()entifying multiple influential o!servations in linear Regression. 5ournal oG Applied >tatistics @8, 4@  #".

' Kumar, N. , Nasser, C., an) Sarker, S.+., 8"33. - Ne& Singular Xalue Decomposition Base) Ro!ust <raphical +lustering /echniue an) (ts -pplication in +limatic Data 5ournal oG -eograpCy and -eology, Danadian Denter oG >cience and Education , Xol?@, No. 3, 884?8@>. ' Ryan /.I. 93##4;. /odern !egression /etCods, 2iley, Ne& ork.

'

Ste8art5 '11 0)??>21 &at'ix Alo'itm*5 9ol )1 Easic

Decompositions5 Siam5 &hiladelphia1

' Catri0 Decomposition.