CS 295 (UVM) The LMS Algorithm Fall / 23

The LMS Algorithm

The LMS Objective Function.

Global solution.

Pseudoinverse of a matrix.

Optimization (learning) by gradient descent.

LMS or Widrow-Hoff Algorithms.

Convergence of the Batch LMS Rule.

Compiled by LATEX at 3:50 PM on February 27, 2013.

LMS Objective Function

Again we consider the problem of programming a linear threshold function

x1 w2 w3 wn w1 x2 x3

x_n

w0 y

y D sgn. ^w

^x C ^w

/ D (

1 ; if w

x C w

> 0 ; 1 ; if w

x C ^w

 ⁰ : so that it agrees with a given dichotomy of m feature vectors,

X

D f. ^x

; `

/; : : : ; . x

; `

/ g:

where x

2 R

^and `

2 f ¹ ; C ¹ g ^{, for i} D ¹ ; 2 ; : : : ; m.

CS 295 (UVM) The LMS Algorithm Fall 2013 2 / 23

LMS Objective Function (cont.)

Thus, given a dichotomy

X

D f. ^x

; `

/; : : : ; . x

; `

/ g;

we seek a solution weight vector w and bias w

, such that, sgn w

x

C w

D `

; for i D ¹ ; 2 ; : : : ; m.

Equivalently, using homogeneous coordinates (or augmented feature vectors), sgn b ^w

T

b ^x

D `

; for i D ¹ ; 2 ; : : : ; m, where b ^x

D ¹ ; x

. Using normalized coordinates,

sgn b ^w

T

b ^x

0

D ¹ ; or, b ^w

T

b ^x

0

> 0 ; for i D ¹ ; 2 ; : : : ; m, where b ^x

0

D `

b ^x

^.

LMS Objective Function (cont.)

Alternatively, let b 2 R

satisfy b

> 0 for i D 1 ; 2 ; : : : ; m. We call b a margin vector.

Frequently we will assume that b

D ^1.

Our learning criterion is certainly satisfied if

b w

b ^x

0

D ^b

for i D ¹ ; 2 ; : : : ; m. (Note that this condition is sufficient, but not necessary.) Equivalently, the above is satisfied if the expression

E. b ^w / D

m

X

i

D

 b

b ^w

T

b ^x

0



:

equals zero. The above expression we call the LMS objective function, where LMS stands for least mean square. (N.B. we could normalize the above by dividing the right side by m.)

CS 295 (UVM) The LMS Algorithm Fall 2013 4 / 23

LMS Objective Function: Remarks

Converting the original problem of classification (satisfying a system of inequalities) into one of optimization is somewhat ad hoc. And there is no guarantee that we can find a b ^w

? 2 R

^C

that satisfies E. b ^w

? / D 0. However, this conversion can lead to practical compromises if the original inequalities possess inconsistencies.

Also, there is no unique objective function. The LMS expression,

E. b ^w / D

m

X

i

D

b

b ^w

T

b ^x

0

:

can be replaced by numerous candidates, e.g.

m

X

i

D

ˇ ˇ b

b ^w

T

b ^x

0

ˇ ˇ;

m

X

i

D



1 sgn b ^w

T

b ^x

0



;

m

X

i

D



1 sgn b ^w

T

b ^x

0



; etc.

However, minimizing E. b ^w

? / is generally an easy task.

Minimizing the LMS Objective Function

Inspection suggests that the LMS Objective Function E. b ^w / D

m

X

i

D

b

b ^w

T

b ^x

0

describes a parabolic function. It may have a unique global minimum, or an infinite number of global minima which occupy a connected linear set. (The latter can occur if m < n C 1.) Letting,

X D 0 B B B B

@ b ^x

0

b ^x

0

::

: b ^x

0

1 C C C C A

D 0 B B B

@

`

x O

⁰ _;

x O

⁰ _;

x O

⁰ _;

`

^x O

⁰ ;

^x O

⁰ ;

^x O

⁰ ;

:: : :: : :: : : :: :: :

`

x O

⁰ _;

x O

⁰ _;

x O

⁰ _;

1 C C C A

2 R

^.

^C

^/ ;

then,

E. b ^w / D

m

X

i

D

b

b ^x

0

b ^w

D 0 B B B B

@ b

b ^x

0

b ^w b

b ^x

0

b ^w :: : b

b ^x

0

b ^w

1 C C C C A

2

D b

0 B B B B

@ b ^x

0

b ^x

0

::

: b ^x

0

1 C C C C A

b w

2

D k ^{b X} b ^w j

:

CS 295 (UVM) The LMS Algorithm Fall 2013 6 / 23

Minimizing the LMS Objective Function (cont.)

E. b ^w / D k ^{b X} b ^w k

D . ^{b X} b ^w /

. b X b ^w / D 

b

b ^w

T

X



. b X b ^w / D b ^w

T

X

X b ^w b ^w

T

X

b b

X b ^w C ^b

^b D b ^w

T

X

X b ^{w 2b}

T

X b ^w C k ^b k

As an aside, note that

X

X D .b ^{x 0}

; : : : ; b ^{x 0}

/ 0 B

@ b ^x

0

::

: b ^x

0

1 C A D

m

X

i

D

b ^{x 0}

b ^x

0

i

2 R ^.

^C

^/.

^C

^/ ;

b

X D . ^b

; : : : ; b

/ 0 B

@ b ^x

0

::

: b ^x

0

1 C A D

m

X

i

D

b

b ^x

0

i

2 R

^C

:

Minimizing the LMS Objective Function (cont.)

Okay, so how do we minimize E. b ^w / D b ^w

T

X

X b ^{w 2b}

T

X b ^w C k ^b k

‹

Using calculus (e.g., Math 121), we can compute the gradient of E. b ^w / , and algebraically determine a value of b ^w

? which makes each component vanish. That is, solve

rE.b ^w / D 0 B B B B B B B B B B

@

@E

@ b ^w

@E

@ b ^w

:: :

@E

@ b ^w

1 C C C C C C C C C C A

D ⁰ :

It is straightforward to show that

rE.b ^w / D 2X

X b ^{w 2X}

T

b :

CS 295 (UVM) The LMS Algorithm Fall 2013 8 / 23

Minimizing the LMS Objective Function (cont.)

Thus,

rE.b ^w / D ^2X

^X b ^{w 2X}

T

b D ⁰ if

b w ^? D . ^X

^X /

X

b D ^{X Ž b} ; where the matrix,

X ^Ž D .

X

X /

X

2 R ^.

^C

^/

is called the pseudoinverse of X . If X

X is singular, one defines

X ^Ž D

^lim

!

. X

X C  ^I /

X

: Observe that if X

X is nonsingular,

X ^Ž X D . ^X

^X /

X

X D ^I :

Example

The following example appears in R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, Second Edition, Wiley, NY, 2001, p. 241.

Given the dichotomy,

X

D ˚ . 1 ; 2 /

; 1
; . 2 ; 0 /

; 1
; . 3 ; 1 /

; 1
; . 2 ; 3 /

; 1
we obtain,

X D 0 B B B B B

@ b ^x

0

b ^x

0

b ^x

0

b ^x

0

1 C C C C C A

D 0 B B

@

1 1 2

1 2 0

1 3 1

1 2 3

1 C C A : Whence,

X

X D 0

@

4 8 6

8 18 11

6 11 14

1

A ; and, X ^Ž D . ^X

^X /

X

D 0 B B

@

5 4

13 12

3 4

7 12 1

2 1 6

1 2

1 6

0

0

1 C C A :

CS 295 (UVM) The LMS Algorithm Fall 2013 10 / 23

Example (cont.)

Letting, b D . ¹ ; 1 ; 1 ; 1 /

, then

b w D ^{X Ž b} D 0 B B

@

5 4

13 12

3 4

7 12 1

2 1 6

1 2

1 6

0

0

1 C C A

0 B B B

@ 1 1 1 1 1 C C C A

D 0 B B

@

11 3 4 3 2 3

1 C C A :

Whence,

w

D ¹¹

3 and w D

 ₄ 3 ; ²

3



x₁ x₂

1 1

2 2

3 3

4 4

5

Method of Steepest Descent

An alternative approach is the method of steepest descent.

We begin by representing Taylor’s theorem for functions of more than one variable:

let x 2 R

, and f W R

! R , so

f . x / D ^f . x

; x

; : : : ; x

/ 2 R:

Now let ı x 2 R

, and consider

f . x C ı ^x / D ^f . x

C ı ^x

; : : : ; x

C ı ^x

/:

Define F W R ! R , such that,

F . s / D ^f . x C ^s ı x /:

Thus,

F . 0 / D ^f . x /; and, F . 1 / D ^f . x C ı ^x /:

CS 295 (UVM) The LMS Algorithm Fall 2013 12 / 23

Method of Steepest Descent (cont.)

Taylor’s theorem for a single variable (Math 21/22),

F . s / D ^F . 0 / C ¹ 1 Š ^F

0 . 0 / s C ¹ 2 Š ^F

00 . 0 / s

C ¹ 3 Š ^F

000 . 0 / s

C


:

Our plan is to set s D 1 and replace F . 1 / by f . x C ı ^x / , F . 0 / by f . x / , etc.

To evaluate F ⁰ . 0 / we will invoke the multivariate chain rule, e.g., d

ds f u . s /; v . s /
D @ f

@ u . u ; v / u ⁰ . s / C @ f

@ v . u ; v / v ⁰ . s /:

Thus,

F ⁰ . s / D ^dF

ds . s / D ^df

ds . x

C ^s ı x

; : : : ; x

C ^s ı x

/ D @ f

@ x

. x C ^s ı x / ^d

ds . x

C ^s ı x

/ C


C @ f

@ x

. x C ^s ı x / ^d

ds . x

C ^s ı x

/ D @ f

@ x

. x C ^s ı x /
ı ^x

C


C @ f

@ x

. x C ^s ı x /
ı ^x

:

Method of Steepest Descent (cont.)

Thus,

F ⁰ . 0 / D @ f

@ x

. x /
ı ^x

C


C @ f

@ x

. x /
ı ^x

D r ^f . x /

ı x :

CS 295 (UVM) The LMS Algorithm Fall 2013 14 / 23

Method of Steepest Descent (cont.)

Thus, it is possible to show

f . x C ı ^x / D f . x / C r f . x /

ı x C O kı ^x k

D ^f . x / C kr ^f . x / kkı ^x k ^cos  C O kı ^x k

; where  defines the angle between r ^f . x / and ı x. If kı ^x k  ^{1, then}

ı f D f . x C ı x / f . x /  kr f . x /kkı x k cos :

Thus, the greatest reduction ı f occurs if cos  D 1, that is if ı x D r ^{f , where}

 > 0. We thus seek a local minimum of the LMS objective function by taking a sequence of steps

b w . t C ¹ / D b ^w . t /  rE  b w . t / 

:

Training an LTU using Steepest Descent

We now return to our original problem. Given a dichotomy X

D f. ^x

; `

/; : : : ; . x

; `

/g

of m feature vectors x

2 R

^with `

2 f ¹ ; 1 g ^{for i} D ¹ ; : : : ; m, we construct the set of normalized, augmented feature vectors

X b ⁰

D f.`

; `

x

;

; : : : ; `

x

;

/

2 R

^C

j ⁱ D ¹ ; : : : ; m g:

Given a margin vector, b 2 R

^{, with b}

> 0 for i D ¹ ; : : : ; m, we construct the LMS objective function,

E. b ^w / D ¹ 2

m

X

i

D

. b ^w

T

b ^x

0

b

/

D ¹

2 k X b ^{w b} k

;

(the factor of

is added with some foresight), and then evaluate its gradient

rE.b ^w / D

m

X

i

D

. b ^w

T

b ^x

0

b

/ b ^x

0

D ^X

. X b ^{w b} /:

CS 295 (UVM) The LMS Algorithm Fall 2013 16 / 23

Training an LTU using Steepest Descent (cont.)

Substitution into the steepest descent update rule,

b w . t C ¹ / D b ^w . t / rE  b w . t / 

; yields the batch LMS update rule,

b w . t C ¹ / D b ^w . t / C 

m

X

i

D



b

b ^w . t /

b ^x

0

 b ^x

0

:

Alternatively, one can abstract the sequential LMS, or Widrow-Hoff rule, from the above:

b w . t C ¹ / D b ^w . t / C  

b b ^w . t /

b ^x

0 . t /  b ^x

0 . t /:

where b ^{x 0} . t / 2 b X ⁰

is the element of the dichotomy that is presented to the LTU at

epoch t. (Here, we assume that b is fixed; otherwise, replace it by b . t / .)

Sequential LMS Rule

The sequential LMS rule

b w . t C ¹ / D b ^w . t / C  h

b b ^w . t /

b ^x

0 . t / i b ^x

0 . t /;

resembles the sequential perceptron rule,

b w . t C ¹ / D b ^w . t / C  2 h

1 sgn b ^w . t /

b ^x

0 . t /
i b ^x

0 . t /:

Sequential rules are well suited to real-time implementations, as only the current values for the weights, i.e. the configuration of the LTU itself, need to be stored. They also work with dichotomies of infinite sets.

CS 295 (UVM) The LMS Algorithm Fall 2013 18 / 23

Convergence of the batch LMS rule

Recall, the LMS objective function in the form

E. b ^w / D ¹ 2 ^X b ^{w b}

2

;

has as its gradient,

rE.b ^w / D ^X

^X b ^{w X}

T

b D ^X

. X b ^{w b} /:

Substitution into the rule of steepest descent,

b w . t C ¹ / D b ^w . t /  rE b ^w . t /
;

yields,

b ^w . t C ¹ / D b ^w . t /  X

X b ^w . t / b

Convergence of the batch LMS rule (cont.)

The algorithm is said to converge to a fixed point b ^w

? , if for every finite initial value

b ^w . 0 / < 1 ^,

t

!1 lim b ^w . t / D b ^w

? :

The fixed points b ^w

? satisfy rE.b ^{w ?} / D ^{0, whence}

X

X b ^w

? D ^X

^b :

The update rule becomes,

b w . t C ¹ / D b ^w . t /  X

X b ^w . t / b
D b ^w . t /  X

X b ^w . t / b ^w

?

Let ı b ^w . t / D

b ^w . t / b ^w

? . Then,

ı b ^w . t C ¹ / D ıb ^w . t /  X

X ı b ^w . t / D ^I  X

X /ı b ^w . t /:

CS 295 (UVM) The LMS Algorithm Fall 2013 20 / 23

Convergence of the batch LMS rule (cont.)

Convergence b ^w . t / ! b ^w

? occurs if ı b ^w . t / D b ^w . t / b ^w

? ! 0. Thus we require that ı b ^w . t C ¹ /

<

ı b ^w . t /

. Inspecting the update rule, ı b ^w . t C ¹ / D 

I  X

X  ı b ^w . t /;

this reduces to the condition that all the eigenvalues of I  X

X have magnitudes less than 1.

We will now evaluate the eigenvalues of the above matrix.

Convergence of the batch LMS rule (cont.)

Let S 2 R ^.

^C

^/.

^C

^/ denote the similarity transform that reduces the symmetric matrix X

X to a diagonal matrix ƒ 2 R ^.

^C

^/.

^C

^{/ . Thus, S}

^S D ^SS

D ^{I, and}

S X

X S

D ƒ D diag.

; 

; : : : ; 

/;

The numbers, 

; 

; : : : ; 

, represent the eigenvalues of X

X . Note that 0  

for i D ⁰ ; 1 ; : : : ; n. Thus,

S ı b ^w . t C ¹ / D ^S . I  X

X / S

S ı b ^w . t /;

D . ^I ƒ/ S ı b ^w . t /:

Thus convergence occurs if

k ^S ı b ^w . t C ¹ /k < k ^S ı b ^w . t /k;

which occurs if the eigenvalues of I ƒ all have magnitudes less than one.

CS 295 (UVM) The LMS Algorithm Fall 2013 22 / 23

Convergence of the batch LMS rule (cont.)

The eigenvalues of I ƒ equal 1 

, for i D 0 ; 1 ; : : : ; n. (These are in fact the eigenvalues of I  X

X .) Thus, we require that

1 < 1 

< 1 ; or 0 < 

< 2 ; for all i. Let 

D ^max

0







denote the largest eigenvalue of X

X , then convergence requires that

0 <  < ²



: