DSP-CIS. Part-III : Optimal & Adaptive Filters. Chapter-8 Recursive Least Squares Algorithms. & Chapter-9 : Fast Recursive Least Squares Algorithms

(1)

Part-III : Optimal & Adaptive Filters

Chapter-8 : Recursive Least Squares Algorithms

&

Chapter-9 : Fast Recursive Least Squares Algorithms

Marc Moonen

Dept. E.E./ESAT-STADIUS, KU Leuven [email protected] www.esat.kuleuven.be/stadius/

Part-III : Optimal & Adaptive Filters

Wieners Filters & the LMS Algorithm

•  Introduction / General Set-Up

•  Applications

•  Optimal Filtering: Wiener Filters

•  Adaptive Filtering: LMS Algorithm

– 

Recursive Least Squares Algorithms

•  Least Squares Estimation

•  Recursive Least Squares (RLS)

•  Square Root Algorithms

Fast Recursive Least Squares Algorithms Kalman Filters

•  Introduction – Least Squares Parameter Estimation

•  Standard Kalman Filter

•  Square-Root Kalman Filter

Chapter-7

Chapter-8

Chapter-9

Chapter-10

(2)

DSP-CIS 2017 / Part-III / Chapter-8: RLS Algorithms & Chapter-9: Fast RLS Algorithms 3 / 40 3

1. Least Squares (LS) Estimation

Prototype optimal/adaptive filter revisited

+ filter filter input

error desired signal

filter output filter parameters

filter structure ?

→ FIR filters

(=pragmatic choice) cost function ?

→ quadratic cost function (=pragmatic choice)

Least Squares & RLS Estimation

uk−3

uk−2

uk−1 uk

4

FIR filters(=tapped-delay line filter/‘transversal’ filter)

filter input

0

w0[k] w1[k] w2[k] w3[k]

u[k] u[k-1] u[k-2] u[k-3]

b w

a a+bw

+ error desired signal

filter output

y_k = N!−1

l=0

w_l· uk−l

y_k=w^T · uk where

w^T ="

w₀ w₁ w₂ · · · wN−1# u^T_k ="

u_k u_k₋₁ u_k₋₂ · · · uk−N+1#

u[k]

y[k]

d[k]

e[k]

PS: Shorthand notation uk=u[k], yk=y[k], dk=d[k], ek=e[k], Filter coefficients (‘weights’) are wl (replacing bl of previous chapters) For adaptive filters wlalso have a time index wl[k]

w^T=! w₀ w₀ … w_L

" #

$ u_k^T= u! _k u_k−1 … u_k−L

" #

$

y

_k

= w

_l

⋅u

_k−l

l=0 L

∑ ^{= w}

^T

^{⋅ u}

^k

^{= u}

^T^k

^{⋅ w}

(3)

DSP-CIS 2017 / Part-III / Chapter-8: RLS Algorithms & Chapter-9: Fast RLS Algorithms 5 / 40

Quadratic cost function MMSE :(see Lecture 8)

J_MSE(w) =E{e²_k} = E{|dk− yk|²} = E{|dk− w^Tu_k|²} Least-squares(LS) criterion :

if statistical info is not available, may use an alternative ‘data-based’ criterion...

J_LS(w) =

!L k=1

e²_k=

!L k=1

|dk− yk|²= "_L

k=1|dk− w^Tu_k|² Interpretation? : see below

J

_LS

(w) = e

_l²

l=1 k

∑ ⁼ ⁽ ^d

^l

^{− y}

^l

⁾

²

l=1 k

∑ ⁼ ( ^d

^l

^{− u}

^l^T

^w )

²

l=1 k

∑

J_MSE(w) = Ε e

{ }

_k² ^{= Ε d}

{ ⁽

^k^{− y}^k

⁾

²

}

^{= Ε d}

{ (

^k^{− u}^T^k^w

)

²

}

Least Squares & RLS Estimation

6

filter input sequence : u₁, u₂, u₃, . . . , u_L

corresponding desired response sequence is : d₁, d₂, d₃, . . . , d_L

⎡

⎢⎢

⎢⎣ e₁ e₂ ...

e_L

⎤

⎥⎥

⎥⎦ ' () * error signale

=

⎡

⎢⎢

⎢⎣ d₁ d₂ ...

d_L

⎤

⎥⎥

⎥⎦ ' () *

d

−

⎡

⎢⎢

⎣ u^T₁ u^T₂ ...

u^T_L

⎤

⎥⎥

⎦ ' () *

U

·

⎡

⎢⎢

⎣ w₀ w₁ ...

w_N

⎤

⎥⎥

⎦ ' () *

w

cost function : J_LS(w) =+_L

k=1e²_k =∥e∥²₂=∥d − Uw∥²₂

→ linear least squares problem : minw∥d − Uw∥²₂

e_k d_k

d_k u_k

u_k^T

J_LS(w) = e_l²

l=1 k

∑

^{= e} 2

2 = d − Uw ₂² e₁

e₂

! e_k

!

"

#

##

$

%

&

error signal e"#$

= d₁ d₂

! d_k

!

"

#

##

$

%

&

d

"#$

−

u₁^T u₂^T

! u_k^T

!

"

#

##

$

%

&

U

" #% %$ .

w₀ w₁

! w_L

!

"

#

##

$

%

&

w

"#$

(4)

J_LS(w) =

!L k=1

e²_k=∥e∥²₂=e^T · e = ∥d − Uw∥²₂

minimum obtained by setting gradient = 0 : 0 = [∂J_LS(w)

∂w ]w=w_LS= [ ∂

∂w(d^Td + w^TU^TUw− 2w^TU^Td)]w=w_L

= [2 U" #$ %^TU Xuu

w− 2 U"#$%^Td Xdu

]w=w_LS

Xuu· wLS=Xdu → wLS=X⁻¹uuXdu ‘normal equations’.

J_LS(w) = e_l²

l=1 k

∑

^{= e} ²²^{= e}^T^{.e = d −Uw} ²²

This is the

‘Least Squares Solution’

‘Normal equations’

(L+1 equations in L+1 unknowns)

Least Squares & RLS Estimation

9

Note : correspondences with Wiener filter theory ?

♣ estimate ¯Xuuand ¯Xduby time-averaging (ergodicity!) estimate{ ¯Xuu} =1

L

!L k=1

uk· u^Tk =1

LU^T· U = 1 LXuu

estimate{ ¯Xdu} =1 L

!L k=1

uk· dk= 1

LU^T· d = 1 LXdu

leads to same optimal filter :

estimate{wW F} = (_L¹Xuu)⁻¹· (_L¹Xdu) =X⁻¹uu· Xdu=w_LS estimate ℵ

{

uu

}

⁼¹_k^. ^u^l^.

l=1 k

∑

^u^l^T⁼¹_k^{. U}^T^{U =}¹_k^.ℵ^uu

estimate ℵ

{

du

}

⁼¹_k^. ^u^l^.

l=1 k

∑

^d^l⁼¹_k^{. U}^T^{d =}¹_k^.ℵ^du

(5)

Note : correspondences with Wiener filter theory ? (continued)

♣ Furthermore (for ergodic processes!) : X¯uu= lim

L→∞

1 L

!L k=1

uk· u^Tk = lim

L→∞

1 LXuu

X¯du= lim

L→∞

1 L

!L k=1

uk· dk= lim

L→∞

1 LXdu

so that

lim_L_→∞w_LS=w_{W F}. ℵuu= lim_k→∞ 1

k. u_l.

l=1 k

∑

^u^T^l ^{= lim}^k→∞¹_k^.ℵ^uu

ℵ_du = lim_k→∞1 k. u_l.

l=1 k

∑

^d^l ^{= lim}^k→∞¹_k^.ℵ^du

lim_k→∞w_LS = w_WF

Least Squares & RLS Estimation

11

2. Recursive Least Squares (RLS)

For a fixed data segment 1...L, least squares problem is minw∥d − Uw∥²₂

d =

⎡

⎢⎢

⎢⎣ d₁ d₂ ...

d_L

⎤

⎥⎥

⎥⎦ U =

⎡

⎢⎢

⎣ u^T₁ u^T₂ ...

u^T_L

⎤

⎥⎥

⎦

wLS = [U^TU]⁻¹· U^Td .

Wanted : recursive/adaptive algorithms L→ L + 1, sliding windows, etc...

minw_k d₁ d₂

! d_k

!

"

##

#

$

%

&

d_k

"#$

−

u₁^T u₂^T

! u_k^T

!

"

#

$

%

&

U_k

" #% %$ .

w₀ w₁

! w_L

!

"

##

#

$

%

&

w_k

"#$

2 2

Can LS solution @ time k be computed from solution @ time k-1 ? k

Matrices and vectors now with time index added

w[k] =

^ℵuu[ k ]−1

.ℵdu[ k ]

= U # $

_k^T

U

_k

% &

−1

.U

_k^T

d

_k

(6)

2.1 Standard RLS

It is observed thatXuu(L + 1) =Xuu(L) +uL+1u^T_L+1 Thematrix inversion lemma states that

[Xuu(L + 1)]⁻¹= [Xuu(L)]⁻¹−^[Xûu_1+u^(L)]T⁻¹û^L+1û^T^L+1^[Xûu^(L)]⁻¹ L+1[Xuu(L)]−1u_L+1

Result :

wLS(L + 1) =wLS(L) + [!Xuu(L + 1)]"# ⁻¹uL+1$

Kalman gain

· (dL+1− u^TL+1wLS(L))

! "# $

a priori residual

=standard recursive least squares (RLS) algorithm Remark :O(N²)operations per time update

Remark : square-root algorithms with better numerical properties see below

ℵuu[k]−1 =ℵuu[k −1]−1 − ( 1 1+ u_k^Tℵuu[k −1]−1u_k

).k_kk_k^T with k_k=ℵuu[k −1]−1uk

w_LS[k ] = wLS[k −1] + ℵuu[k ]−1u_k 'Kalman gain vector'!###"###$

=( 1

1+ukTℵuu[ k−1]⁻¹uk ). kk

%###&###'. (d_k− uk

Tw_LS[k −1]) 'a priori residual'

!###"###$

ℵuu[k] =ℵ

uu[k −1]+ u_k.u_k^T (and ℵ du[k] =ℵ

du[k −1]+ u_k.d_k)

(check ‘matrix inversion lemma’ in Wikipedia)

With this it is proved that:

Rank-1 updates

O(L²) instead of O(L³)

Least Squares & RLS Estimation

16

2.3 Exponentially Weighted RLS Exponentially weighted RLS

JLS(w) =!L

k=1λ^2(L^−k)e²_k

0 < λ < 1 isweighting factor or forget factor

1

1−λis a ‘measure of the memory of the algorithm’

wLS(L) = [U (L)" ^T#$U (L)]⁻¹%

[Xuu(L)]⁻¹

[U (L)^Td(L)]

" #$ %

Xdu(L)

with

d(L) =

⎡

⎢⎢

⎣ λ^L−1d1

λ^L⁻²d2

...

λ⁰dL

⎤

⎥⎥

⎦ U (L) =

⎡

⎢⎢

⎢⎣

λ^L⁻¹u^T₁ λ^L⁻²u^T₂

...

λ⁰u^T_L

⎤

⎥⎥

⎥⎦ J_LS(w) = λ^{2(k−l )}e_l²

l=1 k

∑

: Goal is to give a smaller weight to ‘older’ data, i.e.

minw_k λ^k−1d₁ λ^k−2d₂

! λ⁰d_k

"

#

$$

$

%

&

'' '' '

d_k

" #$ $%

−

λ^k−1u₁^T λ^k−2u₂^T

! λ⁰u^T_k

"

#

$$

$

%

&

'' '' '

U_k

"$$#$$% .

w₀ w₁

! wL

"

#

$$

$

%

&

'' '' ' wk

"#%

2 2

w

_k

=

^ℵuu[ k ]−1

.ℵdu[ k ]

= U # $

_k^T

U

_k

% &

−1

.U

_k^T

d

_k

Which leads to…

(7)

2.3 Exponentially Weighted RLS It is observed thatXuu(L + 1) = λ²Xuu(L) +uL+1u^T_L+1

hence

[Xuu(L + 1)]⁻¹=_λ¹2[Xuu(L)]⁻¹−

1

λ2[Xuu(L)]⁻¹u_L+1u^T_L+1¹ λ2[Xuu(L)]⁻¹ 1+¹

λ2u^T_L+1[Xuu(L)]⁻¹u_L+1

wLS(L + 1) =wLS(L) + [Xuu(L + 1)]⁻¹uL+1· (dL+1− u^TL+1wLS(L)) . i.e. exponential weighting hardly changes RLS formulas.. (easy!) ℵuu[k]−1 = 1

λ²ℵuu[k −1]−1 − ( 1 1+1

λ²u_k^Tℵuu[k −1]−1u_k

).k_kk_k^T with k_k= 1

λ²ℵuu[k −1]−1u_k

w_LS[k] = w_LS[k −1] +ℵuu[k]−1u_k.(d_k− u^T_kw_LS[k −1])

ℵuu[k] = λ².ℵuu[k −1]+ uk.uk

T (and ℵdu[k] = λ².ℵdu[k −1]+ uk.dk)

Least Squares & RLS Estimation

18

3. Square-root Algorithms

• Standard RLS exhibits unstable roundoﬀ error accumulation, hence not the algorithm of choice in practice

• Alternative algorithms (‘square-root algorithms’), which have been proved to be stable numerically, are based on orthogonal matrix decompositions, namely QR decomposition (+ QR updating, inverse QR updating, see below)

(8)

3.1 QRD-based RLS Algorithms

QR Decomposition for LS estimation least squares problem

minw∥d − Uw∥²₂

‘square-root algorithms’ based on QR decomposition (QRD) :

!"#$U L×N

= Q!"#$

L×N

· R!"#$

N×N

Q^T· Q = I, Q is orthogonal R is upper triangular

U

kx ( L+1)! = Q

kxk

! . R 0

!

"

# $

%&

kx ( L+1)

!

= Q!

Q(:,1:L+1)! . R

( L+1) x ( L+1)! square * rectangular rectangular * square

Least Squares & RLS Estimation

21

Everything you need to know about QR decomposition

Example :

! "#U $

⎡

⎢⎢

⎣ 1 6 10 2 7 −11 3 8 12 4 9 −13

⎤

⎥⎥

⎦=

! "#Q $

⎡

⎢⎢

⎣

0.182 0.816 0.174 0.365 0.408 −0.619 0.547 0 0.716 0.730 −0.408 −0.270

⎤

⎥⎥

⎦·

! "#R $

⎡

⎣5.477 14.605−5.112 0 4.082 8.981 0 0 20.668

⎤

⎦

Remark : QRD≈ Gram-Schmidt Remark : U^T · U = R^T· R

R is Cholesky factor or square-root of U^T · U

→ ‘square-root’ algorithms ! Q

(9)

QRD for LS estimation if

U = Q· R then

Q^T·! U d"

=! R z"

with this

wLS = R⁻¹· z min_w d −Uw₂² =

(**)

min_w Q^T(d −Uw)

2

2= minw

z

*

"

#$ %

&

' − R

0

"

#$ %

&

'w

2 2

(**) orthogonal transformation preserves norm

R ⋅ w

_LS

= z ⇒ w

_LS

= R

⁻¹

⋅ z = [ ! Q

^T

U ]

⁻¹

⋅ ! Q

^T

d

U

kx ( L+1)! = Q

kxk! . R 0

!

"

# $

%&

kx ( L+1)

!

= Q!

Q(:,1:L+1)! . R

( L+1) x ( L+1)!

This is a numerically better way of computing the LS solution, better than wLS= U!" ^TU#$⁻¹⋅U^Td

triangular backsubstitution

Least Squares & RLS Estimation

23

QR-updating for RLS estimation Assume we have computed the QRD at time k

Q(k)˜ ^T ·!

U(k) d(k)"

=!

R(k) z(k)"

The corresponding LS solution is wLS(k) = [R(k)]⁻¹· z(k) Our aim is to update the QRD into

Q(k + 1)˜ ^T·!

U (k + 1) d(k + 1)"

=!

R(k + 1) z(k + 1)"

and then compute wLS(k + 1) = [R(k + 1)]⁻¹· z(k + 1)

k-1 R[k −1] z[k −1]

"

# $

% =Q[k −1]! ^T⋅" U_k−1 d_k−1

# $

%

w_LS[k] = R[k]⁻¹⋅ z[k]

R[k] z[k]

!" #

$ =Q[k]! ^T ⋅! U_k d_k

" #

$

w_LS[k −1] = R[k −1]⁻¹⋅ z[k −1]

(10)

QR-updating for RLS estimation

It is proved that the relevant QRD-updating problem is

!R(k + 1) z(k + 1)

0 ⋆

"

← Q(k + 1)^T·

!R(k) z(k) u^T_k+1 d_k+1

"

PS: This is based on a QR-factorization as follows:

R[k −1]

uk T

"

#

$

%

&

' '

(L+2)×(L+1)

! "# #$

= Q[k]

(L+2)×(L+2)! . R[k]

0

"

#

$$

%

&

''

(L+2)×(L+1)

! "# $#

R[k] z[k]

0!0 ∗

"

#

$$

%

&

''= Q[k]^T ⋅ R[k −1] z[k −1]

u^T_k d_k

"

#

$$

%

&

''

Least Squares & RLS Estimation

25

QR-updating for RLS estimation

!R(k + 1) z(k + 1)

0 ⋆

"

← Q(k + 1)^T·

!R(k) z(k) u^T_k+1 d_k+1

"

R(k + 1)· wLS(k + 1) = z(k + 1).

=square-root (information matrix) RLS

Remark . with exponential weighting

!R(k + 1) z(k + 1)

0 ⋆

"

← Q(k + 1)^T·

!λ R(k) λ z(k) u^T_k+1 d_k+1

"

R[k] z[k]

0!0 ∗

"

#

$$

%

&

''= Q[k]^T⋅ R[k −1] z[k −1]

u_k^T d_k

"

#

$

%

&

' ' w_LS[k] = R[k]⁻¹⋅ z[k] =‘triangular backsubstitution’

R[k] z[k]

0!0 ∗

"

#

$$

%

&

''= Q[k]^T⋅ λ⋅ R[k −1] λ⋅ z[k −1]

uk

T dk

"

#

$$

%

&

''

(11)

QRD updating

!R(k + 1) z(k + 1)

0 ⋆

"

← Q(k + 1)^T·

!R(k) z(k) u^T_k+1 d_k+1

"

basic tool isGivens rotation

G_i,j,θdef=

i j

↓ ↓

⎡

⎢⎢

⎣

I_i₋₁ 0 0 0 0

0 cos θ 0 sin θ 0

0 0 I_j_−i−1 0 0

0 − sin θ 0 cos θ 0

0 0 0 0 I_m_−j

⎤

⎥⎥

⎦

← i

← j R[k] z[k]

0!0 ∗

"

#

$$

%

&

''= Q[k]^T⋅ R[k −1] z[k −1]

u_k^T d_k

"

#

$

%

&

' '

Least Squares & RLS Estimation

27

QRD updating

Givens rotation applied to a vector ˜x = G_i,j,θ· x :

˜

x_i = cos θ· xi+ sin θ· xj

˜

x_j =− sin θ · xi+ cos θ· xj

˜

x_l= x_l for l̸= i, j

˜

x_j = 0 iﬀ tan θ =^x_x^j_i !

(12)

QRD updating

!R(k + 1) z(k + 1)

0 ⋆

"

← Q(k + 1)^T·

!R(k) z(k) u^T_k+1 d_k+1

"

sequence ofGivens transformations

⎡

⎢⎢

⎣

x x x x x x x x x x x x x

⎤

⎥⎥

⎦ →

⎡

⎢⎢

⎣

x x x x x x x x x x x x

⎤

⎥⎥

⎦ →

⎡

⎢⎢

⎢⎣

x x x x x x x x x x x

⎤

⎥⎥

⎥⎦→

⎡

⎢⎢

⎢⎣

x x x x x x x x x

⋆

⎤

⎥⎥

⎥⎦ Q(k +1) is constructed as a product/sequence of^[k]

3-by-3 example

R[k] z[k]

0!0 ∗

"

#

$$

%

&

''= Q[k]^T⋅ R[k −1] z[k −1]

u_k^T d_k

"

#

$

%

&

' '

Least Squares & RLS Estimation

29

!R(k + 1) z(k + 1)

0 ⋆

"

→ Q(k + 1)^T·

!R(k) z(k) u^T_k+1 dk+1

"

0

0 R24

R11 R12 R13 R14

R33 R34

R44 R22

0

z11

z21

z31

z41 R23

d u(4)

u(3)

u(1) u(2)

(delay) memory cell

a’

b’

b a rotation cell

4-by-4 example

R[k] z[k]

0!0 ∗

"

#

$$

%

&

''= Q[k]^T⋅ R[k −1] z[k −1]

uk

T dk

"

#

$$

%

&

''

u[k] u[k-1] u[k-2] u[k-3] d[k]

(13)

Residual extraction

!R(k + 1) z(k + 1)

0 ϵ

"

= Q(k + 1)^T·

!R(k) z(k) u^T_k+1 d_k+1

"

From this it is proved that the ‘a posteriori residual’ is d_k+1− u^T_k+1wLS(k + 1) = ϵ ·#_N

i=1cos(θ_i) and the ‘a priori residual’ is

d_k+1− u^T_k+1wLS(k) =^#^N ^ϵ

i=1cos(θ_i) R[k] z[k]

0!0 ε

!

"

##

$

%

&

&= Q[k]^T⋅ R[k −1] z[k −1]

u_k^T d_k

!

"

#

$

%

&

d_k− u_k^Tw_LS[k] =

ε

⋅ cos(

θ

i)

i=1

∏

L+1

d_k− uk

Tw_LS[k −1] = ε

cos(θi)

i=1

∏

L+1

Least Squares & RLS Estimation

31

Residual extraction (v1.0) :

(delay) memory cell

a’

b’

b a

cos

rotation cell

cos cos cos cos

0

0 R24

R11 R12 R13 R14

R33 R34

R44 R22

0

0 1

2

3

4 1

z11

z21

z31

z41 R23

d u(4)

u(3)

u(1) u(2)

output

d_k+1− u^T_k+1wLS(k + 1) = ϵ ·!N i=1cos(θ_i)

u[k] u[k-1] u[k-2] u[k-3] d[k]

d_k − uk

Tw_LS[k] =ε⋅ cos(θi)

i=1

∏

L+1

ε

(14)

Least Squares & RLS Estimation

Residual extraction (v1.1) :

cos 0

0 R24

R11 R12 R13 R14

R33 R34

R44 R22

0

z11

z21

z31

z41 R23

(delay) memory cell

a’

b’

b a rotation cell d

u(4) u(3)

u(1) u(2)

0

0 1

output1

d_k+1− u^T_k+1wLS(k + 1) = ϵ ·!N i=1cos(θ_i)

u[k] u[k-1] u[k-2] u[k-3] d[k]

dk − uk

TwLS[k] =ε⋅ cos(θi)

i=1

∏

L+1

ε

Least Squares & RLS Estimation

³³

Example

near-end signal + residual echo far-end signal

d

0

0 0

0

0 u[k-1] u[k-2] u[k-3]

u[k]

1 0

(15)

Wieners Filters & the LMS Algorithm

•  Introduction / General Set-Up

•  Applications

•  Optimal Filtering: Wiener Filters

•  Adaptive Filtering: LMS Algorithm

– 

Recursive Least Squares Algorithms

•  Least Squares Estimation

•  Recursive Least Squares (RLS)

•  Square Root Algorithms

Fast Recursive Least Squares Algorithms Kalman Filters

•  Introduction – Least Squares Parameter Estimation

•  Standard Kalman Filter

•  Square-Root Kalman Filter

Chapter-7

Chapter-8

Chapter-9 Chapter-10

Fast Recursive Least Squares Algorithms

Ο(L)

Ο(L²)

(16)

31)

Fast Recursive Least Squares Algorithms

Se e p .3 8 fo r a si g n al fl o w g ra p h o f th is

(17)

33)

34)

Least Squares & RLS Estimation

w = arg minw|| d − U.w ||2

2= (U^T.U)⁻¹U^T.d

⇒ !w = argminw_!|| d − (UΠ). !w ||₂²= (Π^TU^T.UΠ)⁻¹Π^TU^T.d = Π⁻¹w e[k] = d[k] − u[k]^T.w

⇒ !e[k] = d[k] − u[k]^TΠ. !w = d[k] − u[k]^TΠ.Π⁻¹w = e[k]

(18)

Fast Recursive Least Squares Algorithms

39/40:

38:

37:

(19)

C

B A

E D

Least Squares & RLS Estimation

(20)

Ο(L)

“n-th layer” (n=2)

DSP-CIS. Part-III : Optimal & Adaptive Filters. Chapter-8 Recursive Least Squares Algorithms. & Chapter-9 : Fast Recursive Least Squares Algorithms

Part-III : Optimal & Adaptive Filters

Chapter-8 : Recursive Least Squares Algorithms

&

Chapter-9 : Fast Recursive Least Squares Algorithms

Marc Moonen

Dept. E.E./ESAT-STADIUS, KU Leuven [email protected] www.esat.kuleuven.be/stadius/

Part-III : Optimal & Adaptive Filters

Wieners Filters & the LMS Algorithm

Recursive Least Squares Algorithms

Fast Recursive Least Squares Algorithms Kalman Filters

Chapter-7

Chapter-8

Chapter-9

Chapter-10

Least Squares & RLS Estimation

y

= w

⋅u

∑ = w

⋅ u

= u

⋅ w

J

(w) = e

∑ = ( d

− y

)

∑ = ( d

− u

w )

∑

{ }

{ (

)

}

{ (

)

}

Least Squares & RLS Estimation

∑

∑

Least Squares & RLS Estimation

{

}

∑

{

}

∑

∑

∑

Least Squares & RLS Estimation

w[k] =

= U # $

U

% &

.U

d

Least Squares & RLS Estimation

∑

w

=

= U # $

U

% &

.U

d

Least Squares & RLS Estimation

Least Squares & RLS Estimation

R ⋅ w

= z ⇒ w

= R

⋅ z = [ ! Q

U ]

⋅ ! Q

d

Least Squares & RLS Estimation

Least Squares & RLS Estimation

Least Squares & RLS Estimation

Least Squares & RLS Estimation

∑ ^{= w}

^{⋅ u}

^{= u}

^{⋅ w}

∑ ⁼ ⁽ ^d

^{− y}

⁾

∑ ⁼ ( ^d

^{− u}

^w )

{ ⁽

⁾