The p-norm generalization of the LMS algorithm for adaptive filtering

The p-norm generalization of the LMS algorithm for adaptive filtering

Jyrki Kivinen

University of Helsinki Manfred Warmuth

University of California, Santa Cruz Babak Hassibi

California Institute of Technology

Least Mean Squares (LMS) update

Pick learning rate η > 0. Initialize w0 = 0 _∈ Rⁿ At time t, for t = 1, . . . , T , the algorithm

• observes input x_{t ∈} Rⁿ

• makes prediction w_{t−1 ·} x_{t ∈} R

• observes feedback y_t ∈ R, and

• updates its hypothesis as

wt = wt−1 − η(wt−1 · xt − y^t)xt

Main Results

• Techniques from machine learning lead to generalizations of LMS

• H^∞-optimal filtering in signal processing is similar to relative on-line loss bounds in machine learning

Motivation

• Non-Gaussian modeling

• Get away from rotation invariant algorithms

• Develop algorithms that work well when instances orthogonal and target weight vectors “sparse”

Expected Bounds for LMS

• Assume y_t = u_· x_t + ν_t, where ν_t iid with E[ν_t²] = ε. Then E

"

1 T

T

X

t=1

(u_· xt − wt−1 · xt)²

#

≤ ε + 1

T X₂²||u_||²₂

Better algorithms exist for probabilistic setting However our goal is to weaken the assumptions

H

bound for LMS

Assume ||x_t||2 ≤ X² for all t. Choose η = 1/X₂² For any u _∈ Rⁿ

PT

t=1(u_· x_{t −} w_{t−1 ·} x_t)² PT

t=1(u _· xt − y^t)² + X₂²||u_||²₂ ^{≤ 1}

• If some u with small norm is good predictor then LMS must approximate predictions of u

• Bound holds for any u ^{and (}xt, y_t)

• No probabilistic assumptions

• LMS is H^∞-optimal:

No algorithm can achieve ratio < 1

∀u ^{and (}xt, y_t)

Two related problems

A priori filtering: Control Theory

Try to match u_· xt

T

X

t=1

(u_· xt − w_{t−1 ·} xt)²

Prediction: On-line Learning

Try to match y_t

T

X

t=1

(y_t − w_{t−1 ·} x_t)²

Comparison of known LMS-related bounds

• For η = α/X₂²,

T

X

t=1

(u_· x_{t −} w_{t−1 ·} x_t)² ≤

T

X

t=1

(u _· x_{t − yt})² + X₂²||u_||²₂

• For η = α/X₂² (0 < α < 1)

T

X

t=1

(y_t − w_{t−1 ·} x_t)² ≤ 1 1 − α

T

X

t=1

(u_· x_{t − yt})²

| {z }

Lossu

+ 1

αX₂²||u_||²₂ tuned α

≤ Loss_u + 2√

Loss_uX2||u_||2 + X₂²||u_||²₂

[CBLW96]

Generalizing the LMS bound

• Replace ||x_||2||u_||2 by ||x_||p||u_||q where 1/p + 1/q = 1 and ||x_||p = (P

i|xⁱ|^p)^1/p

• Instead of comparing predictions to u _· xt for a fixed target u compare to u_{t ·} x_t where u_t may change

• Replace (·)² by more general loss

Basic LMS

∆_t = −η(w^t−1 · x^t − y^t)x_t

"!

#

"!

#

"!

#





∆_t

w_t−1 + wt

p-norm LMS

Write θ_t = f (w_t)

"!

#

"!

#





"!

#

"!

#

"!

#

'

&

$

% '

&

$

%

C C C





C C C







∆_t

w_t−1 w_t

θ_t−1 + θt

f f⁻¹

”W -space”

”Θ-space”

p-norm LMS

wt = f⁻¹(f⁽w_t−1⁾ _{− η(}w_{t−1 ·} xt − y^t)xt) where f^:Rⁿ _→ R^{n given by}

f_i(w) = sign(w_i)|wi|^q−1

||w_||^qq⁻²

and f_i⁻¹(θ) = sign(θ_i)|θi|^p−1

||θ_||^pp⁻²

When p = q = 2, then f(w) = w: LMS

For large p, f⁻¹ emphasizes differences in components

A priori filtering bound

Theorem Assume ||xt||^p ≤ X^p for all t, and let η = 1/((p − 1)Xp²) Then for any u the p-norm algorithm satisfies

T

X

t=1

(u _· xt − wt−1 · xt)² ≤

T

X

t=1

(y_t − u_· xt)² + (p − 1)Xp²||u_||²_q

• 1/p + 1/q = 1 and 2 ≤ p < ∞, 1 < q ≤ 2

• How do we get the dual norm pair (∞, 1) (where ||x_||∞ = max_i|xⁱ|)?

For p = 2 ln n, (p − 1)||x_||²_p||u_||q² ≤ (2e ln n)||x_||²_∞||u_||²₁

Comparison with basic LMS

New bounds incomparable with old ones because for p > 2 and q < 2

||x_||p < ||x_||2 and ||u_||q > ||u_||2

Compare p = 2 and p = O(log n) in two extreme cases:

Sparse target, dense instances: Let u = (1, 0, . . . , 0) and x = (1, . . . , 1).

• ||x_||²_2||u_||²₂ = n²

• (log n)||x_||²_∞||u_||²₁ = log n

• Thus large p better

Dense target, sparse instances: Let u = (1, 1, . . . , 1) and x = (1, 0, . . . , 0).

• ||x_||²_2||u_||²₂ = n²

• (log n)||x_||²_∞||u_||²₁ = n²log n

• Thus p = 2 better

The p-norm LMS can behave like EG

Hadamard Matrix:

→ +1 +1 +1 +1 instances → +1 −1 +1 −1

→ +1 +1 −1 −1

→ +1 −1 −1 +1

↑ ↑ ↑ ↑

targets

• Instances are orthogonal

• Target weight vectors are units

• LMS: error ≥ 1 − _n^k

• p-norm LMS with p = O(log n): error ≥ ^{ln n}_k

Time-varying target

Up to now, model has been yt = u _· xt + noise where target u is fixed Generalize this to y_t = ut · xt + noise where target ut may vary over time Example 1: target makes one jump Choose a,b _∈ R^{n and take}

ut =

 a for 1 ≤ t ≤ T /2 b for T /2 < t ≤ T

Example 2: target moves steadily Choose a,b _∈ Rⁿ and take ut = T − t

T − 1a + t − 1 T − 1b

Algorithms for time-varying target

Old update:

w⁰_t = f⁻¹(f(w_t−1) − η(w_{t−1 ·} x_{t − yt})x_t) Bounding update:

wt =

 w⁰_t if ||w⁰_{t||q ≤ Uq} U_q ^w0t

||w⁰t||^q otherwise where U_q > 0 is a norm bound

We rescale the weight vector whenever q-norm larger than U_q

Bound for time-varying target

Theorem Assume ||x_{t||p ≤ Xp} for all t, and let η = 1/((p − 1)Xp²) Then if ||ut||^q ≤ U^q for all t, the bounded p-norm LMS satisfies

T

X

t=1

(u_{t ·} x_{t −} w_{t−1 ·} x_t)² ≤

T

X

t=1

(y_t − u_{t ·} x_t)² + (p − 1)X_p²U_q²

+ 2(p− 1)Xp²U_q

T−1

X

t=1

||u_{t+1 −} u_t||q

• Only total distance P

t||u_{t+1 −} u_t||q traveled by the target matters

• Cost 2(p− 1)X_p²U_q per unit target movement

• For fixed target u_t+1 = u_t, we recover previous bound

• However U_q needs to be known in advance

Bregman divergences

• Key tool in analyzing and understanding the algorithms

• Fix strictly convex differentiable F :Rⁿ _→ R. Denote the gradient by f = ∇F .

• Now the Bregman divergence d_F:Rⁿ _× Rⁿ _→ R is

d_F(u,w) = F (u) − F (w) − f(w) · (u₋ w)

w u

dF(u, w) F

d_F(u,w) is the error of first- order Taylor approximation of F (u) around w

Basic properties of Bregman divergences

• d_F(u,w⁾ _{≥ 0, dF}⁽u,w^{) = 0 iff} u ⁼ w

• not symmetrical (in general)

• does not satisfy triangle inequality

• d_F(u,w) convex in u, not necessarily in w Connection to exponential families (roughly):

• F is cumulant function, f is link function

• w is expectation parameter, f⁽w) canonical parameter

• d_F(u,w) is the KL divergence between distributions parameterized by u and w

Example: p-norm divergence

•

F (w) = 1

2||w_||²_q

• Then the gradient f = ∇F satisfies f_i(w^{) = sign(wi)|wi|}

q−1

||w_||^q_q^{−2 and f}

i−1(θ^{) = sign(θi)|θi|}

p−1

||θ_||^p_p⁻²

• The divergence is

d_F(u,w^{) = 1}

2||u_||²_{q −} ¹

2||w_||²_{q −} f⁽w⁾ _{· (}u ₋ w).

• Special case p = q = 2 gives d_F(u,w^{) = 1}_2||u ₋ v_||²₂

Deriving the updates

• Define a regularized instantaneous loss C_t(w) = d_F(w,w_t−1) + η

2(y_t − w _· x_t)²

• Basic aim is to have

w_t = argmin

w

C_t(w)

• Minimize by setting ∇C^t(w_t) = 0, obtaining the implicit update f⁽wt) = f⁽wt−1) − η(wt · xt − y^t)xt

• Approximate wt · xt ≈ wt−1 · xt to obtain the update f(wt) = f(w_t−1) − η(w_{t−1 ·} xt − y^t)xt

Analyzing the update

• Measure of progress

d_F(u,w_t−1)− dF(u,w_t) = η(y_t − w_{t−1 ·} x_t)x_{t · (}u ₋ w_t−1) − dF(w_t−1,w_t)

• Massage the term (y_t − w_{t−1 ·} x_t)x_{t · (}u₋ w_t−1) until (u _· x_{t −} w_{t−1 ·} x_t)² and (y_t − u_· x_t)² appear; throw rest away

• Estimate d_F(wt−1,wt) in terms of ||xt||^p

• Sum over t = 1, . . . , T

Conclusion

• LMS and normalized LMS can be derived from an optimization problem involving a certain Bregman divergence

• Different Bregman divergences lead to different algorithms, with loss bounds in terms of different norms

• Bounds can be generalized for time-varying targets (and generalized linear models, not presented in the talk); proofs easy

• Algorithms for p = 2 can be kernelized, for p > 2 probably not Bottom line: Machinery from on-line machine learning carries over to H^∞-optimal filtering

Where are we headed?

• Develop p-norm Kalman filter

• Prove relative loss bounds