LEAST-MEAN-SQUARE ADAPTIVE FILTERS

(1)

LEAST-MEAN-SQUARE

ADAPTIVE FILTERS

(2)

LEAST-MEAN-SQUARE

ADAPTIVE FILTERS

Edited by

S. Haykin and B. Widrow

(3)

This book is printed on acid-free paper.

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 8400, fax (978) 750-4744. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, New Jersey 07030, (201) 748-6011, fax (201) 748-6008, E-Mail: [email protected].

For ordering and customer service, call 1-800-CALL-WILEY. Library of Congress Cataloging-in-Publication Data:

Least-mean-square adaptive ﬁlters/edited by S. Haykin and B. Widrow p. cm.

Includes bibliographical references and index. ISBN 0-471-21570-8 (cloth)

1. Adaptive ﬁlters—Design and construction—Mathematics. 2. Least squares. I. Widrow, Bernard, 1929- II. Haykin, Simon,

1931-TK7872.F5L43 2003 621.38150324—dc21

2003041161 Printed in the United States of America.

(4)

This book is dedicated to Bernard Widrow for inventing the LMS ﬁlter and investigating its theory and applications

(5)

CONTRIBUTORS

A. A. (LOUIS) BEEX, Systems Group—DSP Research Laboratory, The Bradley Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA 24061-0111

ROBERT R. BITMEAD, Department of Mechanical and Aerospace Engineering, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093-0411

HANS BUTTERWECK, Technische Universiteit Eindhoven, Faculteit Elektrotech-niek, EH 5.29, Postbus 513, 5600 MB Eindhoven, Netherlands

ZHE CHEN, Department of Electrical and Computer Engineering, CRL 102, McMaster University, 1280 Main Street West, Hamilton, Ontario, Canada L8S 4K1

DENIZ ERDOGMUS, Computational NeuroEngineering Laboratory, EB 451, Building 33, University of Florida, Gainesville, FL 32611

STEVEN L. GAY, Acoustics and Speech Research Department, Bell Labs, Room

2D-531, 600 Mountain Ave., Murray Hill, NJ 07974

PROF. DR.-ING. EBERHARD HA¨ NSLER, Institute of Communication Technology,

Darmstadt University of Technology, Merckstrasse 25, D-64283 Darmstadt, Germany

BABAK HASSIBI, Department of Electrical Engineering, 1200 East California

Blvd., M/C 136-93, California Institute of Technology, Pasadena, CA 91101 SIMON HAYKIN, Department of Electrical and Computer Engineering, McMaster

University, 1280 Main Street West, Hamilton, Ontario, Canada L8S 4K1 JOHN HOMER, School of Computer Science and Electrical Engineering, The

University of Queensland, Brisbane 4072

MAX KAMENETSKY, Stanford University, David Packard Electrical Engineering,

350 Serra Mall, Room 263, Stanford, CA 94305-9510

IVENM. Y. MAREELS, Department of Electrical and Electronic Engineering, The

University of Melbourne, Melbourne Vic 3010

(8)

V. H. NASCIMENTO, Department of Electronic Systems Engineering, University of Sa˜o Paulo, Brazil

JOSE C. PRINCIPE, Computational NeuroEngineering Laboratory, EB 451, Building 33, University of Florida, Gainesville, FL 32611

YADUNANDANA N. RAO, Computational NeuroEngineering Laboratory, EB 451,

Building 33, University of Florida, Gainesville, FL 32611

ALIH. SAYED, Department of Electrical Engineering, Room 44-123A Engineering

IV Bldg, University of California, Los Angeles, CA 90095-1594

GERHARD UWE SCHMIDT, Institute of Communication Technology, Darmstadt

University of Technology, Merckstrasse 25, D-64283 Darmstadt, Germany BERNARD WIDROW, Stanford University, David Packard Electrical Engineering,

350 Serra Mall, Room 273, Stanford, CA 94305-9510

JAMES R. ZEIDLER, Department of Electrical and Computer Engineering, University of California, San Diego, La Jolla, CA 92092

(9)

INTRODUCTION: THE LMS FILTER

(ALGORITHM)

SIMON HAYKIN

The earliest work on adaptive filters may be traced back to the late 1950s, during which time a number of researchers were working independently on theories and applications of such filters. From this early work, the least-mean-square ðLMSÞ algorithmemerged as a simple, yet effective, algorithm for the design of adaptive transversal (tapped-delay-line) filters.

The LMS algorithm was devised by Widrow and Hoff in 1959 in their study of a pattern-recognition machine known as the adaptive linear element, commonly referred to as the Adaline [1, 2]. The LMS algorithm is a stochastic gradient algorithm in that it iterates each tap weight of the transversal ﬁlter in the direction of the instantaneous gradient of the squared error signal with respect to the tap weight in question.

Letww^ðnÞdenote the tap-weight vector of the LMS ﬁlter, computed at iteration (time step) n. The adaptive operation of the ﬁlter is completely described by the recursive equation (assuming complex data)

^

w

wðnþ1Þ ¼ww^ðnÞ þmuðnÞ½dðnÞ ww^HðnÞuðnÞ*; ð1Þ whereuðnÞis the tap-input vector,dðnÞis the desired response, andmis the step-size parameter. The quantity enclosed in square brackets is the error signal. The asterisk denotes complex conjugation, and the superscriptHdenotes Hermitian transposition (i.e., ordinary transposition combined with complex conjugation).

Equation (1) is testimony to the simplicity of the LMS filter. This simplicity, coupled with desirable properties of the LMS filter (discussed in the chapters of this book) and practical applications [3, 4], has made the LMS filter and its variants an important part of the adaptive signal processing kit of tools, not just for the past 40 years but for many years to come. Simply put, the LMS filter has withstood the test of time.

Although the LMS ﬁlter is very simple in computational terms, its mathematical analysis is profoundly complicated because of its stochastic and nonlinear nature. Indeed, despite the extensive effort that has been expended in the literature to

(10)

analyze the LMS ﬁlter, we still do not have a direct mathematical theory for its stability and steady-state performance, and probably we never will. Nevertheless, we do have a good understanding of its behavior in a stationary as well as a nonstationary environment, as demonstrated in the chapters of this book.

The stochastic nature of the LMS filter manifests itself in the fact that in a stationary environment, and under the assumption of a small step-size parameter, the filter executes a form ofBrownian motion.Specifically, the small step-size theory of the LMS filter is almost exactly described by the discrete-time version of the Langevin equation1[3]:

DnkðnÞ ¼nkðnþ1Þ nkðnÞ

¼ mlknkðnÞ þfkðnÞ; k¼1;2;. . .;M; ð2Þ

which is naturally split into two parts: a damping forcemlknkðnÞand a stochastic forcef_kðnÞ. The terms used herein are deﬁned as follows:

M¼order (i.e., number of taps) of the transversal ﬁlter around which the LMS ﬁlter is built

lk¼kth eigenvalue of the correlation matrix of the input vectoruðnÞ, which

is denoted byR

fkðnÞ ¼kth component of the vectormQHuðnÞe*_oðnÞ

Q¼unitary matrix whose M columns constitute an orthogonal set of eigerivectors associated with the eigenvalues of the correlation matrixR

eoðnÞ ¼optimum error signal produced by the corresponding Wiener ﬁlter

driven by the input vectoruðnÞand the desired responsedðnÞ

To illustrate the validity of Eq. (2) as the description of small step-size theory of the LMS ﬁlter, we present the results of a computer experiment on a classic example of adaptive equalization. The example involves an unknown linear channel whose impulse response is described by the raised cosine [3]

hn ¼ 1 2 1þcos 2p W ðn2Þ ; n¼1;2;3; 0; otherwise 8 < : ð3Þ

where the parameterW controls the amount of amplitude distortion produced by the channel, with the distortion increasing with W. Equivalently, the parameter W

controls the eigenvalue spread (i.e., the ratio of the largest eigenvaiue to the smallest eigenvalue) of the correlation matrix of the tap inputs of the equalizer, with the eigenvalue spread increasing with W. The equalizer has M¼11 taps. Figure 1 presents the learning curves of the equalizer trained using the LMS algorithm with the step-size parameter m ¼0:0075 and varying W. Each learning curve was obtained by averaging the squared value of the error signaleðnÞversus the number of iterations n over an ensemble of 100 independent trials of the experiment. The

1

The Langevin equation is the “engineer’s version” of stochastic differential (difference) equations.

(11)

continuous curves shown in Figure 1 are theoretical, obtained by applying Eq. (2). The curves with relatively small ﬂuctuations are the results of experimental work. Figure 1 demonstrates close agreement between theory and experiment.

It should, however, be reemphasized that application of Eq. (2) is limited to small values of the step-size parameterm. Chapters in this book deal with cases whenmis large.

REFERENCES

1. B. Widrow and M. E. Hoff, Jr. (1960). “Adaptive Switching Circuits,”IRE WESCON Conv. Rec.,Part 4, pp. 96 – 104.

2. B. Widrow (1966). “Adaptive Filters I: Fundamentals,” Rep. SEL-66-126 (TR-6764-6), Stanford Electronic Laboratories, Stanford, CA.

3. S. Haykin (2002).Adaptive Filter Theory, 4th Edition, Prentice-Hall.

4. B. Widrow and S. D. Stearns (1985).Adaptive Signal Processing, Prentice-Hall.

Figure 1 Learning curves of the LMS algorithm applied to the adaptive equalization of a communication channel whose impulse response is described by Eq. (3) for varying eigenvalue spreads: Theory is represented by continuous well-deﬁned curves. Experimental results are represented by ﬂuctuating curves.

LEAST-MEAN-SQUARE ADAPTIVE FILTERS

LEAST-MEAN-SQUARE

ADAPTIVE FILTERS

LEAST-MEAN-SQUARE

ADAPTIVE FILTERS

S. Haykin and B. Widrow

CONTENTS

CONTRIBUTORS

INTRODUCTION: THE LMS FILTER

(ALGORITHM)