7. Section 2.8 introduces the notion of instrumental variables to deal with the “errors-
in-variables” problem.
Section 2.9 provides a summary and discussion that conclude the chapter.
2.1 INTRODUCTION
The idea of model building shows up practically in every discipline that deals with statistical
data analysis. Suppose, for example, we are given a set of random variables and the as-
signed task is to find the relationships that may exist between them, if any. In regression, which is a special form of function approximation, we typically find the following scenario: • One of the random variables is considered to be of particular interest; that random
variable is referred to as a dependent variable, or response.
• The remaining random variables are called independent variables, or regressors; their role is to explain or predict the statistical behavior of the response.
• The dependence of the response on the regressors includes an additive error term, to account for uncertainties in the manner in which this dependence is formulated;
Section 2.2 Linear Regression Model: Preliminary Considerations 69
the error term is called the expectational error, or explanational error, both of which are used interchangeably.
Such a model is called the regression model.1
There are two classes of regression models: linear and nonlinear. In linear regres-
sion models, the dependence of the response on the regressors is defined by a linear func-
tion, which makes their statistical analysis mathematically tractable. On the other hand, in nonlinear regression models, this dependence is defined by a nonlinear function, hence the mathematical difficulty in their analysis. In this chapter, we focus attention on linear regression models. Nonlinear regression models are studied in subsequent chapters.
The mathematical tractability of linear regression models shows up in this chapter in two ways. First, we use Bayesian theory2to derive the maximum a posteriori estimate of the vector that parameterizes a linear regression model. Next, we view the parameter estimation problem using another approach, namely, the method of least squares, which is perhaps the oldest parameter-estimation procedure; it was first derived by Gauss in the early part of the 19th century. We then demonstrate the equivalence between these two approaches for the special case of a Gaussian environment.
2.2 LINEAR REGRESSION MODEL: PRELIMINARY CONSIDERATIONS
Consider the situation depicted in Fig. 2.1a, where an unknown stochastic environment is the focus of attention. The environment is probed by applying a set of inputs, consti- tuting the regressor
(2.1) where the superscript T denotes matrix transposition. The resulting output of the envi- ronment, denoted by d, constitutes the corresponding response, which is assumed to be scalar merely for the convenience of presentation. Ordinarily, we do not know the func- tional dependence of the response d on the regressor x, so we propose a linear regres- sion model, parameterized as:
(2.2)
where w1, w2, ..., wMdenote a set of fixed, but unknown, parameters, meaning that the en- vironment is stationary. The additive term ε, representing the expectational error of the model, accounts for our ignorance about the environment. A signal-flow graph depiction of the input–output behavior of the model described in Eq. (2.2) is presented in Fig. 2.1b.
Using matrix notation, we may rewrite Eq. (2.2) in the compact form
(2.3) where the regressor x is defined in terms of its elements in Eq. (2.1). Correspondingly, the parameter vector w is defined by
(2.4) w = [w1, w2, ..., wM]T d = wT x + ε d = a M j = 1 wjxj + ε x = [x1, x2, ..., xM]T
whose dimensionality is the same as that of the regressor x; the common dimension
M is called the model order. The matrix term wTx is the inner product of the vectors w and x.
With the environment being stochastic, it follows that the regressor x, the response
d, and the expectational error ε are sample values (i.e., single-shot realizations) of the
random vector X, the random variable D, and the random variable E, respectively. With such a stochastic setting as the background, the problem of interest may now be stated as follows:
Given the joint statistics of the regressor X and the corresponding response D, estimate the unknown parameter vector w.
When we speak of the joint statistics, we mean the following set of statistical parameters: • the correlation matrix of the regressor X;
• the variance of the desired response D;
• the cross-correlation vector of the regressor X and the desired response D. It is assumed that the means of both X and D are zero.
In Chapter 1, we discussed one important facet of Bayesian inference in the con- text of pattern classification. In this chapter, we study another facet of Bayesian infer- ence that addresses the parameter estimation problem just stated.
FIGURE 2.1 (a) Unknown stationary stochastic environment. (b) Linear regression model of the environment. Desired response (Output) d Regressor (Input vector) x (a) (b) Unknown stochastic environment: w Expectational error Desired response d w1 w2 wM e xM x2 x1 Regressor x • • • • • •
2.3 MAXIMUM A POSTERIORI ESTIMATION OF THE PARAMETER VECTOR
The Bayesian paradigm provides a powerful approach for addressing and quantifying the uncertainty that surrounds the choice of the parameter vector w in the linear re- gression model of Eq. (2.3). Insofar as this model is concerned, the following two re- marks are noteworthy:
1. The regressor X acts as the “excitation,” bearing no relation whatsoever to the parameter vector w.
2. Information about the unknown parameter vector W is contained solely in the desired response D that acts as the “observable” of the environment.
Accordingly, we focus attention on the joint probablity density function of W and D, con- ditional on X.
Let this density function be denoted by pW,D | X(w, d | x). From probability theory, we know that this density function may be expressed as
(2.5) Moreover, we may also express it in the equivalent form
(2.6) In light of this pair of equations, we may go on to write
(2.7)
provided that . Equation (2.7) is a special form of Bayes’s theorem; it em- bodies four density functions, characterized as follows:
1. Observation density:This stands for the conditional probability density function
pD | W, X(d | w, x), referring to the “observation” of the environmental response d due to the regressor x, given the parameter vector w.
2. Prior:This stands for the probability density function pW(w), referring to infor- mation about the parameter vector w, prior to any observations made on the en- vironment. Hereafter, the prior is simply denoted by π(w).
3. Posterior density:This stands for the conditional probability density function
pW|D, X(w | d, x), referring to the parameter vector w “after” observation of the environment has been completed. Hereafter, the posterior density is denoted by π(w | d, x). The conditioning response–regressor pair (x, d) is the “observation model,” embodying the response d of the environment due to the regressor x.
4. Evidence:This stands for the probability density function pD(d), referring to the “information” contained in the response d for statistical analysis.
pD(d) Z 0 pWD, X(w
d, x) = pD W, X(dw, x)pW(w) pD(d) pW, DX(w, dx) = pDW, X(dw, x)pW(w) pW, DX(w, dx) = pWD, X(wd, x)pD(d)Section 2.3 Maximum A Posteriori Estimation of the Parameter Vector 71
The observation density pD|W, X(d | w, x) is commonly reformulated mathematically as the likelihood function, defined by
(2.8) l(w
d, x) = pDW, X(dw, x)Moreover, insofar as the estimation of the parameter vector w is concerned, the evi- dence pD(d) in the denominator of the right-hand side of Eq. (2.7) plays merely the role of a normalizing constant. Accordingly, we may express Eq. (2.7) in words by stating the following:
The posterior density of the vector w parameterizing the regression model is proportional to the product of the likelihood function and the prior.
That is,
(2.9) where the symbol signifies proportionality.
The likelihood function l(w| d, x), considered on its own, provides the basis for the
maximum-likelihood (ML) estimate of the parameter vector w, as shown by
(2.10) For a more profound estimate of the parameter vector w, however, we look to the posterior density (w|d, x). Specifically, we define the maximum a posteriori (MAP)
estimate of the parameter vector w by the formula
(2.11) We say that the MAP estimator is more profound than the ML estimator for two important reasons:
1. The Bayesian paradigm for parameter estimation, rooted in the Bayes’ theo-
rem as shown in Eq. (2.7) and exemplified by the MAP estimator of Eq. (2.11),
exploits all the conceivable information about the parameter vector w. In contrast,
the ML estimator of Eq. (2.10) lies on the fringe of the Bayesian paradigm, ignoring the prior.
2. The ML estimator relies solely on the observation model (d, x) and may there-
fore lead to a nonunique solution. To enforce uniqueness and stability on the solution, the prior(w) has to be incorporated into the formulation of the estima-
tor ; this is precisely what is done in the MAP estimator.
Of course, the challenge in applying the MAP estimation procedure is how to come up with an appropriate prior, which makes MAP more computationally demanding than ML.
One last comment is in order. From a computational perspective, we usually find it more convenient to work with the logarithm of the posterior density rather than the posterior density itself. We are permitted to do this, since the logarithm is a monotoni- cally increasing function of its argument. Accordingly, we may express the MAP esti- mator in the desired form by writing
(2.12) where “log” denotes the natural logarithm. A similar statement applies to the ML estimator.
wMAP = arg max
w log((w
d, x)) wMAP = arg maxw (w
d, x) wML = arg maxw l(w
d, x)Parameter Estimation in a Gaussian Environment
Let xiand didenote the regressor applied to the environment and the resulting response,