E(y)= β0+ β1x1+ β2x2+ β3x3+ β4x4+ β5x5
where x1, x2, . . . , x5are all quantitative variables that are not functions of other
independent variables.
Note: βi represents the slope of the line relating y to xi when all the other x’s are held fixed.
Recall that in the straight-line model (Chapter 3)
y = β0+ β1x+ ε
β0 represents the y-intercept of the line and β1 represents the slope of the line.
From our discussion in Chapter 3, β1 has a practical interpretation—it represents
the mean change in y for every 1-unit increase in x. When the independent variables are quantitative, the β parameters in the first-order model specified in the box have similar interpretations. The difference is that when we interpret the β that multiplies one of the variables (e.g., x1), we must be certain to hold the values of the remaining
independent variables (e.g., x2, x3) fixed.
To see this, suppose that the mean E(y) of a response y is related to two quantitative independent variables, x1and x2, by the first-order model
E(y)= 1 + 2x1+ x2
In other words, β0 = 1, β1= 2, and β2= 1.
Now, when x2= 0, the relationship between E(y) and x1is given by
E(y)= 1 + 2x1+ (0) = 1 + 2x1
A graph of this relationship (a straight line) is shown in Figure 4.1. Similar graphs of the relationship between E(y) and x1for x2= 1,
and for x2 = 2,
E(y)= 1 + 2x1+ (2) = 3 + 2x1
also are shown in Figure 4.1. Note that the slopes of the three lines are all equal to
β1= 2, the coefficient that multiplies x1.
Figure 4.1 Graphs of E(y)= 1 + 2x1+ x2for x2= 0,1,2 x1 x2 = 2 x2 = 1 x2 = 0 1 0 1 2 3 4 5 6 7 8 3 2 y
Figure 4.1 exhibits a characteristic of all first-order models: If you graph E(y) versus any one variable—say, x1—for fixed values of the other variables, the result
will always be a straight line with slope equal to β1. If you repeat the process for
other values of the fixed independent variables, you will obtain a set of parallel straight lines. This indicates that the effect of the independent variable xion E(y) is independent of all the other independent variables in the model, and this effect is measured by the slope βi (as stated in the box).
The first-order model is the most basic multiple regression model encountered in practice. In the next several sections, we present an analysis of this model.
4.4 Fitting the Model: The Method of Least Squares
The method of fitting multiple regression models is identical to that of fitting the straight-line model in Chapter 3—namely, the method of least squares. That is, we choose the estimated model
ˆy= ˆβ0+ ˆβ1x1+ · · · + ˆβkxk that minimizes
SSE =(yi− ˆyi)2
As in the case of the straight-line model, the sample estimates ˆβ0, ˆβ1, . . . , ˆβkwill be obtained as solutions to a set of simultaneous linear equations.∗
The primary difference between fitting the simple and multiple regression models is computational difficulty. The (k+ 1) simultaneous linear equations that
∗Students who are familiar with calculus should note that ˆβ
0, ˆβ1, . . . , ˆβkare the solutions to the set of equations
∂SSE/∂β0= 0, ∂SSE/∂β1= 0, . . . , ∂SSE/∂βk= 0. The solution, given in matrix notation, is presented in
must be solved to find the (k+ 1) estimated coefficients ˆβ0, ˆβ1, . . . , ˆβk are often difficult (tedious and time-consuming) to solve with a calculator. Consequently, we resort to the use of statistical computer software and present output from SAS, SPSS, and MINITAB in examples and exercises.
Example 4.1
A collector of antique grandfather clocks sold at auction believes that the price received for the clocks depends on both the age of the clocks and the number of bidders at the auction. Thus, he hypothesizes the first-order model
y = β0+ β1x1+ β2x2+ ε
where
y = Auction price (dollars) x1 = Age of clock (years)
x2 = Number of bidders
A sample of 32 auction prices of grandfather clocks, along with their age and the number of bidders, is given in Table 4.1.
(a) Use scattergrams to plot the sample data. Interpret the plots.
(b) Use the method of least squares to estimate the unknown parameters β0, β1,
and β2of the model.
(c) Find the value of SSE that is minimized by the least squares method.
GFCLOCKS
Table 4.1 Auction price data
Number of Auction Number of Auction Age, x1 Bidders, x2 Price, y Age, x1 Bidders, x2 Price, y
127 13 $1,235 170 14 $2,131 115 12 1,080 182 8 1,550 127 7 845 162 11 1,884 150 9 1,522 184 10 2,041 156 6 1,047 143 6 845 182 11 1,979 159 9 1,483 156 12 1,822 108 14 1,055 132 10 1,253 175 8 1,545 137 9 1,297 108 6 729 113 9 946 179 9 1,792 137 15 1,713 111 15 1,175 117 11 1,024 187 8 1,593 137 8 1,147 111 7 785 153 6 1,092 115 7 744 117 13 1,152 194 5 1,356 126 10 1,336 168 7 1,262
Solution
(a) MINITAB side-by-side scatterplots for examining the bivariate relationships between y and x1, and between y and x2, are shown in Figure 4.2. Of the
two variables, age (x1)appears to have the stronger linear relationship with
auction price (y). Figure 4.2 MINITAB
side-by-side scatterplots for the data of Table 4.1
(b) The model hypothesized is fit to the data in Table 4.1 with SAS. A portion of the printout is reproduced in Figure 4.3. The least squares estimates of the β parameters (highlighted) are ˆβ0 = −1,339, ˆβ1= 12.74, and ˆβ2 = 85.95.
Therefore, the equation that minimizes SSE for this data set (i.e., the least squares prediction equation) is
ˆy= −1,339 + 12.74x1+ 85.95x2
Figure 4.3 SAS regression output for the auction price model, Example 4.1
(c) The minimum value of the sum of the squared errors, also highlighted in Figure 4.3, is SSE= 516,727.
Example 4.2
Problem
Refer to the first-order model for auction price (y) considered in Example 4.1. Interpret the estimates of the β parameters in the model.
Solution
The least squares prediction equation, as given in Example 4.1, is ˆy= −1,339 + 12.74x1+ 85.95x2. We know that with first-order models, β1represents the slope of
the line relating y to x1 for fixed x2. That is, β1 measures the change in E(y) for
every one-unit increase in x1 when the other independent variable in the model is
held fixed. A similar statement can be made about β2: β2 measures the change in
E(y)for every one-unit increase in x2 when the other x in the model is held fixed.
Consequently, we obtain the following interpretations: ˆ
β1= 12.74: We estimate the mean auction price E(y) of an antique clock to
increase $12.74 for every 1-year increase in age (x1) when the number of bidders
(x2) is held fixed.
ˆ
β2= 85.95: We estimate the mean auction price E(y) of an antique clock to
increase $85.95 for every one-bidder increase in the number of bidders (x2)
when age (x1) is held fixed.
The value ˆβ0= −1,339 does not have a meaningful interpretation in this example.
To see this, note that ˆy= ˆβ0 when x1 = x2= 0. Thus, ˆβ0= −1,339 represents the
estimated mean auction price when the values of all the independent variables are set equal to 0. Because an antique clock with these characteristics—an age of 0 years and 0 bidders on the clock—is not practical, the value of ˆβ0has no meaningful
interpretation. In general, ˆβ0will not have a practical interpretation unless it makes
sense to set the values of the x’s simultaneously equal to 0.
4.5 Estimation of
σ2, the Variance of
ε
Recall that σ2 is the variance of the random error ε. As such, σ2 is an important measure of model utility. If σ2= 0, all the random errors will equal 0 and the prediction equation ˆy will be identical to E(y), that is, E(y) will be estimated without error. In contrast, a large value of σ2 implies large (absolute) values of ε and larger deviations between the prediction equation ˆy and the mean value E(y). Consequently, the larger the value of σ2, the greater will be the error in estimating the model parameters β0, β1, . . . , βk and the error in predicting a value of y for a specific set of values of x1, x2, . . . , xk. Thus, σ2 plays a major role in making inferences about β0, β1, . . . , βk, in estimating E(y), and in predicting y for specific values of x1, x2, . . . , xk.
Since the variance σ2 of the random error ε will rarely be known, we must
use the results of the regression analysis to estimate its value. Recall that σ2 is the
variance of the probability distribution of the random error ε for a given set of values for x1, x2, . . . , xk; hence, it is the mean value of the squares of the deviations of the y-values (for given values of x1, x2, . . . , xk) about the mean value E(y).∗Since ∗Because y= E(y) + ε, then ε is equal to the deviation y − E(y). Also, by definition, the variance of a random
variable is the expected value of the square of the deviation of the random variable from its mean. According to our model, E(ε)= 0. Therefore, σ2= E(ε2).
the predicted value ˆy estimates E(y) for each of the data points, it seems natural to use
SSE=(yi− ˆyi)2 to construct an estimator of σ2.