Statistical properties of the OLS estimator

(1)

Statistical properties of the OLS estimator

Gábor Ruzsa

Corvinus University Budapest

(2)

Statistical properties of the OLS estimator

Outline

1 Closed form solution for the OLS estimator

2 The classical linear regression model assumptions

3 Statistical properties of the OLS estimator

(3)

Statistical properties of the OLS estimator Closed form solution for the OLS estimator

Deriving the statistical properties of the OLS estimator

To derive the statistical properties of the OLS estimator and perform statistical inference, we need to specify:

1 the population regression model

2 the sampling method and its consequences for the probability distribution of sample values

3 the OLS estimator expressed as a function of the observed sample values

(4)

Deriving the statistical properties of the OLS estimator

â The population model and the consequences of the sampling method are described by a set of assumptions, known as

• Classical Linear Regression Model (CLRM) assumptions; or

• Gauss-Markov assumptions

â The starting point for our analysis is a sample of observations indexed by

i = 1, 2, . . . , n

.

• observed regressor values: (xi1, . . . , xik)

• observed dependent variable values: yi

• values of the error term (unobserved): ui

â The regression model is assumed to hold for each sample element

i

:

y

i

= x

⁰_i

β + u

i

= β

0

+ β

1

x

i1

+ . . . + β

k

x

ik

+ u

i

(5)

Data structure in matrix representation

• List of dependent variable values:

y =





 y

1

y

2 .. .

y

_n







• List of error term values:

u =





 u

1

u

2 .. .

u

n







• Matrix of regressor values:

X =







1 x

11

. . . x

1j

. . . x

1k ..

. ... ... ...

1 x

_i1

. . . x

_ij

. . . x

_ik

..

. ... ... ...

1 x

n1

. . . x

nj

. . . x

nk







(6)

Data structure in matrix representation

X =







1 x

11

. . . x

1j

. . . x

1k ..

. ... ... ...

1 x

_i1

. . . x

_ij

. . . x

_ik

..

. ... ... ...

1 x

n1

. . . x

nj

. . . x

nk







=





 [ x

⁰₁

]

.. .

[ x

⁰_i

]

.. .

[ x

⁰_n

]







←

vector of observed regressor values for individual

i

=











 1











 X

1





 . . .





 X

j





 . . .





 X

k













↑

list of observed sample values for regressor

x

j

(7)

Data structure in matrix representation

Example: Individual determinants of wages earned by employees

Vector of regressors:

[x

1

, x

2

, x

3

, x

4

, x

5

] = [age, male, iq, educ, exper]

→

x⁰1= [1, 26, 0, 132, 17, 2]

x⁰2= [1, 52, 1, 114, 12, 33]

.. .

x⁰n= [1, 38, 0, 125, 15, 16]

→

X1=





 26 52 .. . 38





 X2=





 0 1 .. . 0





 X3=





 132 114 .. . 125





 X4=





 17 12 .. . 15





 X5=





 2 33

.. . 16







↑ list of ages

↑ gender list

↑ list of IQ scores

↑ list of educ.

levels

↑ list of exper.

levels

(8)

Matrix representation for the regression model

• Regression model for the sample as a whole:

y = Xβ + u





 y

₁

y

₂ .. .

y

n







=







1 x

₁₁

. . . x

_1k

1 x

₂₁

. . . x

_2k

..

. ... ...

1 x

n1

. . . x

nk











 β

₀

β

₁ .. .

β

k





 +





 u

₁

u

₂ .. .

u

n







• Vector of fitted values:

y = ˆ





 ˆ y

1

ˆ y

2

.. .

ˆ y

_n







→ ˆy = X ˆβ

• Vector of residual values:

ˆ u =





 ˆ u

₁

ˆ u

₂

.. .

ˆ u

n







→ ˆu = y − ˆy

(9)

Algebra of OLS in matrix notation (*)

â Idea: construct a regression hyperplane that fits best to the observed data.

• Choose ˆβ0, ˆβ1, . . . , ˆβk

such that Pn

i=1uˆ²i → min SSE =

n

X

i=1

yi− ˆβ0− ˆβ1xi1− . . . − ˆβkxik

2

| {z }

= (yi− ˆyi)²= ˆu²_i

→ min

â First order conditions of optimality

• FOC0: ∂(·)

∂ ˆβ0

= −2

n

X

i=1

| {z }

= (yi− ˆyi) = ˆui

= 0

⇐⇒ ˆu⁰1 = 0

• FOCj: ∂(·)

∂ ˆβj

= −2

n

X

i=1

| {z }

= (yi− ˆyi) = ˆui

xij= 0

⇐⇒ ˆu⁰Xj= 0

(10)

Algebra of OLS in matrix notation (*)

â Matrix representation for the system of normal equations ˆ

u⁰[1, X1, . . . , Xk] = [0, 0, . . . , 0], i.e.uˆ⁰X = 0⁰

⇐⇒ X⁰ˆu = 0

⇐⇒ X⁰

y − X ˆβ

= 0

⇐⇒ X⁰y = X⁰X ˆβ

â Linear algebraic solution

• IfX⁰Xis non-singular, then β = (Xˆ ⁰X)⁻¹X⁰y is the unique solution to the OLS normal equations.

System of OLS normal equations and the OLS estimator

• X⁰X ˆβ = X⁰y ^• β = (Xˆ ⁰X)⁻¹X⁰y

(11)

Statistical properties of the OLS estimator The classical linear regression model assumptions

Classical Linear Regression Model assumptions

CLRM.1 Linear population model

The linear regression model

y = x

⁰

β = β

0

+ β

1

x

1

+ . . . + β

k

x

k

+ u

holds for each element of the population.

CLRM.2 Random sampling

A simple random sample of elements and corresponding variable values is taken from the population:

(y

i

, x

i1

, . . . , x

ik

, u

i

)

for each

i = 1, 2, . . . , n

.

CLRM.3 No perfect multicollinearity

The regressors are linearly independent of each other, i.e. no linear combination of

(x

1

, . . . , x

k

)

is constant in the sample.

(12)

Classical Linear Regression Model assumptions

CLRM.4 Exogeneity of regressors

E (u | x) = 0

for any combination of regressor values

x = (x

1

, . . . , x

k

)

, i.e.

the mean of the error term is independent of the level of regressors.

CLRM.5 Homoskedasticity

Var (u | x) = σ

² for any combination of regressor values

x = (x

1

, . . . , x

k

)

, i.e. the variance of the error term is independent of the level of regressors.

CLRM.6 Uncorrelated errors / Absence of residual autocorrelation

Cov (u

i

, u

j

| x

i

, x

j

) = 0

for any pair

(i, j)

of observations, i.e. the error term is uncorrelated across sample elements.

(13)

Classical Linear Regression Model assumptions

CLRM.7 Normally distributed errors

F (u | x) ∼ N

, i.e. the error term is normally distributed for any level of the regressors.

(14)

Comments on the CLRM assumptions

CLRM.1 Linear population model

The linear regression model

y = x

⁰

β = β

0

+ β

1

x

1

+ . . . + β

k

x

k

+ u

holds for each element of the population.

â The assumption of linearity is restrictive about how the regression coefficients are attached to the variables but is NOT restrictive about the functional form in which the variables enter the regression model.

â Functional transformations of the variables and different kinds of interaction effects can be easily accomodated to the linear model.

→ see General Linear Model

(15)

Comments on the CLRM assumptions

CLRM.2 Random sampling

A simple random sample of elements and corresponding variable values is taken from the population:

(y

i

, x

i1

, . . . , x

ik

, u

i

)

for each

i = 1, 2, . . . , n

.

â For a simple random sample, the distributional properties of the population are inherited by the sample elements.

→ The probability distribution of (yi, xi1, . . . , xik, ui) is identical with the population frequency distribution of (y, x1, . . . , xk, u).

â A simple random cross-sectional sample taken from a sufficiently large population yields IID (independent and identically distributed) variable values.

IID property

Considering any particular variable

w ∈ (y, x

₁

, . . . , x

_k

, u)

, the values of

w

are independent and identically distributed across sample elements.

(16)

Comments on the CLRM assumptions

â The IID property doesn’t mean that the components of

x = (x

1

, . . . , x

k

)

should be independent of each other. On the contrary, in most cases there is a noticeable pattern of correlation among the regressors.

Example: Individual determinants of wages earned by employees

Vector of regressors:

(x

1

, x

2

, x

3

, x

4

, x

5

) = [age, male, iq, educ, exper]

• By the IID property, each individual to be selected to the sample has the same probability of being of a certain age and gender, having a certain level of intelligence, education, and work experience.

• Futhermore, the observed values of these characteristics for individuals already in the sample has no influence on the variable values for sample elements yet to be selected.

• On the other hand, we are likely to find a noticeable pattern of relationship among the regressors:

. positive correlation between [iq] and [educ]

. positive correlation between [age] and [exper]

. negative correlation between [educ] and [exper]

(17)

Comments on the CLRM assumptions

CLRM.3 No perfect multicollinearity

The regressors are linearly independent of each other, i.e. no linear combination of

(x

1

, . . . , x

k

)

is constant in the sample.

â CLRM.3 rules out the possibility of having constant variables in the model, i.e. each regressor must exhibit some variability in the sample.

â Perfect multicollinearity means that a set of explanatory variables are perfectly correlated, so there’s no way of separating their partial effects.

• Mathematically: X⁰X is singular, thus (X⁰X)⁻¹ doesn’t exist and the OLS estimator β = (Xˆ ⁰X)⁻¹X⁰y cannot be computed.

• Solution: drop one of the redundant variables from the model.

(18)

Comments on the CLRM assumptions

E (u | x) = 0

x = (x

1

, . . . , x

k

)

, i.e.

â

Cov (u, x) = 0

follows as a direct consequence, i.e. the error term is uncorrelated with each regressor.

• Actually, uncorrelatedness is a weaker property than exogeneity: it only rules out the existence of a linear relationship between the mean of the error term and the level of regressors.

â Exogeneity (implying uncorrelatedness) is a crucial assumption for having an unbiased estimator of

β

_j.

• If a regressor (xj) and the error term were correlated, the OLS estimates would attribute to xj some of the variation in y that actually came from the error term.

→ That is to say, the OLS estimator βˆj would be biased for βj.

(19)

Comments on the CLRM assumptions

Var (u | x) = σ

x = (x

1

, . . . , x

k

)

â Homoskedasticity means that each realization of the error term is drawn from a distribution with the same variance and thus varies to the same extent around the regression line.

â Heteroskedasticity refers to the contrary case in which the variance of the error term depends on the level of regressors.

• When the model is heteroskedastic the OLS estimator loses its efficiency.

• Even though the estimated coefficients may remain unbiased, they become less reliable, i.e. their standard errors increase.

(20)

Comments on the CLRM assumptions

Cov (u

_i

, u

_j

| x

_i

, x

_j

) = 0

for any pair

(i, j)

â Residual autocorrelation means that the error term – the residual component of a regression model – exhibits positive or negative correlation across sample elements.

â Residual autocorrelation (typically positive) often arises in time series regression models, whereby a random shock observed in one time period affects the random shocks occurring in subsequent time periods.

â In cross-sectional regression models residual autocorrelation can generally be ruled out.

• Simple random sampling yields IID values for each variable as well as for the error term. Absence of residual autocorrelation follows as a direct consequence.

(21)

Comments on the CLRM assumptions

F (u | x) ∼ N

â Normality of the error term is an auxiliary assumption.

• It is necessary for deriving the exact (small sample) sampling distribution of the OLS estimator βˆ.

• It is not necessary for the normality of the asymptotic (large sample) sampling distribution of βˆ (which is guaranteed by the Central Limit Theorem).

• It is not necessary for proving the unbiasedness, the consistency and the efficiency of the OLS estimator.

• Neither is it necessary for obtaining the OLS standard errors.

(22)

Extension of the CLRM assumptions to IID samples (*)

E (u | x) = 0

x = (x

1

, . . . , x

k

)

, i.e.

â Consequence for the sample:

E (u

i

| x

i

) = 0

for each element

i

.

• In the case of independent sample elements, the conditioning set can be extended to the whole data matrix: E (ui| X) = 0.

• For the sample as a whole: E (u | X) = 0. (strict exogeneity) Strict exogeneity

E (u | X) = 0

, i.e. the expected sample values of the error term are zero for any particular realization of the data matrix.

(23)

Extension of the CLRM assumptions to IID samples (*)

Var (u | x) = σ

x = (x

1

, . . . , x

k

)

Var (u

_i

| x

i

) = σ

² for each element

i

.

• In the case of independent sample elements, the conditioning set can be extended to the whole data matrix: Var (ui| X) = σ².

Cov (u

_i

, u

_j

| x

_i

, x

_j

) = 0

for any pair

(i, j)

• In the case of independent sample elements, the conditioning set can be extended to the whole data matrix: Cov (ui, uj| X) = 0.

(24)

Extension of the CLRM assumptions to IID samples (*)

Homoskedasticity combined with the absence of residual autocorrelation yields spherical error terms for the sample as a whole.

Spherical error term

Var (u | X) = σ

²

I

n×n, i.e. sample values of the error term are uncorrelated and they have constant and equal variances for any particular realization of the data matrix.

â Components of the sampling variance-covariance matrix for the error term:







Var ( u1| X) . . . Cov ( u1, ui| X) . . . Cov ( u1, un| X) ..

.

.. .

.. . Cov ( ui, u1| X) . . . Var ( ui| X) . . . Cov ( ui, un| X)

.. .

.. . Cov ( un, u1| X) . . . Cov ( un, ui| X) . . . Var ( un| X)







=







σ². . . 0 . . . 0 ..

. .. .

.. . 0 . . . σ². . . 0

.. .

.. . 0 . . . 0 . . . σ²







(25)

Extension of the CLRM assumptions to IID samples (*)

F (u | x) ∼ N

Pr (u

_i

| x

i

) ∼ N

for each element

i

.

• In the case of independent sample elements, the conditioning set can be extended to the whole data matrix: Pr (ui| X) ∼ N.

• For the sample as a whole: Pr (u | X) ∼ N. (joint normality) â Notation

• F (·): population frequency distribution

• Pr (·): probability distribution

(26)

Extension of the CLRM assumptions to IID samples (*)

â Combining all previous assumptions allows us to fully characterize the probability distribution of the error term conditionally on the data matrix.

• type of distribution: Pr (u | X) ∼ N

• expected value: E (u | X) = 0

• variance-covariance matrix: Var (u | X) = σ²In×n

â These results provide the basis for deriving the statistical properties of the OLS estimator.

(27)

Statistical properties of the OLS estimator Statistical properties of the OLS estimator

OLS estimator – Expected value

Unbiasedness

Under CLRM.1-4, the OLS estimator is unbiased:

E ˆ β X

= β

â Cases in which the OLS estimator is biased:

1 Functional misspecification of the model (violation of CLRM.1)

2 Non-random sample selection (violation of CLRM.2)

→sample selection bias

3 Correlation between any of the regressors and the error term (violation of CLRM.4)

→spurious correlation, omitted variables bias, endogeneity bias, etc.

(28)

Algebra for obtaining the expected value (*)

â OLS estimator and the sampling error

• β = (Xˆ ⁰X)⁻¹X⁰y = β + (X⁰X)⁻¹X⁰u

• β − β = (Xˆ ⁰X)⁻¹X⁰u

â Expected value of the sampling error

• E ˆβ − β X

= (X⁰X)⁻¹X⁰E ( u | X) = 0

â Expected value of the OLS estimator

• E ˆβ X

= β + (X⁰X)⁻¹X⁰E ( u | X) = β

(29)

OLS estimator – Sampling variances and covariances

Matrix of sampling variances and covariances

Under CLRM.1-6, the sampling variance-covariance matrix of the OLS estimator is:

Var ˆ β

X

= σ

²

(X

⁰

X)

⁻¹

â Components of the sampling variance-covariance matrix for

β ˆ

:

Var ˆβ X

=







Var ˆβ0

X

. . . Cov ˆβ0, ˆβj

X

. . . Cov ˆβ0, ˆβk

X ..

.

.. .

.. . Cov ˆβj, ˆβ0

X

. . . Var ˆβj

X

. . . Cov ˆβj, ˆβk

X ..

.

.. .

.. . Cov ˆβk, ˆβ0

X

. . . Cov ˆβk, ˆβj

X

. . . Var ˆβk

X







(30)

OLS estimator – Standard errors

OLS standard errors

• SE ˆβj

= r

dVar ˆβj

X

= s

ˆ σ²

(n − 1) var (xj) 1 − R²_j

• R²j= R_x²_j·{x₁,...,x_j−1,x_j+1,...,x_k} is the multiple coefficient of determination obtained from regressing xj on (x1, . . . , xj−1, xj+1, . . . , xk)

â The estimator is more stable

• the lower the residual variance

ˆ

σ²= dVar (u) = SSE /(n − k − 1)

→ i.e. the greater the model’s explanatory power

• the bigger the sample size (n)

• the greater the sample variance of xj (var (xj))

• the weaker the linear dependence between xj and all other regressors R²_j

(31)

OLS estimator – Consistency

Consistency

Under CLRM.1-4, the OLS estimator is consistent:

plim

n→∞

β ˆ

_(n)

= β

â Convergence in probability (plim):

n→∞lim Pr

β − ε < ˆβ_(n)< β + ε

= 1 for any positive margin ε > 0

• As the sample size is increased, the parameter estimates converge, with asymptotic probability 1, to the true population regression coefficient values.

â Consistency is the most important property of any estimator.

(32)

Unbiasedness, stability, consistency – Putting it together

â Desirable properties for an estimator:

• unbiasedness: E ˆβ X

= β

• high stability: Var ˆβ X

the smaller, the better

• consistency: plim

n→∞

βˆ_(n)= β

â How are these three properties related to each other?

• Unbiasedness and stability are independent properties.

• Consistency is related to both.

. consistency implies asymptotic unbiasedness:

n→∞lim E ˆβ_(n) X

= β

. consistency implies infinitely high asymptotic stability:

n→∞lim Var ˆβ_(n) X

= 0

(33)

Unbiasedness, stability, consistency – Putting it together

â Unbiasedness and stability are independent properties.

high stability low stability

unbiased

biased

(34)

OLS estimator – Efficiency

Gauss-Markov theorem

Under CLRM.1-6, the OLS estimator is the most efficient among all linear unbiased estimators of

β

.

â Efficiency refers to the property of having a small sampling variance in comparison with other estimators (for the same set of model parameters).

â By the Gauss-Markov theorem, no other linear unbiased estimator of

β

exists which has a smaller sampling variance than the OLS estimator.

• The OLS estimator has the minimum variance, i.e. it is the most efficient.

• The OLS estimator is BLUE (best linear unbiased estimator).

(35)

OLS estimator – Normality

Normal sampling distribution

Under CLRM.1-7, the sampling distribution of the OLS estimator is normal:

Pr ˆ β | X

∼ N

β, σ

²

(X

⁰

X)

⁻¹

â The critical assumption for deriving the normality of the OLS estimator is the joint normality of the vector of errors:

Pr (u | X) ∼ N

(see CLRM.7)

• The normality of βˆ follows from the fact that it is a linear combination of the vector of errors.

â This result will be helpful for testing hypotheses about the regression coefficients when the sample size is relatively small.

â CLRM.7 is not required for large samples, because asymptotically normal distribution of

β ˆ

is provided by the Central Limit Theorem anyway.