THE CENTRAL LIMIT THEOREM TORONTO

(1)

THE CENTRAL LIMIT THEOREM

DANIEL R ¨UDT

UNIVERSITY OF

TORONTO

MARCH, 2010

(2)

1 Introduction

The Central Limit Theorem (CLT) is one of the most remarkable results in probability theory because it’s not only very easy to phrase, but also has very useful applications.

The CLT can tell us about the distribution of large sums of random variables even if the distribution of the random variables is almost unknown. With this result we are able to approximate how likely it is that the arithmetic mean deviates from its expected value. I give such an example in chapter 4.

The CLT provides answers to many statistical problems. Using the CLT we can verify hypotheses by making statistical decisions, because we are able to determine the asymptotic distribution of certain test statistics.

As a warm up, we attempt to understand what happens to the distribution of random variables if we sum them.

Suppose X and Y are continuous and independent random variables with densities fX and fY. If we define 11A(x) to be the indicator function of a set A, then we want to recall that for independent random variables

P (X ∈ A, Y ∈ B) = Z

R

Z

R

11_A(x)11_B(y)f_X(x)f_Y(y) dx dy .

Now we can see that the density of X + Y is given by the convolution of f_X and f_Y. P (X + Y ≤ z) =

Z

R

Z

R

11_[x+y≤z](x, y)fX(x)fY(y) dx dy = Z

R

Z z−y

−∞

fX(x) dx

fY(y) dx dy

= Z

R

Z z

−∞

fX(x − y) dx

fY(y) dx dy^{F ubini}= Z z

−∞

Z

R

fX(x − y)fY(y) dy dx

= Z z

−∞

f_X∗ f_Y(x) dx .

In order to visualize this result I did some calculations. I determined the density of the sum of independent, uniformly distributed random variables. The following pictures will show the density of Pn

i=1Xi for Xi∼ U [−0.5, 0.5].

(4)

n = 1 n = 2

n = 3 n = 10

If we compare these graphs to the density of a standard normally distributed random variable, we can see remarkable similarities even for small n.

Density of a standard normally distributed random variable

This result leads us to suspect that sums of random variables somehow behave normally.

The CLT frames this fact.

(5)

2 Mathematical Background

In this section I just want to recall some of the basic definitions and theorems used in this paper. Elementary definitions of probability theory are assumed to be well known. To keep things simple, we just consider the sample space Ω = R.

Definition. A random variable X is called normally distributed with parameters µ and σ (X ∼ N (µ, σ)) if the density of the random variable is given by

φ(x) = 1

√ 2πσ² e

−(x−µ)2 2σ2 .

Definition. If a random variable X with probability measure P_X has density f , then we define the distribution function FX as follows

F_X(x) = P (X ≤ x) = P_X((−∞, x]) = Z x

−∞

dP_X(x) = Z x

−∞

f (x) dx

Definition. A sequence of random variables X₁, X₂... is converging in distribution to a random variable X if

n→∞lim F_X_n(x) = F_X(x)

for every x ∈ R at which FX is continuous. We can write this as X_n⇒ X.^d

The following theorem is also used to define convergence in distribution and will become handy in proving the Central Limit Theorem:

Theorem 2.1. Suppose X₁, X₂... is a sequence of random variables, then X_n⇒ X if and^d only if

n→∞lim E[f (X_n)] = E[f (X)]

for every bounded and continuous function f.

Definition. The characteristic function of a random variable X is defined by

ϕ(t) = Ee^itX = Z

R

e^itxdPX(x) .

(6)

Proposition. Every characteristic function ϕ is continuous, ϕ(0) = 1 and |ϕ(t)| ≤ 1.

Theorem 2.2. If X,Y are random variables and ϕ_X(t) = ϕ_Y(t) for all t ∈ R, then

X= Y^d i.e. X and Y have the same distribution.

Theorem 2.3. Suppose X and Y are independent random variables, then

ϕ_X+Y(t) = ϕ_X(t) · ϕ_Y(t) ∀t ∈ R .

3 The Central Limit Theorem

The Central Limit Theorem. If {X_n} is a sequence of identically and independent distributed random variables each having finite expectation µ and finite positive variance σ², then:

X1+ X2+ ... + Xn− n · µ σ ·√

n

⇒ N (0, 1)d

i.e. a centered and normalized sum of independent and identically distributed (i.i.d.) random variables is distributed standard normally as n goes to infinity.

4 Examples

4.1 Roulette

It’s nothing new that on average you should lose when playing roulette. Despite this it’s still interesting to examine the chances of winning. The CLT gives an answer to this question:

(7)

A roulette wheel has 37 numbers in total. 18 are black, 18 are red and 1 is green. Players are allowed to bet on black or red. Assume a player is always betting $1 on black. We define X_i to be the winnings of the ith spin. X₁, X₂, ... are clearly independent and

P (X_i= 1) = 18

37, P (X_i = −1) = 19 37 E(Xi) = − 1

37, V ar(Xi) = E(Xi2) − [E(Xi)]²≈ 1 .

We want to approximate the probability that Sn= X1+ ... + Xn is bigger than 0 P (Sn> 0) = P S_n− nµ

√nσ > −nµ

√nσ

.

Let’s say we want to play n = 100 times, then

√−nµ

nσ = 100 · (1/37)

√100 = 10 37. Now the CLT states that

P (S_n> 0) ≈ P (X > 10 37)

for a standard normally distributed random variable X. Since P (X > ¹⁰₃₇) ≈ 0.39 we can assume that the chance to win money by playing roulette 100 times is about 39%.

4.2 Cauchy Distribution

The Cauchy distribution shows that the conditions of finite variance and finite expectation can not be dropped.

Definition. A random variable X is called Cauchy distributed if the density of X is given by

f (x) = 1 π(1 + x²).

Proposition. If X is a Cauchy distributed random variable, then E[X] = ∞ and V ar[X] = ∞ .

(8)

Lemma. If X is Cauchy distributed, then

ϕ_X(t) = e^−|t|.

Proposition. If {Xn} is a sequence of independent Cauchy distributed random variables, then

Y_n= _n¹Pn

i=1X_i has Cauchy distribution.

Proof. To prove this statement we want to compute the characteristic function of Yn and compare it to the characteristic function of a Cauchy distributed random variable. If they are the same, then the claim follows by Theorem 2.2.

ϕ_Y_n(t) =

n

Y

i=1

ϕ_Xi

n

(t) =

n

Y

i=1

ϕ_X_i t n

=

ϕ_X₁ t n

n

= e^|t|ⁿn

= e^−|t|

The first step is true because of the Theorem about sums of independent random variables and its characteristic functions (Theorem 2.3). The third step follows from Theorem 2.2 since all random variables are identically distributed.

So as we can see the arithmetic mean of Cauchy distributed random variables is always Cauchy distributed and therefore the CLT doesn’t hold.

5 Historical Background

The CLT has a long and vivid history. It developed over time and there are many different versions and proofs of the CLT.

1st Chapter (1776 - 1829)

In 1776 Laplace published a paper about the inclining angles of meteors. In this paper he tried to calculate the probability that the actual data collected differed from the theoretical mean he calculated. This was the first attempt to study summed random variables. From this it is clear that the CLT was motivated by statistics. His work was continued by Poisson,

(9)

who published two papers in 1824 and 1829. In these papers he tried to generalize the work of Laplace, and also make it more rigorous. At this time probability theory still was not considered a branch of real mathematics. For most mathematicians it was sufficient that the CLT worked in practice, so they didn’t make a lot of effort to give real proofs.

2nd Chapter (1870 - 1913)

This mindset changed during the 19th century. Bessel, Dirichlet and especially Cauchy turned probability theory into a respected branch of pure mathematics. They succeeded in giving rigorous proofs, but there were still some issues. They had problems in dealing with distributions with infinite support and quantifying the rate of convergence. Moreover the conditions for the CLT were not satisfying.

Between 1870 and 1913 the famous Russian mathematicians Markov, Chebyshev and Lya- punov did a lot of research on the CLT and are considered to be the most important con- tributers to the CLT. To prove the CLT they worked in two different directions: Markov and Chebyshev attempted to prove the CLT using the ”Method of Moments”, whereas Lyapunov was using characteristic functions.

3rd Chapter (1920 - 1937)

During that period Lindeberg, Feller and L´evy studied the CLT. Lindeberg was able to apply the CLT to random vectors and he quantified the rate of convergence. His proof was a big step since he was able to give all sufficient conditions of the CLT. Later, Feller and L´evy also succeeded in giving also all necessary conditions, which could be proven by the work of Cramer. The CLT, as it is known today, was born.

The CLT today

People have continued to improve the CLT. There has been various research on theorems related to dependent random variables, but the basic principles of Lindeberg, Feller and L´evy are still up to date.

(10)

6 Proof

6.1 Outline of the Proof

The idea of the proof is to use nice properties of characteristic functions. The L´evy continuity theorem states that the distribution of the limit of random variables is uniquely defined by the limit of the corresponding characteristic functions. So all we have to do is understand the limit of the characteristic functions of our summed random variables. We will see that the characteristic function of summed i.i.d. random variables behave very well.

The first step is to understand the L´evy continuity theorem. The second step will deal with the evaluation of the characteristic function of summed i.i.d. random variables. The final step will put all things together in a very nice and smooth proof.

6.2 L´evy Continuity Theorem

The actual proof of the CLT is straight forward. The difficulty is to understand all the con- tributing theorems and lemmas. Since the most important theorem is the L´evy continuity theorem, I want to have a close look at this result.

L´evy Continuity Theorem. Suppose X₁, X₂, ...X_n, X are random variables and ϕ(t)1, ..., ϕ(t)n, ϕX(t) the corresponding sequence of characteristic functions, then

Xn⇒ X ⇐⇒ lim

n→∞ϕn(t) = ϕX(t) ∀t ∈ R . To understand how the proof works we need some more tools:

Bounded Convergence Theorem. If X, X1, X2, ... are random variables, Xn d

⇒ X, C ∈ R and |Xn| ≤ C for all n ∈ N, then

n→∞lim E[Xn] = E[X] .

Definition. A sequence of random variables Xn is defined as tight if ∀ > 0 there exists a M ∈ R s.t. P (|Xn| > M ) ≤ for all n ∈ N.

Lemma 6.2.1. If Xnis tight, then there exists a random variable X s.t.

X_n⇒ X .^d

(11)

Lemma 6.2.2. If Xnis tight and if each subsequence of Xnthat converges at all converges to a random variable Y , then also

Xn d

⇒ Y Proof of the L´evy Continuity Theorem

“⇒“:

Since cos(tXn) and sin(tXn) are continuous and bounded functions we can see that by Theorem 2.1

ϕn(t) = E[e^itX] = E[cos(tXn)] + iE[sin(tXn)]^n→∞−→ E[cos(tX)] + iE[sin(tX)] = ϕ_n(t)

“⇐“:

This part of the proof proceeds in two steps. First we want to show that pointwise convergence of characteristic functions implies tightness. After this, we are able to use nice properties of tight random variables to prove the claim.

We will show tightness by estimating the following term which will turn out to be a nice upper bound for the probability that |X_n| is big. For arbitrary δ > 0

δ⁻¹ Z _δ

−δ

(1 − ϕ_n(t)) dt = δ⁻¹ Z _δ

−δ

1 − E[e^itXⁿ] dt = δ⁻¹ Z _δ

−δ

E[1 − e^itXⁿ] dt

= δ⁻¹ Z δ

−δ

Z

R

1 − e^itxdPn(x)

dt

F ubini

= δ⁻¹ Z

R

Z δ

−δ

1 − e^itxdt

dPn(x)

= δ⁻¹ Z

R

2δ −

Z δ

−δ

cos(tx) + i sin(tx) dt

dPn(x)

= δ⁻¹ Z

R

2δ −

Z δ

−δ

cos(tx) dt

dPn(x)

= δ⁻¹ Z

R

2δ − 2 sin(δx)

x dP_n(x)

= 2 Z

R

1 −sin(δx)

δx dP_n(x).

Now that this term has a nice shape we want to find a lower bound for it. We know that 1 − ^sin(ux)_ux ≥ 0. This is true because: | sin x| = |Rx

0 cos(y)dy| ≤ |x|. So the integral gets smaller if we discard an interval

(12)

2 Z

R

1 −sin(δx)

ux dP_n(x) ≥ 2 Z

|x|≥δ/2

1 −sin(δx)

δx dP_n(x) ≥ 2 Z

|x|≥δ/2

1 − 1

|δx|

| {z }

≥1/2

dP_n(x)

≥ Z

|x|≥δ/2

dP_n(x) = P_n({x : |x| ≥ δ/2})

= P (X_n≥ |δ/2|).

Pick > 0. Because ϕ is continuous in 0 and ϕ(0) = 1 we can find a δ > 0 s.t.

|1 − ϕ(t)| ≤

4 ∀|t| ≤ δ . We can use this to estimate the following term

δ⁻¹ Z δ

−δ

1 − ϕ(t) dt

≤ δ⁻¹2δ 4 =

2. Since |ϕ_n(t)| ≤ 1 the bounded convergence theorem implies

Z δ

−δ

(1 − ϕn(t)) dt^n→∞−→

Z δ

−δ

(1 − ϕ(t)) dt . Because of that there exists a N ∈ R s.t. for all n > N

δ⁻¹ Z δ

−δ

(1 − ϕn(t)) dt − δ⁻¹ Z δ

−δ

(1 − ϕ(t)) dt

≤ 2. If we put the three bounds together, we get for all n > N

P (X_n≥ |δ/2|) ≤ δ⁻¹ Z δ

−δ

1 − ϕ_n(t) dt = δ⁻¹ Z δ

−δ

(1 − ϕ_n(t)) + (1 − ϕ(t)) − (1 − ϕ(t)) dt

≤

δ⁻¹ Z _δ

−δ

1 − ϕ_n(t) dt − δ⁻¹ Z _δ

−δ

(1 − ϕ(t)) dt

+

δ⁻¹ Z _δ

−δ

(1 − ϕ_n(t)) dt

≤ .

We just proved that the point δ/2 has the property that the probability of |Xn| exceeding this value is small for infinite many n. To show tightness we just need to find a bound for all the finite cases. Because P_n([−m, m])^m→∞−→ 1 we know that for all n ∈ {1, ..., N } there exists a mn∈ R such that

(13)

P (|Xn| ≥ m_n) ≤ .

Now define M = max{m₁, ..., m_N, δ/2}. Because of the monotonicity of distribution functions we just proved that P (|X_n| > M ) ≤ for all n ∈ N ⇒ Xn is tight.

Lemma 6.2.1 tells us that Xnhas a limit. Because of Lemma 6.2.2 we just need to show that every converging subsequence converges to X. So Suppose X_n_k converges to some random variable Y in distribution. Then we know because of “⇐“ that Y has the characteristic function ϕ(t). Now Y = X because of Theorem 2.2. Since this holds for any converging^d subsequence we just have shown

X_n⇒ X .^d

6.3 Lemmas

To apply the L´evy Continuity Theorem to the characteristic function of summed i.i.d.

random variables we need two more lemmas.

Lemma 6.3.1. Suppose X is a random variable and E|X|² < ∞, then ϕ_X(t) can be written as the following Taylor expansion

ϕ_X(t) = 1 + itE[X] +t²

2EX² + o t² Recall o(t²) means that as t → 0 ^o(t_t2²⁾ → 0.

The Lemma can be proven by using the following estimate for the error term o(t²) <

min(^|X|_3!³,^2|X|₂ ²).

Lemma 6.3.2. Suppose c_n is a sequence of complex numbers and c_n^n→∞−→ c, then

n→∞lim

1 +cn

n

= e^c

(14)

6.4 Proof of the Central Limit Theorem

Without loss of generality, we can assume that µ = 0 and σ² = 1 because E hXn−µ

σ

i

= 0 and V ar

hXn−µ σ

i

= 1.

The L´evy Continuity Theorem tells us that it is sufficient to show that the characteristic function of our sum converges to the characteristic function of a standard normally distributed random variable, i.e.

ϕSn√ n

(t) = ϕSn

√t n

→ e^−t²^/2 as n → ∞ and Sn= X1+ X2+ ... + Xn.

Now we want to use independence by applying Theorem 2.3 to the sum. Fix t ∈ R

ϕ_S_n

t

√n

=

n

Y

k=1

ϕ_X_k

t

√n

=

ϕ_X₁

t

√n

n

.

The second equality is true because all random variables are identically distributed and therefore have the same characteristic function (Theorem 2.2). Lemma 6.3.1 and the basic fact that V ar[Y ] = EY² − E[Y ]² for any random variable Y yields

ϕ_X₁

t

√n

n

=





1 + i t

√nE[X1]

| {z }

=0

+t²

nEX₁²

| {z }

=1

+o t² n







n

=

1 + t²

2n+ o 1 n

n

= 1 +t²/2 + n · o _n¹ n

!n

.

By using Lemma 6.3.2 and because o n⁻¹ /n⁻¹ → 0 as n → ∞ we just identified the limit

n→∞lim ϕ_S_n

t

√n

= e^−t²^/2.

(15)

7 References

P. Billingsley (1986) Probability and Measure. John Wiley and Sons, New York R. Durrett (2010) Probability: Theory and Examples

M. Mether (2003) The History Of The Central Limit Theorem

THE CENTRAL LIMIT THEOREM TORONTO