Exercises for Stochastic Calculus
Abdelkader BENHARI
Probability measure, Review of probability theory, Markov chains, Recurrence transition matrices, Stationary distributions, Hitting times, Poisson processes, Renewal theory, Branching processes, Branching and point processes, Martingales in discrete time, Brownian motion, Martingales in continuous time
Show that if A and B belongs to the σ-algebra F then also B\A ∈ F (for de nition of σ-algebra, see De nition 1.3). Also show that F is closed under countable intersections, i.e. if Ai∈ Ffor i = 1, 2, . . . , then ∩∞i=1Ai∈ F.
Proof. 1) B\A = B ∩ Ac = (Bc ∪ A)c and since Bc ∈ F, Bc ∪ A ∈ F so
(Bc∪ A)c∈ F.
2) Take Ai ∈ F for i = 1, 2, . . . . Since ∪∞i=1A
c ∈ F it follows that
(∪∞i=1Ac)c= ∩∞i=1A ∈ F 2.
Though a fair die once. Assume that we only can observe if the number obtained is small, A = {1, 2, 3} and if the number is odd, B = {1, 3, 5}. Describe the resulting probability space; in particular, describe the σ-algebra F generated by A and B in terms of a suitable partition (for de nition of a partition, see De nition 1.9) of the sample space.
Proof. Looking at the Venn diagram insert diagram, we conclude that there are at most four partitions of the space, A∩B = {1, 3}, A∩Bc= {2}, Ac∩ B =
{5} and (A ∪ B)c = {4, 6} of which none is an empty sets. These partitions
can be combined in 24= 16dierent ways to generate the σ-algebra F de ned
below.
F ={∅, Ω, A, B, Ac, Bc, A ∪ B, A ∪ Bc, Ac∪ B, Ac∪ Bc,
A ∩ B, A ∩ Bc, Ac∩ B, Ac∩ Bc, (A ∩ Bc) ∪ (Ac∩ B), (A ∩ B) ∪ (Ac∩ Bc)}
={∅, Ω, {1, 2, 3}, {1, 3, 5}, {4, 5, 6}, {2, 4, 6}, {1, 2, 3, 5}, {1, 2, 3, 4, 6}, {1, 3, 4, 5, 6}, {2, 4, 5, 6}, {1, 3}, {2}, {5}, {4, 6}, {2, 5}, {1, 3, 4, 6}}.
The probability measure on each of the sets in F may be deduced by using the additivity of the probability measure and the probability measure of each of the partitions, P (A ∩ B) = 2/6, P (A ∩ Bc) = 1/6, P (Ac∩ B) = 1/6 and
P ((A ∪ B)c) = 2/6. 3.
Given a probability space (Ω, F, P) and functions X : Ω → R, Y : Ω → R. De ne Z = max{X, Y }.
1. Show that Z is F-measurable if both X and Y are F-measurable. 2. Find a special case where Z is F measurable even though neither X nor
Y is. Proof. 1)
{ω ∈ Ω : Z(ω) ≤ x} = {ω ∈ Ω : max{X(ω), Y (ω)} ≤ x} 1.XERCISEeX
where the last conclusion is based on the result of proposition 3.2.
2) Suppose you threw two coins and our σ-algebra F was generated by the
single set A = {HT, T H}, i.e. F = {∅, Ω, A, Ac} = {∅, {HH, T T, HT, T H}, {HT, T H}, {HH, T T }}.
Let X =n 1 if ω ∈ {HT } 0 if ω ∈ {HH, T T, T H} Y = n 1 if ω ∈ {T H} 0 if ω ∈ {HH, T T, HT } . Then Z =n 1 if ω ∈ {HT, T H} 0 if ω ∈ {HH, T T }, hence X and Y are not F-measurable while Z is. 4.
Toss a fair soin n = 4 times. describe the sample space Ω. we want to consider functions X : ω → {−1, 1}.
1. describe the probability space (Ω, F1, P1)thar arises if we want each
out-come to be a legitimate event. How many F1-measurable functions X
with E[X] = 0 are there?
2. Now describe the probability space (Ω, F2, P2)that arises if we want that
only combinations of sets of the type
Ai= {ω ∈ Ω : Number of heads = i}, i = {0, 1, 2, 3, 4}
should be events. How many F2-measurable functions X with E[X] = 0
are there?
3. Solve 2) above for some other n.
Proof. 1) Each coin that is tossed has to possible outcomes, and since there are four coins to be tossed, there are 24= 16 possible outcomes, or partitions.
So the sample space is Ω = {HHHH, HHHT, HHT H, etc.} with 16 members that describes all the information from that four tosses. F1 is the power set
of Ω consisting of all combinations of sets of Ω (don't forget that ∅ allways is included in a σ-algebra ), hence F1= σ(Ω). P1is deduced by rst observing that
each ω ∈ Ω has P(ω) = 1/16 and then using the additivity och the probability measure.
For any function X : Ω → {−1, 1} that is measurable with respect to F1
there is a set A ∈ F1 such that
X(ω) =n 1 if ω ∈ A −1 if ω ∈ Ac.
Hence E[X] = P1(A) − P1(Ac) so if E[X] = 0 we must have P1(A) = P1(Ac),
and since there are 16 partitions each with probability measure 1/16, there are
16
8 ways of combining the partitions so that there are equally many of them in A and Ac, hence there are 168 possible nctions X : Ω → {−1, 1} such that
2) The sets A0= {T T T T } A1= {T T T H, T T HT, T HT T, HT T T } A2= {T T HH, T HHT, HHT T, HT T H, T HT H, HT HT } A3= {HHHT, HHT H, HT HH, T HHH} A4= {HHHH},
de nes a partition of Ω. The σ-algebra F2 is the σ-algebra generated by this
partition. De ne Pi= P(Ai), then P0= 1/16, P1= 4/16, P2= 6/16, P3= 4/16
and P4 = 1/16. For any function X : Ω → {−1, 1} that is measurable with
respect to F2 there is a a set A ∈ F2such that
X(ω) =n 1 if ω ∈ A −1 if ω ∈ Ac.
Hence E[X] = P2(A) − P2(Ac)so if E[X] = 0 we must have P1(A) = P1(Ac).
We may write E[X] = k0P0+ k1P1+ k2P2+ k3P3+ k4P4= k0/160+ 4k1/16 +
6k2/16 + 4k3/16 + k4/16 which is zero either if ki = (−1)i or if ki = −(−1)i.
Hence, there are two possible F2-measurable functions X : Ω → {−1, 1} such
that E[X] = 0.
3) Using the same notation as in 2), we conclude that, for any given in-teger n > 0, A0, A1, . . . , An is a partition of Ω with σ-algebra Fn generated
by this partition, and with Pn(Ai) = n
i (1/2)i for i = 0, 1, . . . , n. And E[X] =Pni=0ki
n
i(1/2)i for ki= ±1, where it should be noted that P n i=0
n
i (1/2)i= 1. For any function X : Ω → {−1, 1} that is measurable with respect
to Fn there is a a set A ∈ Fn such that
X(ω) =n 1 if ω ∈ A −1 if ω ∈ Ac.
Hence E[X] = Pn(A) − Pn(Ac)so if E[X] = 0 we must have Pn(A) = Pn(Ac) =
1/2.
For even n, we have
X i=odd n i(1/2)i= 1/2 X i=even n i(1/2)i= 1/2
so E[X] = 0 if ki= (−1)i or ki= −(−1)i. For odd n, we have (n−1)/2 X i=0 n i(1/2)i= 1/2 n X i=(n−1)/2+1 n i(1/2)i= 1/2
Consider the probability space (Ω, F, P) where Ω = [0, 1], F is the Borel σ-algebra on Ω and P is the uniform probability measure on (Ω, F). Show that the two random variables X(ω) = ω and Y (ω) = 2|ω − 1/2| have the same distribution, but that P(X = Y ) = 0.
Proof. (For de nition of distribution function see de nition 3.4) Since X is uniform, the distribution function is
P(X ≤ x) = 0 for x ∈ (−∞, 0] P(X ≤ x) = x for x ∈ [0, 1] P(X ≤ x) = 1 for x ∈ [1, ∞) so for Y we get, with x ∈ [0, 1],
P(Y ≤ x) = P(2|X − 1/2| ≤ x) = P(|2X − 1| ≤ x) = {by symmetry} = 2P(0 ≤ 2X − 1 ≤ x) = 2P(1/2 ≤ X ≤ (x + 1)/2) = 2((x + 1)/2 − 1/2) = x.
Hence P(X ≤ x) = P(Y ≤ x) = x for x ∈ [0, 1]. But since P(X = Y ) = P(X = 2|X − 1/2|)
= P({ω ∈ Ω : X = 2X − 1/2} ∪ {ω ∈ Ω : X = −2X + 1/2}) = P({ω ∈ Ω : X = 1} ∪ {ω ∈ Ω : X = 3}) = P(∅) = 0, we conclude that P (X = Y ) = 0.
6.
Show that the smallest σ-algebra containing a set A is that intersection of all σ-algebras containing A. Also, show by counter example that the union of two σ-algebras is not necessarily a σ-algebra .
Proof. Let GA be the set of all σ-algebras containing A, we want to show that
∩G∈GAG = {ω ∈ Ω : ω ∈ G for every G ∈ GA} = F is a σ-algebra ; if it is, then it is the smallest σ-algebra since otherwise, there would be a σ-algebra containing A that was smaller than F , this would be a contradiction since then this σ-algebra would be included in G and therefore F would not be minimal. To check that F is a σ-algebra , we simply use De nition 1.3.
1. ∅ ∈ G ∀G ∈ GA hence ∅ ∈ F.
2. If ω ∈ G ∀G ∈ GA then ωc∈ G ∀G ∈ GA hence ωc∈ F.
3. If ω1, ω2, . . . ∈ G ∀G ∈ GAthen ∪∞i=1ωi∈ G ∀G ∈ GA hence ω1, ω2, . . . ∈ F.
so F is a σ-algebra .
To show that the union of two σ-algebras is not necessarily a σ-algebra , take F1= {∅, Ω, A, Ac}and F2= {∅, Ω, B, Bc}where A ⊂ B. Then F1∪ F2=
{ω ∈ Ω : ω ∈ F1 or F2} = {∅, Ω, A, Ac, B, Bc} is not a σ-algebra since, e.g.
Bc∪ A /∈ F1∪ F2.
given a probability space (Ω, F,P) and a random variable X. Let A be a sub-σ-algebra of F, and consider ˆX = E[X|A].
1. Show that E[(X − ˆX)Y ] = 0if Y is an A-measurable random variable. 2. Show that the A-measurable random variable Y that minimize E[(X−Y )2]
is Y = E[X|A].
Proof. 1) Using equality 4 followed by equality 2 in Proposition 4.8 we get E[(X −X)Y ] = E[E[(X − ˆˆ X)Y |A]] = E[E[(X − ˆX)|A]Y ]
= E[( ˆX − ˆX)Y ] = 0.
2) Using equality 3 followed by equality 2 in Proposition 4.8 we get E[(X − Y )2] = E[X2] − 2E[XY ] + E[Y2]
= E[X2] − 2E[E[XY |A]] + E[Y2] = E[X2] − 2E[Y E[X|A]] + E[Y2]
= E[X2] − E[E[X|A]2] + E[E[X|A]2] − 2E[Y E[X|A]] + E[Y2] = E[X2] − E[E[X|A]2] + E[(Y − E[X|A])2]
| {z }
≥0
,
hence E[(X − Y )2]is minimized when Y = E[X|A].
8.
Let X and Y be two integrable random variables de ned on the same probability space (Ω, F, P). Let A be a sub-σ-algebra such that X is A-measurable.
1. Show that E[Y |A] = X implies that E[Y |X] = X.
2. Show by counter example that E[Y |X] = X does not necessarily imply that E[Y |A] = X.
Proof. (For de nition of conditional expectation see De nition 4.6, for proper-ties of the conditional expectation, see Proposition 4.8.)
1) Since X is A-measurable, σ(X) ⊆ A hence, using equality 4 in Proposition 4.8, we get E[Y |X]= E[Y | σ(X)∆ | {z } ⊆A ] = E[E[Y |A] | {z } =X |X] = E[X|X] = X.
2) Let X and Z be independent and integrable random variables and assume that E[Z] = 0 and de ne Y = X + Z. Let A = σ(X, Z), then
E[Y |σ(X)] = E[X + Z|σ(X)] = X + 0 = X, while
Series 2 : review of probability theory
Exercise 1
For each given p let X have a binomial distribution with parameters p and N. Suppose N is itself binomially distributed with parameters q and M, M > N. (a) Show analytically that X has a binomial distribution with parameters pq and M. (b) Give a probabilistic argument for this result.
Exercise 2
Using the central limit theorem for suitable Poisson random variables, prove that lim n→∞e −n n X k=0 nk k! = 1 2. Exercise 3
Let X and Y be independent, identically distributed, positive random variables
with continuous density function f (x) satisfying f (x) > 0 for x > 0. Assume, further, that U = X − Y and V = min(X, Y) are independent random variables. Prove that
f (x) = (
λe−λx for x > 0;
0 elsewhere,
for some λ > 0.
Hint: Show first that the joint density function of U and V is fU,V(u, v) = f (v) f (v + |u|).
Next, equate this with the product of the marginal densities for U, V. Exercise 4
Let X be a nonnegative integer-valued random variable with probability generating function f (s) =P∞
n=0ansn. After observing X, then conduct X binomial trials with probability
p of success. Let Y denote the resulting number of successes. (a) Determine the probability generating function of Y.
(b) Determine the probability generating function of X given that Y = X.
(c) Suppose that for every p (0 < p < 1) the probability generating functions of (a) and (b) coincide. Prove that the distribution of X is Poisson, i.e., f (s) = eλ(s−1)for some λ > 0.
Series 2: review of probability theory Solutions
Exercise 1
(a) Since the distribution of N is binomial(M, q), we have
fN(s) = E[sN] = (sq + 1 − q)M.
Since the conditional distribution of X given N = n is Binomial(n, p), we have fX|N=n(s) = E[sX|N = n] = (sp + 1 − p)n.
We can compute the generating function of X, say fX(s), by conditioning on N. We obtain
fX(s) = E[sX] = M X n=0 E[sX | N = n] P[N = n] = M X n=0 (sp + 1 − p)nP[N = n] = fN(sp + 1 − p) = (sp + 1 − p)q + 1 − qM = s(pq) + 1 − pqM . Here we recognize the generating function of the binomial(M, pq) distribution. Here’s another approach. For k ∈ {0, 1, 2, ..., M}, we get
P[X = k] = M X n=0 P[X = k| N = n] P[N = n] = M X n=k n k ! pk(1 − p)n−k M n ! qn(1 − q)M−n = M! k!(M − k)!(pq) k M X n=k (M − k)! (n − k)!(M − n)!((1 − p)q) n−k(1 − q)M−k = M k ! (pq)k M−kX j=0 M − k j ! ((1 − p)q)j(1 − q)M−k− j = M k ! (pq)k((1 − p)q + (1 − q))M−k = M k ! (pq)k(1 − pq)M−k. This is the binomial(M, pq) distribution.
(b) Imagine M kids. Each one of them will toss a silver coin and a gold coin. With a silver coin the probability of heads is q. With a gold coin, it is p. Let N be the number of kids who will get heads with the silver coin and let X be the number of kids who will get heads on both tosses. Clearly the
Exercise 2
Let X1, X2, X3, ... be independent random variables, each with distribution Poisson(1), and let Sn =
Pn
k=1Xk. Since the mean and the variance of the Poisson(1) distribution are both equal to 1, the
cen-tral limit theorem gives us
lim n→∞ P" Sn√− n n ≤ x # = Z x −∞ 1 √ 2π e −u2/2 du for every x ∈ R. With x = 0, this gives us
lim n→∞ P" Sn√− n n ≤ 0 # = 1 2. This last equation can be written as
lim
n→∞
P[Sn ≤ n] = 1 2.
In view of the lemma below, the distribution of Sn is Poisson(n). Therefore, the last equation can be
written as lim n→∞e −n n X k=0 nk k! = 1 2.
Lemma. If U ∼ Poisson(α) and V ∼ Poisson(β) and if U and V are independent, then U + V ∼ Poisson(α + β).
The elementary proof is left as an exercise. Exercise 3
Let fX,Y(x, y) denote the joint density of X and Y. Let fU,V(u, v) denote the joint density of U and V. For
the moment we do not make any independence assumptions.
Fix u > 0 and v > 0. The following computation will be easy to follow if you look at the first quadrant of R2and visualize the set of points (x, y) for which min(x, y) ≤ v and x − y ≤ u.
fU,V(u, v) = ∂2 ∂u∂vP[(U ≤ u) ∩ (V ≤ v)] = ∂ 2 ∂u∂vP[(X − Y ≤ u) ∩ (min(X, Y) ≤ v)] = ∂ 2 ∂u∂v (P[(X, Y) ∈ A] + P[(X, Y) ∈ B])
where A = {(x, y) ∈ R2: v < y < ∞ and 0 < x < v} and B = {(x, y) ∈ R2: 0 < y ≤ v and 0 < x ≤ u + y}. Since A depends only on v, we have
∂2 ∂u∂vP[(X, Y) ∈ A] = 0. Thus we have fU,V(u, v) = ∂ 2 ∂u∂vP[(X, Y) ∈ B] = ∂ 2 ∂u∂v Z v 0 Z u+y 0 fX,Y(x, y) dx dy = ∂ ∂u Z u+v 0 fX,Y(x, v) dx = fX,Y(u + v, v).
A similar calculation gives us fU,V(u, v) = fX,Y(v − u, v) for the case where u > 0 and v < 0. (You should
do the computation; be careful with the domain of integration in the (x, y) plane). In summary, we have shown that
(1) fU,V(u, v) =
(
fX,Y(v + |u|, v) if u ∈ R and v > 0,
0 otherwise.
If we assume that X and Y are independent and identically distributed with density f , then equation (1) can be written as
fU,V(u, v) =
(
f (v + |u|) f (v) if u ∈ R and v > 0,
0 otherwise.
If we assume furthermore that U and V are independent, i.e., fU,V(u, v) = fU(u) fV(v) for all v > 0 and
u ∈ R, then the above result gives us
f (v) f (v + |u|) = fU(u) fV(v) v > 0, − ∞ < u < ∞.
In particular we have
(2) f (u + v) = g(u) h(v) u > 0, v > 0
with g(u) = fU(u) and h(v) = fV(v)/ f (v). From (2) we get, for all x > 0 and y > 0,
P[X > x + y] P[X > x] = R∞ y f (x + v)dv R∞ 0 f (x + v)dv = g(x)Ry∞h(v)dv g(x)R0∞h(v)dv = R∞ y h(v)dv R∞ 0 h(v)dv
Thus the ratio P[X > x + y]/P[X > x] does not depend on x. Taking the limit as x → 0, we see that this ratio is equal to P[X > y]. Thus we have
(3) P[X > x + y] = P[X > x] P[X > y] pour tout x > 0, y > 0. The lemma below allows us to conclude that X ∼ exponential(λ).
Lemma. Let X be a non negative random variable.
(a) If X ∼ exponential(λ) for some λ > 0, then equation (3) is true. (b) If equation (3) is true, then X ∼ exponential(λ) (for some λ > 0). The proof is left to the reader.
Exercise 4
(a) The distribution of Y given X = n is Binomial(n, p). Thus, E(sY|X = n) = (sp + 1 − p)n. Using this, we have
fY(s) = E(sY) = ∞ X n=0 E(sY|X = n) P(X = n) = ∞ X n=0 (sp + 1 − p)n· P(X = n) = f (sp + 1 − p). (b) We have ∞ X X∞
To conclude, fX|X=Y(s) = ∞ X n=0 snP(X = n | X = Y) = 1 f (p) ∞ X n=0 sn pn P(X = n) = f (sp) f (p). (c) From parts (a) and (b) we have
f (1 − p + ps) = f (ps)f (p) 0 < p < 1 and − 1 < s < 1. If we fix p and we take derivative w.r.t. s, we obtain
f′(1 − p + ps) p = f
′(ps) p
f (p) 0 < p < 1 and − 1 < s < 1. Now we divide by p on both sides and we evaluate at s = 1. We obtain
f′(1) = f ′(p) f (p) 0 < p < 1 i.e. (4) E[X] = f ′(p) f (p) 0 < p < 1
When we solve equation (4), subject to the boundary condition f (1) = 1, we get f (s) = eλ(s−1) (where λ = E[X]). We conclude that the distribution of X is Poisson.
Series 3: Markov chains
Exercise 1
Determine the classes and the periodicity of the various states for a Markov chain with transition probability matrix
(a) 0 0 1 0 1 0 0 0 1 2 1 2 0 0 1 3 1 3 1 3 0 , (b) 0 1 0 0 0 0 0 1 0 1 0 0 1 3 0 2 3 0 . Exercise 2
Consider repeated independent trials of two outcomes S (sucess)
or F (failure) with probabilities p and q, respectively. Determine the mean of the number of trials required for the first occurence of the event SF (i.e., sucess followed by failure). Do the same for the event SSF and SFS.
Exercise 3
Let a Markov chain contain r states. Prove the following:
(a) If a state k can be reached from j, then it can be reached in r − 1 steps or less.
(b) If j is a recurrent state, there exists α (0 < α < 1) such that for n > r the probability that the first return to state j occurs after n transitions is 6 αn.
Exercise 4
In this exercise, consider a random walk on the integers such that Pi,i+1= p, Pi,i−1= q for all integer i (0 < p < 1, p + q = 1).
(a) Determine Pn0,0.
(b) For any real a and natural n, define a n ! = a(a − 1) · · · (a − n + 1) n! . Prove that 2n n ! = (−1)n −1/2 n ! 22n.
(c) Let P be the generating function associated to the sequence un = Pn0,0, i.e., P(x) =P∞n=0unxn. Show
that P(x) = (1 − 4pqx2)−1/2.
(d) Prove that the generating function of the recurrence time from state 0 to state 0 is F(x) = 1 − p
1 − 4pqx2.
Exercise 5
A player has k coins. At each round independently, he gains one coin with probability p, or loses one coin with probability 1 − p = q. He stops to play if he reaches 0 (he loses) or N (he wins). What is the probability to win ?
Exercise 6
Suppose X1,X2, . . .are independent withP(Xk = +1) = p, P(Xk = −1) = q = 1 − p, where p
>q With S0 = 0, set Sn= X1+ · · · + Xn, Mn= max{Sk : 0 6 k 6 n} and Yn= Mn−Sn.
If T (a) = min{n : Sn= a}, show
P max 06k6T (a)Yk<y ! = y 1+y a if p = q = 1/2; "p q− p q y+1 1−pqy+1 #a if p , q
Hint: The bivariate process (Mn,Yn) is a random walk on the positive lattice. What is the probability that
Solutions 3 Solutions
Exercise 1
(a) We have 1 → 3, 3 → 2 and 2 → 1, so 1, 2 and 3 are in the same class. Also, for states i = 1, 2, 3 we have P(i, 4) = 0, so class {1, 2, 3} is recurrent. Since P(4, 4) = 0, 4 itself forms a transient class. The classes are thus {1, 2, 3}, {4}. Periodicity is only defined for recurrent states, so it suffices to determine the period of 1, 2 and 3. Since they are in the same class, they have the same period, so it suffices to determine the period of 1. We have P2(1, 1) > P(1, 3)P(3, 1) > 0 and P3(1, 1) = P(1, 3)P(3, 2)P(2, 1) > 0, so the period of 1 must divide 2 and 3, hence it is equal to 1.
(b) Since 1 → 2, 2 → 4, 4 → 3, 4 → 1 and 3 → 2, there is a single recurrent class {1, 2, 3, 4}. Let us determine the period of state 4. We have P(4, 4)P2(4, 4) = 0. Also, P3(4, 4) = 1, so P3n(4, 4) = 1 for all n. Fix N ∈ N, N > 3 and N not divisible by 3; we can then write N = 3n + k for some n ∈ N and k = 1 or 2. Then, PN(4, 4) = P3n+k(4, 4) = 4 X i=1 P3n(4, i)Pk(i, 4) = P3n(4, 4)Pk(4, 4) = 0.
We thus have PN(4, 4) > 0 if and only if N is a multiple of 3, so the period of 4 (and every other state) is 3.
Exercise 2
Let ξ1, ξ2, . . .be a sequence of letters chosen independently at random so that for each i, P(ξi= S ) = p =
1 − q = 1 − P(ξi = F).
• Time until SF. Let X0 = ∅∅, X1 = ∅ξ1and Xn = ξn−1ξn for n > 2, where in each case we are simply
concatenating the symbols. (Xn) is then a Markov chain on the state space {∅∅, ∅S , ∅S , S S , S F, FS , FF}.
Define, for n > 0, an= P∅∅(min{k : Xk = S F} = n), bn= P∅∅(Xn= S F), cn = PS F(Xn = S F); F∅∅,S F(s) = ∞ X n=0 ansn, P∅∅,S F(s) = ∞ X n=0 bnsn, PS F,S F(s) = ∞ X n=0 cnsn,
where s ∈ C with |s| < 1. We have
b0= b1 = 0, bn = pq ∀n > 2; c0= 1, c1= 0, cn= pq ∀n > 2, so P∅∅,S F(s) = ∞ X n=2 pqsn= pqs 2 1 − s, PS F,S F(s) = 1 + ∞ X n=2 pqsn= 1 + pqs 2 1 − s. Using the formula Fx,y(s)Py,y(s) = Px,y(s) (which applies when x , y, we have
pqs2
and F∅∅,S F′ (1) = 1/pq. We are now done:
E(min{n : Xn= S F}) = F′
∅∅,S F(1) = 1/pq.
• Time until SSF. Now put Y0= ∅∅∅, Y1= ∅∅ξ1, Y2= ∅ξ1ξ2and Yn= ξn−2ξn−1ξnfor n > 3. Define
an= P∅∅∅(min{k : Xk = S S F} = n), bn = P∅∅∅(Xn = S S F), cn= PS S F(Xn= S S F); F∅∅∅,S S F(s) = ∞ X n=0 ansn, P∅∅∅,S S F(s) = ∞ X n=0 bnsn, PS S F,S S F(s) = ∞ X n=0 cnsn. We have b0= b1 = b2= 0, bn = p2q ∀n > 3; c0= 1, c1= c2 = 0, cn = p2q ∀n > 3, so that P∅∅∅,S S F(s) = ∞ X n=3 p2qs3= p 2qs3 1 − s, PS S F,S S F(s) = 1 + ∞ X n=3 p2qs3= 1 + p 2qs3 1 − s and F∅∅∅,S S F(s) = p2qs3 1 − s + p2qs3. We thus have F′∅∅∅,S S F(s) = 3p 2qs2(1 − s + p2qs3) − p2qs3(−1 + 3p2qs2) (1 − s + p2qs3)2 and F∅∅∅,S S F′ (1) = 1/p2q = E(min{n : Xn= S S F}).
• Time until SFS. Similarly to before, put
an= P∅∅∅(min{k : Xk = S FS } = n), bn = P∅∅∅(Xn = S FS ), cn= PS FS(Xn= S FS ); F∅∅∅,S FS(s) = ∞ X n=0 ansn, P∅∅∅,S FS(s) = ∞ X n=0 bnsn, PS FS ,S FS(s) = ∞ X n=0 cnsn. We now have b0 = b1= b2= 0, bn= p2q ∀n > 3; c0 = 1, c1 = 0, c2 = pq, cn= p2q ∀n > 3, so that P∅∅∅,S FS(s) = ∞ X n=3 p2qs3 = p 2qs3 1 − s, PS FS ,S FS(s) = 1 + pqs 2+ ∞ X n=3 p2qs3= 1 + pqs2+ p 2qs3 1 − s and F∅∅∅,S FS(s) = p2qs3 1−s 1 + pqs2+ p2qs3 1−s = p 2qs3 1 − s + pqs2+ (p2q − pq)s3. Finally, F∅∅∅,S FS′ (s) = 3p 2qs2(1 − s + pqs2+ (p2q − pq)s3) − p2qs3(−1 + 2pqs + 3(p2q − pq)s2)) (1 − s + pqs2+ (p2q − pq)s3)2 ; F∅∅∅,S FS′ (1) = 3p 2q(pq + p2q − pq) − p2q(−1 + 2pq + 3p2q − 3pq) (pq + p2q − pq)2 = 1 p2q + 1 p, so E(min{n : Yn= S FS }) = p12q+ 1p.
Exercise 3
(a) We assume that k , j. The fact that k can be reached from j means that there exists a sequence of states x0,x1, . . . ,xN such that x0 = j, xN = k and P(xi,xi+1) > 0 for all i. Define
Γ = {i ∈ {1, . . . , N − 1} : ∃ j1∈ {1, . . . , i}, j2∈ {i + 1, . . . , N} : xj1 = xj2}
(the set of indices i such that xiis between two repeated entries in the sequence x1, . . . ,xN). By
enumerat-ing {0, 1, . . . , N}\Γ = {a0, . . . ,aM} with a0 = 0 < a1 < . . . <aM= N, it is easy to see that P(xai,xai+1) > 0 for each i and that xa0,xa1,xa2, . . .xaM has no repetitions. This implies that this sequence has at most r elements, so the path xa0 → xa1 → . . . → xaM−1 → xaM shows that k can be reached from j in at most r − 1 steps.
(b) Define τj = min{n > 1 : Xn = j}. Assume that j → k. Since j is recurrent, we must also have k → j,
and by part (a), j can be reached from k in at most r − 1 steps. This implies that Pk(τj 6r − 1) > 0
or, equivalently, that
Pk(τj >r) < 1. Since the state space is finite, we thus conclude that the number
θ= minnPk(τj >r) : k can be reached from jo
is smaller than 1. Now, fix n ∈ N. We have ⌊n/r⌋r − 1 < n, so the event {τj > n} is contained in
{τj> ⌊n/r⌋r − 1}. Using this, we have
Pj(τj >n) 6 Pj(τj> ⌊n/r⌋r − 1) = Pj( j < {X1, . . . ,X⌊n/r⌋r−1}) = X k: j→k Pj( j < {X1, . . . ,X(⌊n/r⌋−1)r−1}, X(⌊n/r⌋−1)r−1= k) · Pk(τj>r) 6 X k: j→k Pj( j < {X1, . . . ,X(⌊n/r⌋−1)r−1}, X(⌊n/r⌋−1)r−1= k) · θ = θ Pj(τj >(⌊n/r⌋ − 1)r − 1).
Proceeding inductively, we get
Pj(τj >n) 6 θ⌊n/r⌋, ∀ n > r We can now choose α ∈ (0, 1) such that, for any n > r,
θ⌊n/r⌋6αn, completing the proof.
Exercise 4
(a) Let ξ1, ξ2, . . . denote a sequence of independent and identically distributed random variables with
P(ξi = −1) = q, P(ξi = 1) = p for each i. Putting X0 ≡ 0 and Xn =Pn
i=1ξn, we see that Xnis a Markov
chain with transitions given by P. Since for any n > 1 we have Xn− Xn+1 ∈ {−1, 1}, Xn has the same
parity as n. As a consequence, Pn(0, 0) = 0 when n is odd. If n is even, then P(Xn= 0) = P n X i=1 ξi = 0 = X y1,...,yn∈{−1,1}n:P yi=0 P(ξ1 = y1, . . . , ξn= yn) = n n/2 ! pn/2qn/2. (b) −1/2! 1 −1! −3! −(2n − 1)!
Then, (−1)n −1/2 n ! 22n= (−1)2n· 1 n!· 22n 22n · (2n)! n! = 2n n ! (c) By part (a), P(x) = ∞ X n=0 u2nx2n= ∞ X n=0 2n n ! (pqx2)n. Using (b), we obtain P(x) = ∞ X n=0 −1/2 n ! (−4pqx2)n = (1 − 4pqx2)−1/2.
(d) Using the formula F(x)P(x) = P(x) − 1 and part (c), we immediately get F(x) = 1 − p1 − 4pqx2.
(e) Writing F(x) =P∞ k=0 fkxk, we have P(∃n : Xn= 0) = ∞ X k=0 P(min{n : Xn = 0} = k) = ∞ X k=0 fk = F(1) = 1 − p 1 − 4pq. Exercise 5
Let (Xn)n∈Nbe the Markov chain representing the fortune of the player, and let τ be the time when the
game ends, that is,
τ= inf{n ≥ 0 : Xn = 0 or Xn= N}.
Let ukbe the probability that starting from k coins, the player wins, that is,
uk = Pk[Xτ= N],
where Pk = P[ · | X0 = k]. For 0 < k < N, we have
Pk[Xτ= N] = Pk[X1= k − 1, Xτ= N] + Pk[X1= k + 1, Xτ= N]. Note that this is equal to
Pk[X1= k − 1] [Xτ= N | X1= k − 1] + Pk[X1= k + 1] Pk[Xτ= N | X1= k + 1]
= q Pk−1[Xτ= N] + p Pk+1[Xτ= N],
where we used the Markov property in the last step. We have proved that uk = quk−1+ puk+1.
Moreover, it is clear that u0= 0 and uN = 1. We solve this recurrence equation as in Karlin-Taylor, (3.6)
in chapter 3. For p = 1/2, we obtain
uk = k N, and otherwise, uk = 1 − (q/p)k 1 − (q/p)N. Exercise 6
From the fact that (Sn) is a Markov chain, it follows that ((Yn,Mn)) is also a Markov chain. To justify
this properly, fix a sequence (y0,m0) = (0, 0), (y1,m1), . . . , (yn,mn); noticing the equality
we have P((Y
n+1,Mn+1) = (y, m) | (Y0,M0) = (0, 0), . . . , (Yn,Mn) = (yn,mn))
= P (maxmax06i6n+1Si = m,
06i6n+1Si) − Sn+1 = y S0 = 0, S1 = m1− y1, . . . ,Sn = mn− yn ! = p if y = yn= 0, m = mn+ 1; or if y > 0, y = yn− 1, m = mn; q if y = yn+ 1, m = mn;
0 in any other case.
In words, (Yn,Mn) takes values in the positive quadrant of Z2 (see the picture in the statement of the
exercise) and moves according to the rules:
• if y > 0, we jump to the right with probability q and to the left with probability p; • if y = 0, we jump to the right with probability q and up with probability p. Notice that we never jump down and we can only jump up on the vertical axis. For a > 0, we have T (a) = min{n : Sn= a} = min{n : Mn= a}, so
( max
06k6T (a)Yk <y
)
=(Yn,Mn) reaches (a, 0) before reaching {y} ×Z+ ,
so, using the Strong Markov Property,
P max 06k6T (a)Yk <y ! = a−1 Y i=0 P (Yn,Mn) reaches (0, i + 1) before reaching (y, i)
(Y0,M0) = (0, i) !
= P before reaching (y, a − 1)(Yn,Mn) reaches (0, a)
(Y0,M0) = (0, a − 1) !a
The probability in the above expression is the probability that the chain in the picture reaches the upper point (0, a) before reaching the rightmost point (y, a − 1). From the gambler’s ruin problem, we see that this is equal to y 1+y if p = q = 1/2; p q− p q y+1 1−pqy+1 if p , q,
Series 4: recurrence, transition matrices
Exercise 1
Let P be the transition matrix of a Markov chain. If j is a transient state, prove that for all i,
+∞
X
n=1
Pni, j <+∞.
Exercise 2
Consider a sequence of Bernoulli trials X1,X2, . . ., where Xn= 1 or 0. Assume
that there exists α > 0 such that, for any n,
P[Xn= 1 | X1, . . . ,Xn−1] ≥ α.
Show that
(1) P[Xn= 1 for some n] = 1,
(2) P[Xn= 1 infinitely often] = 1. Exercise 3
Players I, with fortune a, and player II, with fortune b, play
together (a > 10, b > 10). At each round, player I has probability p to win one unit from player II, else he gives one unit to player II. What is the probability that player I will achieve a fortune a + b − 3 before his fortune dwindles to 5?
Exercise 4
A car can be in one of three states: working (state 1), being repared (state 2), or definitely out of order (state 3). If the car is working on some day, then the probability it will be working the day after is 0.995, and the probability it will need repair is 0.005. If it is being repaired, then the probability it will be working the day after is 0.9, the probability it will still need repair is 0.05, the probability it will be definitely out of order is 0.05.
(1) Write the transition matrix for this Markov chain. (2) Let us define the reduced matrix
R =" 0.995 0.0050.9 0.05 #
.
Show that Q = (I − R)−1is well defined, and that Qi j is the expected total time spent at state j starting
from state i (Hint: study the series P+∞
n=0Rn. To show that Q and this series are well defined, you may
study the spectrum of the transpose of R.)
(3) Starting from a car that is working, what is the expected time before it gets definitely out of order ? (4) How can the result be extended to more general Markov chains ?
Series 4: recurrence, transition matrices Solutions
Exercise 1
Let us write (Xn) for the Markov chain with transition matrix P, Pifor P[· | X0= i], and
τi= inf{n ≥ 1 : Xn= i}. Let
Fni j= Pi[τj = k],
and let Fi j(s) be its generating function, Fi j(s) = P+∞n=0Fi jnsn. Let also Pi j(s) = P+∞n=0Pni jsn. One can
show that
Pj j(s) =
1 1 − Fj j(s)
,
see Karlin-Taylor, Eq. (5.8) of Chapter 2 for a proof. By definition, if the state j is transient, then Pj[τj < +∞] < 1.
This amounts to saying that
+∞ X n=0 Fnj j <1, and thus lim s→1Pj j(s) < +∞.
We have thus proved that
+∞ X n=0 Pnj j< +∞. For i , j, we have Pi j(s) = Fi j(s)Pj j(s)
(see Karlin-Taylor, Eq. (5.10) of Chapter 2), and thus we also have lim s→1Pi j(s) < +∞, or equivalently, +∞ X n=0 Pni j< +∞. Exercise 2 (1) Note that
and as a consequence,
P[∀n ∈ {1, 2, . . .}, Xn= 0] = 0.
(2) If it is not true that Xn = 1 infinitely often, then there exists N such that for all n ≥ N, Xn = 0. We have P[∃N : ∀n ≥ N, Xn = 0] ≤ +∞ X N=0 P[∀n ≥ N, Xn= 0],
and we proceed as in (1) to show that each summand is equal to 0. Exercise 3
Let (Xn)n>0be a Markov chain on Z such that, for each n, the transitions n → n + 1 and n → n − 1 occur
with probability p and 1 − p, respectively. We want to find
P(Xnreaches a + b − 3 before reaching 5 | X0= a),
which is the same as
P(Xnreaches a + b − 8 before reaching 0 | X0= a − 5).
By Gambler’s ruin, this is equal to q p a−5 −1 q p a+b−8 −1 if p , q; a−5 a+b−8 if p = q = 1/2. Exercise 4
(1) The transition matrix is
P = 0.995 0.005 0 0.9 0.05 0.05 0 0 1 .
(2) Remark: for the purpose of this question, computing explicitely the eigenvalues of this 2 × 2 matrix and checking that their modulus are strictly smaller than 1 would be fine. Here, we study the spectrum using arguments that are easier to extend to more general situations.
Note that for any i, one has
X
j
Ri j≤1,
Hence, for any x = (x1,x2) ∈C
2, (1) kRTxk1= X j X i Ri jxi ≤X j X i Ri j|xi| ≤ kxk1.
As a consequence, the spectrum of RT (which is the same as that of R) is contained in {λ ∈C: |λ| ≤ 1}.
Moreover, since
X
j
R2 j <1,
the inequality in (1) is strict if x2 ,0.
Assume that there exists λ ∈Cof modulus 1, and x ∈C
2a non-zero vector such that RTx = λx. Then it
is easy to see that x2,0, and thus kRTxk1<kxk1, a contradiction with the assumption that λ has modulus
one. We have proved that the eigenvalues of RT, which are also those of R, have modulus strictly smaller than 1.
This clearly implies that Q and the series
+∞
X
n=0
Rn are well defined. Now, observe that
(I − R) +∞ X n=0 Rn= +∞ X n=0 Rn− +∞ X n=1 Rn = I, so in fact Q =P+∞ n=0Rn.
Finally, one has to see that Rni j is the probability that starting from i, the walk is at j at time n. By definition, this probability is
X
i=i0,i1,...,in−1,in= j
Pi0 i1· · ·Pin−1in,
where i1, . . . ,in−1 take all possible values in {1, 2, 3}. But since state 3 is absorbing (that is, P3 j = 0 if j , 3), the sum above is zero if one ikis equal to 3. So the sum is equal to
X
i=i0,i1,...,in−1,in= j
Ri0 i1· · ·Rin−1in,
where i1, . . . ,in−1take all possible values in {1, 2}. This last expression is precisely Rni j.
(3) A computation shows that
Q =" 3800 203600 20 #
.
Starting from a car that is working, the expected time before it goes out of order is the sum of expected times spent in states 1 and 2, which is thus 3820 days (and on average, the total number of days during which the car is working before it gets out of order is 3800 days).
(4) A careful examination of the proof of part (2) shows that the only requirement is that the “out of order” state can be reached from any other state in a finite number of steps.
(5) For instance, to get the expected time before getting “SF”, we may consider the (3 × 3) reduced transition matrix R given by
FF FS S S
FF q p 0
FS 0 0 p
S S 0 0 p
.
For the inverse, we obtain
(I − R)−1= 1/p 1 p/q 0 1 p/q 0 0 1/q .
It takes two tosses to “get the chain started”. But the moment of appearance of “SF” is not changed if instead we start in the state “FF”. We get that the expected time before “SF” appears is
1 p + 1 + p q = 1 p + 1 q, as expected.
Series 5: stationary distributions
Exercise 1
Consider a Markov chain with transition probabilities matrix
P = p0 p1 p2 · · · pm pm p0 p1 · · · pm−1 · · · · p1 p2 p3 · · · p0
where 0 < pi <1 for each i and p0+ p1+ · · · + pm= 1. Determine lim
n→∞P
n
i j, the stationary distribution.
Exercise 2
An airline reservation system has two computers only one of which
is in operation at any given time. A computer may break down any given day with probability p. There is a single repair facility which takes 2 days to restore a computer to normal. The facilities are such that only one computer at a time can be dealt with. Form a Markov chain by taking as states the pairs (x, y) where x is the number of machines in operating conditions at the end of a day and y is 1 if a day’s labor has been expended on a machine not yet repaired and 0 otherwise. Enumerating the states in the order (2, 0), (1, 0), (1, 1), (0, 1), the transition matrix is
q p 0 0 0 0 q p q p 0 0 0 1 0 0
where p, q > 0 and p + q = 1. Find the stationary distribution in terms of p and q. Exercise 3
Sociologists often assume that the social classes of successive
generations in a family can be regarded as a Markov chain. Thus, the occupation of a son is assumed to depend only on his father’s occupation and not on his grandfather’s. Suppose that such a model is appropriate and that the transition probability matrix is given by
Son’s class
Lower Middle Upper
Lower .40 .50 .10
Father’s class Middle .05 .70 .25
Upper .05 .50 .45
For such a population, what fraction of people are middle class in the long run? Exercise 4
Suppose that the weather on any day depends on the weather
conditions for the previous two days. To be exact, suppose that if it was sunny today and yesterday, then it will be sunny tomorrow with probability .8; if it was sunny today but cloudy yesterday, then it will be sunny tomorrow with probability .6; if it was cloudy today but sunny yesterday, then it will be sunny tomorrow with probability .4; if it was cloudy for the last two days, then it will be sunny tomorrow with probability .1.
Such a model can be transformed into a Markov chain provided we say that the state at any time is determined by the weather conditions during both that day and the previous day. We say the process is in
• State (S,S) if it was sunny both today and yesterday; • State (S,C) if it was sunny yesterday but cloudy today; • State (C,S) if it was cloudy yesterday but sunny today; • State (C,C) if it was cloudy both today and yesterday.
Enumerating the states in the order (S,S), (S,C), (C,S), (C,C), the transition matrix is .8 .2 .4 .6 .6 .4 .1 .9
Find the stationary distribution for the Markov chain. Exercise 5
(Chapter 3, Problem 4) Consider a discrete time Markov chain with states 0, 1, . . . , N whose matrix has elements Pi j= µi, j = i − 1, λi, j = i + 1, i, j = 0, 1, . . . , N; 1 − λi− µi, j = i; 0, |j − i| > 1.
Suppose that µ0 = λ0 = µN = λN = 0, and all other µi’s and λi’s are strictly positive, and that the initial
Series 5: stationary distributions Solutions
Exercise 1
Since all entries in the transition matrix are strictly positive, the chain is irreducible and positive recurrent, so there exists a unique invariant measure. By the symmetric structure of the matrix, we can guess that the uniform distribution π = (m+11 , . . . ,m+11 ) is invariant, and indeed it is trivial to check that πP = π. Exercise 2
Following the enumeration in the exercise, let (π1, π2, π3, π4) denote the stationary distribution. It must
satisfy: π1+ π2+ π3+ π4 = 1; (1) qπ1+ qπ3 = π1; (2) pπ1+ pπ3+ π4 = π2; (3) qπ2 = π3; (4) pπ2 = π4. (5) Applying (4) in (2), we get qπ1+ q2π2= π1 =⇒ π1= q2 pπ2. Using this, (4) and (5) in (1), we get
q2 pπ2+ π2+ pπ2+ qπ2= 1 =⇒ π2= p 1 + p2. We thus get π1= q 2 1+p2, π2= p 1+p2, π3= qp 1+p2, π4= p2 1+p2. Exercise 3
Since all the entries of the transition matrix are positive, the chain is irreducible and aperiodic, so there exists a unique stationary distribution. Denoting this distribution by π = (x, y, z), the quantity we are looking for – namely, the fraction of people that are middle class in the long run – is equal to y. We know that x + y + z = 1 and also that
(x y z) .40 .50 .10 .05 .70 .25 .05 .50 .45 = (x y z); solving the corresponding system of linear equations, we get x = 131, y =
5
8 and z = 31 104.
Exercise 4
Let x, y, z, w denote the states of the chain in the order that they appear in the transition matrix. The invariant measure is obtained by solving the system
(x y z w) .8 .2 .4 .6 .6 .4 .1 .9 = (x y z w); x + y + z + w = 1.
The unique solution is (x y z w) = (113, 1 11, 1 11, 6 11). Exercise 5
For k ∈ {0, . . . , N}, define f (k) = P(∃n > 0 : Xn = N | X0 = k). Since 0 and N are absorbing, we have
f (0) = 0, f (N) = 1. For k ∈ {1, . . . , N − 1}, we have
f (k) = µkf (k − 1) + λkf (k + 1) =⇒ f (k + 1) − f (k) =
µk
λk
( f (k) − f (k − 1)).
For k ∈ {1, . . . , N}, define ∆k = f (k) − f (k − 1). We can thus rewrite what we have obtained above as
∆k+1 = µλkk∆k; iterating this we get ∆k =
µ1· · · µk−1
λ1· · · λk−1∆1
∀k > 2 We can now find ∆1:
1 = f (N) − f (0) = ∆1+ · · · + ∆N = 1 + µ1 λ1 + · · · + µ1· · · µk−1 λ1· · · λk−1 ! ∆1 =⇒ ∆1= 1 + µ1 λ1 + · · · + µ1· · · µk−1 λ1· · · λk−1 !−1
Then, for k ∈ {1, . . . , N}, we have
f (k) = f (k) − f (0) = ∆1+ · · · ∆k = ∆1+ µ1 λ1∆1· · · + ∆k = 1 + µ1 λ1 + · · · + µ1···µk−1 λ1···λk−1 µ1 λ1 + · · · + µ1···µN−1 λ1···λN−1 .
Series 6: hitting times
Exercise 1
Let (Xn) be an irreducible Markov chain with stationary distribution π. Find Eπ(min{n : Xn = X0}).
Exercise 2
Let P be the transition matrix of an irreducible Markov chain in state space S . A distribution π on S is called reversible for P if, for each x, y ∈ S , we have π(x)P(x, y) = π(y)P(y, x).
(a) Show that, if π is reversible for P, then π is stationary for P.
(b) Assuming that π is reversible for P, show that, for any n and any x0,x1, . . .xn ∈S ,
Pπ(X0= x0, . . . ,Xn= xn) = Pπ(Xn= x0, . . . ,X0= xn).
(c) Random walk on a graph. Let G = (V, E) be a finite connected graph. For each x ∈ V, let d(x) denote the degree of x. Let P be the Markov transition matrix for the Markov chain on V given by
P(x, y) =
( 1
d(x) if (x, y) ∈ E;
0 otherwise. This is called the random walk on G. Show that π(x) = d(x)/P
zd(z) is reversible for P.
Exercise 3
We identify the square grid Λ = {1, . . . , 8}2with a chessboard. In chess, a knight is allowed to move on the board as follows. If the knight is in position (x, y), it can go to any of the positions in
Λ ∩ (x + 1, y + 2), (x − 1, y + 2), (x − 2, y + 1), (x − 2, y − 1), (x − 1, y − 2), (x + 1, y − 2), (x + 2, y − 1), (x + 2, y + 1)
! .
Let (Xn) denote the sequence of positions of a knight that at time zero is in position (1, 1) and at each step
chooses uniformly among its allowable displacements. Find the expected time until the knight returns to (1, 1).
(Hint: describe the chain as in Exercise 2(c).) Exercise 4
Consider a finite population (of fixed size N) of individuals of possible types
A and a undergoing the following growth process. At instants of time t1 <t2 <t3 < . . ., one individual
dies and is replaced by another of type A or a. If just before a replacement time tn there are j A’s and
N − j a’s present, we postulate that the probability that an A individual dies is jµ1/Bj and that an a
individual dies is (N − j)µ2/Bjwhere Bj = µ1j + µ2(N − j). The rationale of this model is predicated on
the following structure: Generally a type A individual has chance µ1/(µ1+ µ2) of dying at each epoch tn
and an a individual has chance µ2/(µ1+ µ2) of dying at time tn. (µ1/µ2can be interpreted as the selective
advantage of A types over a types). Taking account of the sizes of the population it is plausible to assign the probabilities µ1j/Bjand (µ2(N − j)/Bj) to the events that the replaced individual is of type A and type
a, respectively. We assume no difference in the birth pattern of the two types and so the new individual is taken to be A with probability j/N and a with probability (N − j)/N.
(1) Describe the transition probabilities of the Markov chain (Xn) representing the number of individuals
of type A at times tn(n = 1, 2, . . .).
Exercise 5
Let (Xk) be a Markov chain for which i is a recurrent state. Show that
lim
N→∞
P(Xk,i for n + 1 6 k 6 n + N | X0= i) = 0.
If i is a positive recurrent state prove that the convergence in the above equation is uniform with respect to n.
Series 6: hitting times Solutions Exercise 1 Eπ(min{n : Xn= X0}) = X x∈S π(x) · Ex(min{n : Xn= X0}) = X x∈S π(x) · 1 π(x) = #S . Exercise 2 (a) For any x ∈ S ,
X y π(y)P(y, x) =X y π(x)P(x, y) = π(x)X y P(x, y) = π(x), so π is stationary. (b) Pπ(X0= x0, . . . ,Xn= xn) = π(x0)P(x0,x1)P(x1,x2) · · · P(xn−1,xn) = P(x1,x0)π(x1)P(x1,x2) · · · P(xn−1,xn) = P(x1,x0)P(x2,x1)π(x2)P(x2,x3) · · · P(xn−1,xn) = · · · = P(x1,x0)P(x2,x1)P(x3,x2) · · · P(xn,xn−1)π(xn) = Pπ(Xn = x0, . . . ,X0= xn).
(c) Let x, y ∈ V. If x and y are not adjacent, we have π(x)P(x, y) = π(y)P(y, x) = 0. Otherwise, π(x)P(x, y) = Pd(x) zd(z) · 1 d(x) = 1 P zd(z) = d(y) P zd(z) · 1 d(y) = π(y)P(y, x). Exercise 3
We give a graph structure to the chessboard by taking the vertex set to be Λ and setting edges x ∼ y if and only if the knight can go from x to y (this implies that it can also go from y to x, so the relation is symmetric). The stochastic sequence (Xn) is then a random walk on this graph.
We claim that the graph is connected (equivalently, that the chain is irreducible). To see this, consider the reduced board Λ′= {1, 2, 3}2. Without leaving Λ′, it is possible to go from (1, 1) to (1, 2): performing the moves (1, 1) → (3, 2) → (1, 3) → (2, 1); of course, reversing these moves allows us to go from (2, 1) to (1, 1). Now, consider two adjacent positions x, y ∈ Λ (x and y differ by 1 in one coordinate and coincide in the other). For any such pair x, y, it is possible to find a 3x3 subset of Λ such that either x or y is one of the corners of the square. Then, except possibly for a rotation, the moves given for Λ′can be repeated so that we can go from x to y and from y to x. Now, for arbitrary points x, y ∈ Λ, we can construct a path from x to y such that at each step we go to an adjacent position, so all cases are now covered.
By Exercise 2(c), the stationary distribution is given by π(x) = d(x)/P
of the graph are given by 2 3 4 4 4 4 3 2 3 4 6 6 6 6 4 3 4 6 8 8 8 8 6 4 4 6 8 8 8 8 6 4 4 6 8 8 8 8 6 4 4 6 8 8 8 8 6 4 3 4 6 6 6 6 4 3 2 3 4 4 4 4 3 2
The expected return time to (1, 1) is given by 1/π((1, 1)) = (P
zd(z))/d((1, 1)) = 336/2 = 168.
Exercise 4
(1) The transition probabilities are given by Pj, j−1 = µ1j(N − j) BjN , Pj, j+1= µ2(N − j) j BjN , Pj j = 1 − Pj, j−1−Pj, j+1, Pi j = 0, for |i − j| > 1.
(2) We are exactly in the context of last week’s Exercise 5. Using what was obtained there, we have P(∃n : Xn= 0 | X0 = 0) = 1, P(∃n : Xn= 0 | X0= N) = 0 and P(∃n : Xn = N | X0= k) = 1 + P1,0 P1,2 + P1,0P2,1 P1,2P2,3 + · · · P1,0···Pk−1,k−2 P1,2···Pk−1,k 1 + P1,0 P1,2 + P1,0P2,1 P1,2P2,3 + · · · P1,0···PN−1,N−2 P1,2···PN−1,N
Since in our present case we have Pk,k−1
Pk,k+1 =
µ1
µ2 for all k, we conclude that
P(∃n : Xn= 0 | X0= k) = 1 − P(∃n : Xn= N | X0 = k) = 1 − 1 +µ1 µ2 + · · · + µ 1 µ2 k−1 1 +µ1 µ2 + · · · + µ 1 µ2 N−1 = µ 1 µ2 N −µ1 µ2 k µ 1 µ2 N −1 . Exercise 5
Let τi = min{k > 1 : Xk = i} be the first return time to i. When i is recurrent, we have Pi(τi < ∞) = 1, so
(1) lim
K→∞Pi(τi >K) = 0.
If, in addition, i is positive recurrent, we have
∞ X l=0 Pi(τi>l) = Ei(τi) < ∞, so that (2) lim K→∞ ∞ X l=K Pi(τi >l) = 0.
It follows from (1) that, if we keep n fixed and take N to infinity, the last expression converges to zero. In addition, if i is positive recurrent, we can use (2) to obtain
n X l=1 Pi(τi >N + l) = N+n X l=N+1 Pi(τi >l) 6 ∞ X l=N+1 Pi(τi>l) → 0
Series 7: recap on Markov chains
Exercise 1
Let S = {0, 1, 2, 3}. We consider the Markov chain on S whose transition matrix is given by
P = 1/4 1/4 1/2 0 1/6 1/6 1/3 1/3 0 0 1/3 2/3 0 0 1/2 1/2 .
1) What are the communication classes, the transient states ? Is the chain aperiodic ? If not, give the period.
2) What is P200? What approximately is P100?
3) Starting from 2, what is the expected number of visits to state 3 before returning to 2 ? Exercise 2
Consider the Markov chain on the state space below, where at each step a neighbour is uniformly chosen among sites that are linked by an edge to the current position.
1) What are the communication classes, the transient states ? Is the chain aperiodic ? If not, give the period.
2) Is there an invariant measure ? Is it unique ? In case it is, give it. 3) Starting from x, what is the probability to hit S before F ?
4) Starting from x, what is the expected number of visits to F before returning to x ? Exercise 3
We have n coins C1, . . . ,Cn. For each k, Ckis biased so that it has probability 1/(2k + 1) of falling heads.
Series 7: recap on Markov chains Solutions
Exercise 1
The communication classes are {0, 1} and {2, 3}. To see that state 0 is transient, note that for the chain starting at 0, there is a non-zero probability that the chain goes to state 2. Once there, there is no path going back to 0, so indeed, the probability that the return time to 0 is infinite is strictly positive. The same holds for state 1. The chain is aperiodic on its recurrent class {2, 3}.
2) We have P200 = (P00)2+ P01P10= 1 16 + 1 24 = 5 48.
To approximate P100(that is, to find what Pnlooks like when n tends to infinity), we first compute the stationary distribution on the communication class {2, 3}, which has transition matrix
"
1/3 2/3 1/2 1/2
# .
We find a stationary distribution π with weights (3/7 4/7). On the set {2, 3}, the chain is irreducible and aperiodic, so starting from any point, the distribution of Xn converges to the stationary distribution.
Hence, for i, j ∈ {2, 3}, one has Pni j → π( j) as n tends to infinity. Clearly, if i ∈ {2, 3} and j ∈ {0, 1}, then Pni j = 0.
If j ∈ {0, 1}, then it is a transient state, so for any i, we have Pni j → 0 (in fact, we proved a stronger statement in exercise 1 of series 3: thatPnPni jis finite). There remains to determine Pni j when i ∈ {0, 1} and j ∈ {2, 3}. Intuitively, the chain enters the class {2, 3} at some point, and then converges to equilibrium in the class {2, 3}, so we should have
(1) lim
n→+∞P n
i j= π( j).
Here is a rigorous proof of this fact. Let
τ = inf{n : Xn∈ {2, 3}}.
Since i is transient, Pi[τ = +∞] = 0. For any ε > 0, we can find N large enough such that Pi[τ ≥ N] ≤ ε.
For j ∈ {2, 3} and n ≥ N, we have
(2) Pni j = Pi[Xn = j] = N−1 X k=0 Pi[Xn= j, τ = k] + Pi[τ ≥ N] | {z } ≤ε . We decompose Pi[Xn = j, τ = k] into Pi[Xn = j, τ = k, Xτ= 2] + Pi[Xn= j, τ = k, Xτ= 3],
and we apply the strong Markov property at time τ on each term. The first term becomes Pi[τ = k, Xτ= 2] P2[Xn−k = j] −−−−−→
n→+∞ Pi[τ = k, Xτ= 2] π( j),
where we used the fact that the Markov chain on {2, 3} is aperiodic. Similarly, Pi[Xn= j, τ = k, Xτ= 3] −−−−−→
and putting the two together, we obtain
Pi[Xn= j, τ = k] −−−−−→
n→+∞ Pi[τ = k] π( j).
So the sum appearing in the right-hand side of (2) is such that
N−1
X
k=0
Pi[Xn = j, τ = k] −−−−−→
n→+∞ Pi[τ < N] π( j),
and 1 − ε ≤ Pi[τ < N] ≤ 1. We have thus proved that
lim sup n→+∞ Pni j ≤ π( j) + ε, and lim inf n→+∞ P n i j ≥(1 − ε)π( j).
Since ε > 0 was arbitrary, this proves (1). Conclusion : P100≃ lim n→+∞P n = 0 0 3/7 4/7 0 0 3/7 4/7 0 0 3/7 4/7 0 0 3/7 4/7 .
3) We can use the fact that this expected value is exactly π(3)/π(2) = 4/3. We can also take a more explicit approach and decompose the possible movements of the chain: with probability 1/3 the walk stays at 2 and therefore the number of visits to 3 is zero. Else it goes to 3, and then stays for a time geometric(1/2), and then goes back to state 2. The expectation we are looking for is thus
1 3 0 + 2 3 +∞ X k=1 k 1 2 !k = 4 3. Exercise 2
1) The Markov chain is a random walk on a graph (see exercise 2 of series 5). Let G = (V, E) denote the graph. G is connected, that is, for any x, y ∈ V, there exists a path x0 = x, x1,x2, . . . ,xN = y such
that (xi,xi+1) ∈ E for each i. We then have Px(y is ever reached) > P(x, x1)P(x1,x2) · · · P(xN−1,y) > 0,
so the chain is irreducible, and there are no transient states. The chain is also aperiodic. To see this, fix any vertex that is the extreme point of one of the diagonal segments. Starting from this vertex, we can return to it in 3 steps (using the diagonal) and also in 4 steps (not using the diagonal), so the period of this vertex must divide 3 and 4, so it is 1. Since the chain is irreducible, we conclude that all vertices have period 1, so the chain is aperiodic.
2) since the chain is irreducible and the state space is finite, there exists a unique invariant measure. We can look for a reversible measure as in exercise 2 of series 5, and we find that the equilibrium distribution πevaluated at a vertex x is equal to the degree of x divided by the sum of the degrees of all vertices. In our case, this gives the following invariant measure.
2/52 3/52 3/52 2/52
3/52 5/52 5/52 3/52
3) Consider the diagram
1/2 1 − b 1 − a 0
b 1/2 1 − c 1 − a
a c 1/2 1 − b
1 a b 1/2
The value at each site represents the probability, starting from this site, to arrive at S before F. Notice that we have used symmetry several times. Using the relation
Px(S is reached before F) = 1 deg(x)
X
y∼x
Py(S is reached before F),
valid for any x < {S , F}, we obtain the system of linear equations a = 13+ 1 3b + 1 3c b = 13a +13 c = 25a + 2 5· 1 2+ 1 5(1 − c)
Solving this gives b = 47. 4) This is π(F)/π(x) = 2/3. Exercise 3 Let us write f (x) = n X k=0 akxk.
By developping, we observe that akis the probability of getting exactly k heads. We want to know what
is a1+ a3+ a5+ · · · . Note that f (1) = n X k=0 ak, while f (−1) = n X k=0 ak(−1)k = X k even ak− X k odd ak.
So the quantity we are interested in is
f (1) − f (−1)
2 .
Clearly, f (1) = 1, while an easy computation gives f (−1) = 1/(2n + 1). The probability we are interested in is thus n/(2n + 1).
Series 8: Poisson processes
Exercise 1
Let X1 and X2 be independent exponentially distributed random
variables with parameters λ1and λ2so that
P(Xi >t) = exp(−λit) for t > 0. Let N =( 1 if X2 if X1 <X2; 2 6X1, U = min{X1,X2}= XN, V = max{X1,X2}, W = V − U = |X1−X2|. Show a) P(N = 1) = λ1/(λ1+ λ2) and P(N = 2) = λ2/(λ1+ λ2).
b) P(U > t) = exp{−(λ1+ λ2)t} for t > 0.
c) N and U are independent random variables.
d) P(W > t|N = 1) = exp{−λ2t} and P(W > t|N = 2) = exp{−λ1t} for t > 0.
e) U and W = V − U are independent random variables. Exercise 2
Assume a device fails when a cumulative effect of k shocks occur.
If the shocks happen according to a Poisson process with parameter λ, find the density function for the life T of the device
Answer: f (t) = λktk−1e−λt Γ(k) if t > 0; 0, if t 6 0. Exercise 3
Let {X(t), t > 0} be a Poisson process with intensity parameter λ.
Suppose each arrival is “registered” with probability p, independently of other arrivals. Let {Y(t), t > 0} be the process of “registered” arrivals. Prove that Y(t) is a Poisson process with parameter λp.
Exercise 4
At time zero, a single bacterium in a dish divides into two bacteria. This species of bacteria has the following property: after a bacterium B divides into two new bacteria B1and B2, the subsequent length
