Discrete-Time Markov Chains
Definition 8.3 The transition probability matrix associated with any DTMC is a
matrix,P, whose(i, j)th entry,Pij, represents the probability of moving to statej on the next transition, given that the current state isi.
Observe that the transition probability matrix,P, might have infinite order, if there are infinitely many states. Also observe that by definition,jPij = 1, ∀i, because, given that the DTMC is in statei, it must next transition to some statej.
We begin this chapter by focusing on DTMCs with a finite number of states, M. Later in the chapter, we generalize to DTMCs with an infinite number of states. In this chapter, we do not discuss issues of ergodicity. Specifically, we do not dwell on questions of the existence of limiting probabilities. We simply assume that there exists some limiting probability of being in each state of the chain (to be defined soon), and we defer all discussion of the existence of these limits to Chapter9.
8.3 Examples of Finite-State DTMCs
We start with a few examples of some simple Markov chains to illustrate the key concepts. More involved and interesting examples are saved for the exercises.
8.3.1 Repair Facility Problem
A machine is either working or in the repair center. If it is working today, then there is a 95% chance that it will be working tomorrow. If it is in the repair center today, then there is a 40% chance that it will be working tomorrow. We are interested in questions like “what fraction of time does my machine spend in the repair shop?”
Question: Describe the DTMC for the repair facility problem.
Answer: There are two states, “Working” and “Broken,” where “Broken” denotes that
the machine is in repair. The transition probability matrix is
P = . W B W 0.95 0.05 B 0.40 0.60 / .
The Markov chain diagram is shown in Figure8.1.
0.6 0.95
0.05
0.4
Working Broken
132 discrete-time markov chains
Question: Now suppose that after the machine remains broken for 4 days, the machine
is replaced with a new machine. How does the DTMC diagram change?
Answer: The revised DTMC is shown in Figure8.2. 0.05 0.4 0.4 0.4 1 0.6 0 0.95 Broken Day 1 Working 0 0.6 0.6 Broken Day 2 Broken Day 4 Broken Day 3
Figure 8.2. Markov chain for repair facility problem with 4-day limit.
8.3.2 Umbrella Problem
An absent-minded professor has two umbrellas that she uses when commuting from home to office and back. If it rains and an umbrella is available in her location, she takes it. If it is not raining, she always forgets to take an umbrella. Suppose that it rains with probability p each time she commutes, independently of prior commutes. Our eventual goal is to determine the fraction of commutes during which the professor gets wet.1
Question: What is the state space?
Hint: You can model this with three states!
Answer: The states track the number of umbrellas available at the current location,
regardless of what this current location is. The DTMC is shown in Figure8.3.
The transition probability matrix isP =
⎡ ⎣ 00 1− p p0 1 1− p p 0 ⎤ ⎦ . p 1–p 1.0 1–p p
Figure 8.3. DTMC for umbrella problem.
8.3.3 Program Analysis Problem
A program has three types of instructions: CPU instructions (C), Memory instructions (M), and User interaction instructions (U). In analyzing the program, we note that
8.4 powers ofP: n-step transition probabilities 133 a C instruction with probability 0.7is followed by another C instruction, but with probability0.2is followed by an M instruction and with probability0.1is followed by a U instruction. We also note that an M instruction with probability0.1is followed by another M instruction, but with probability 0.8 is followed by a C instruction, and with probability0.1is followed by a U instruction. Finally, a U instruction, with probability0.9is followed by a C instruction, and with probability0.1is followed by an M instruction.
In the exercises for this chapter and the next, we answer questions like, “What is the fraction of C instructions?” and “What is the mean length of the instruction sequence between consecutive M instructions?” For now, we simply note that the program can be represented as a Markov chain with the transition probability matrix,P:
P = ⎡ ⎣ C M U C 0.7 0.2 0.1 M 0.8 0.1 0.1 U 0.9 0.1 0 ⎤ ⎦
8.4 Powers ofP: n-Step Transition Probabilities
Let Pn = P · P · · · P, multiplied n times. We will use the notation Pn
ij to denote (Pn)
ij.
Question: What doesPn
ijrepresent?
Answer: To answer this, we first consider two examples.
Umbrella Problem
Consider the umbrella problem from before where the chance of rain on any given day isp = 0.4. We then have P = ⎡ ⎣ 00 0.6 0.40 1 0.6 0.4 0 ⎤ ⎦ , P5 = ⎡ ⎣.06 .30 .64.18 .38 .44 .38 .44 .18 ⎤ ⎦ , P30 = ⎡ ⎣.230 .385 .385.230 .385 .385 .230 .385 .385 ⎤ ⎦ .
Observe that all the rows become the same! Note also that, for all the above powers, each row sums to1.
Repair Facility Problem
Now, consider again the simple repair facility problem, with general transition proba- bility matrixP: P = . 1− a a b 1− b / , 0 < a < 1, 0 < b < 1
134 discrete-time markov chains You should be able to prove by induction that
Pn = b+ a(1−a−b)n a+ b a−a(1−a−b)n a+ b b−b(1−a−b)n a+ b a+ b(1−a−b)n a+ b . lim n→∞P n = b a+ b a a+ b b a+ b a a+ b .
Question: Again, all rows are the same. Why? What is the meaning of the row? Hint: Consider a DTMC in statei. Suppose we want to know the probability that it will be in statej two steps from now. To go from stateito statej in two steps, the DTMC must have passed through some statekafter the first step. Below we condition on this intermediate statek:
For anM-state DTMC, as shown in Figure8.4,
Pij2 = M−1
k = 0
Pik · Pk j
=Probability that after 2 steps we will be in statej, given that we are in stateinow.
i
j
Pij=
2
Figure 8.4. Pi j2.
Likewise, then-wise product can be viewed as
Pn ij =
M−1 k = 0
Pikn−1Pk j
= Probability of being in statejinnsteps, given we are in stateinow.
Limiting Probabilities
We now move on to looking at the limit. Consider the(i, j)th entry of the power matrix Pn for largen: lim n→∞P n ij = lim n→∞P n ij
This quantity represents the limiting probability of being in statejinfinitely far into the future, given that we started in statei.
Question: So what is the limiting probability of having0umbrellas?
8.5 stationary equations 135
Question: The fact that the rows oflimn→∞Pn are all the same is interesting because it says what?
Answer: It says that the starting state (i) does not matter.
Definition 8.4 Let
πj = lim n→∞P
n ij.
πjrepresents the limiting probability that the chain is in statej(independent of the starting statei). For anM-state DTMC, with states0,1,. . .,M− 1,
π = (π0, π1, . . . , πM−1), where M−1
i= 0
πi= 1 represents the limiting distribution of being in each state.
Important Note: As defined,πj is a limit. Yet it is not at all obvious that the limitπj exists! It is also not obvious thatπrepresents a distribution (i.e.,iπi= 1), although this latter part turns out to be easy to see (Exercise8.2). For the rest of this chapter, we assume that the limiting probabilities exist. In Chapter9we look at the existence question in detail.
Question: So what is the limiting probability that the professor gets wet?
Answer: The professor gets wet if both (i) the state is 0, that is there are zero umbrellas
in the current location; and (ii) it is raining. So the limiting probability that the professor gets wet on any given day isπ0 · p = (0.23)(0.4) = .092.
Question: Can you see why the limiting probability of having 1 umbrella is equal to
the limiting probability of having 2 umbrellas?
Answer: This is a little tricky. Notice that if we are only looking at the DTMC from the
perspective of 1 versus 2 umbrellas, then the chain becomes symmetric. The collapsed chain is shown in Figure8.5.
1–p 1–p
p
p
Figure 8.5. Compressed umbrella problem.
8.5 Stationary Equations
Question: Based only on what we have learned so far, how do we determineπj = limn→∞Pn
ij?
Answer: We take the transition probability matrixPand raise it to thenth power for some largenand look at thejth column, any row.
136 discrete-time markov chains
Question: Multiplying Pby itself many times sounds quite onerous. Also, it seems one might need to perform a very large number of multiplications if the Markov chain is large. Is there a more efficient way?
Answer: Yes, by solving stationary equations, given in Definition8.5.
Definition 8.5 A probability distribution π = (π0, π1, . . . , πM−1) is said to be