1.4 Thesis overview
2.1.2 Continuous-time Markov chains
Continuous-time Markov chains are continuous-time analogous to discrete-time Markov chains. Continuous-time Markov chains are of principle interest in this thesis, and we provide a little more exposition of the related theory.
Definition 19 (Continuous-time Markov chain).
We call a stochastic process tXptq :tě 0u with discrete state space S a continuous- time Markov chain (CTMC) if for all tě0, sąuě0, i, j, xpuq PS,
PpXpt`sq “j | Xpsq “i, Xpuq “xpuqq “PpXps`tq “j |Xpsq “iq. (2.8) Equation (2.8) is the Markov property for continuous-time processes. It is analogous to the Markov property for the discrete-time case in that it defines a system whose future evolution depends on its history only through the present state. In continuous- time the Markov property specifies that transition probabilities do not depend on the time spent in a particular state.
Definition 20 (Time homogeneity).
We call a CTMC time-homogeneous if for all t, sě0, i, j PS, we have
PpXpt`sq “j | Xpsq “iq “PpXptq “j |Xp0q “iq. (2.9) We will be considering time-homogeneous Markov chains throughout this thesis. Definition 21 (Transition matrix).
In the case of a time-homogeneous CTMC, we define the transition matrix Pptq “ rPijptqsi,jPS,
where for all i, jPS, tě0,
Markov processes 25
Definition 22 (The generator matrix).
We define generator matrix Q“ rqijssuch that
Q“ d dtPptq ˇ ˇ ˇ t“0 “P 1 p0q. (2.11)
Definition 23 (Holding time).
Assuming the process is in state iat time 0, we define
Hi “infttą0 :Xptq ‰iu,
referred to as the holding time in state i. Hi is a strictly positive continuous random
variable.
The following proposition is demonstrated in Sections 5.2.2 and 6.2 of Ross [100]. Proposition 3.
For all iPS, Hi „ Expp´qiiq. That is, the holding time for a CTMC is necessarily exponentially distributed with parameter ´qii.
The following three theorems are demonstrated by Lemma 6.3, Theorem 6.1, and Theorem 6.2 of Ross [100] respectively.
Theorem 4 (Chapman–Kolmogorov equations). For all tě0, sě0,
Ppt`sq “PpsqPptq, (2.12) or equivalently, for all tě0, sě0, i, jPS,
Pijpt`sq “
ÿ
kPS
PikpsqPkjptq. (2.13)
Theorem 5 (Kolmogorov backward equations). For all i, jPS, tě0 P1 ijptq “ ÿ k qikPkjptq, (2.14) or equivalently, P1ptq “QPptq. (2.15)
Theorem 6 (Kolmogorov forward equations).
Under certain regularity conditions (see remark below) we have, for all i, jPS, tě0,
P1 ijptq “ ÿ k qkjPikptq, (2.16) or equivalently P1ptq “PptqQ. (2.17)
26 Markov processes
Remark 1.
The regularity conditions of Theorem 6 are satisfied whenever the process undergoes at most finitely many transitions in a finite time. This is trivially satisfied when the state space is finite, and will be satisfied for all of the models discussed in this thesis. Section 6.9 of Ross [100] establishes the following proposition.
Proposition 4.
The solution to the Kolmogorov backward and forward equations is Pptq “eQt.
Definition 24 (Accessibility).
If Pijptq ą0 for some tě0, we say thatj is accessible from i.
Definition 25 (Communication).
If i and j are accessible from each other, they are said to communicate. We denote the relation thus defined by iØj.
As in the discrete-time case,Ø is an equivalence relation.
Definition 26 (Communicating class).
We refer to the equivalence classes of Ø as communicating classes. Definition 27 (Irreducible continuous-time Markov chain).
We say that a CTMC is irreducible if iØj for all i, jPS. Definition 28 (Embedded chain).
Let tn denote the time at which the nth transition from some state i to some other
state j occurs and let
Xn“ $ ’ & ’ % Xp0q for n“0 lim tÑt`n Xptq for ně1. (2.18)
Then tXn:ně0u is a DTMC tracking the state changes of the CTMC. We refer to
tXn:ně0u as the embedded chain.
Notice that Xn is the value taken byXptqimmediately after thenth change of state.
The following proposition is established in Chapter 6 (immediately preceding Theorem 6.8) of Kulkarni [70].
Markov processes 27
Proposition 5.
The one step transition matrixP“ rPijs of the embedded chain is given by
Pij “ $ ’ ’ ’ ’ ’ ’ ’ & ’ ’ ’ ’ ’ ’ ’ % qij ´qii if qii‰0, i‰j 0 if qii‰0, i“j 0 if qii“0, i‰j 1 if qii“0, i“j. (2.19)
The following proposition follows from Theorem 6.8 of Kulkarni [70]. Proposition 6.
A CTMC is irreducible if and only if its embedded chain is irreducible. Definition 29 (Time to return).
Define
τi “infttąs:Xptq “i| Xp0q “i, Xpsq ‰iu, (2.20) interpreted as the time taken for the process to return to state i given that it started there.
Definition 30 (Recurrence and Transience).
We call a state recurrent if Ppτiă 8q “1, and transient otherwise. Definition 31 (Positive-recurrence and null-recurrence).
If a state i is recurrent, and Epτiq ă 8, we say the statei is positive-recurrent, and if Epτi “ 8q we say i is null-recurrent.
Theorem 6.9 of Kulkarni [70] proves the following. Proposition 7.
A state iof a CTMC is recurrent (transient) if and only if it is recurrent (transient) in the embedded chain.
The same does not apply for positive- and null-recurrence.
Theorems 6.9 and 6.10 of Kulkarni [70] prove that, under the regularity conditions mentioned in Remark 1, transience, recurrence, positive-recurrence and null-recurrence are all class properties (in the sense of communication classes). The next theorem follows from this result.
28 Markov processes
Theorem 7.
For a regular irreducible CTMC all states are together transient, positive-recurrent, or null-recurrent.
As in the discrete-time case we call a CTMC positive-recurrent, null-recurrent or transient if it is irreducible and all states are such.
Definition 32 (Stationary distribution).
We call a vector π “ rπjsa stationary distribution if, for all tě0,
πPptq “π, (2.21)
and,
ÿ
jPS
πj “1. (2.22)
The following corollary is established in Section 6.5 of Ross [100]. Corollary 2.
Any stationary distribution satisfies
πQ“0, (2.23) or equivalently ´qjjπj “ ÿ kPS k‰j πkqkj, (2.24)
where 0 represents a vector of zeros of appropriate size.
We call the system of equations defined by Equation (2.23) the balance equations. Remark 2.
Throughout, we will use 0,1 to denote vectors of zeros and ones of appropriate size respectively. Likewise, we use 0 and 1 to denote matrices full of zeroes and ones of appropriate size respectively. We use ei to represent a vector with a one in the ith
entry and zeros elsewhere.
Definition 33 (Limiting distribution).
Assuming the limits exist and are independent of i, we define limiting distribution
π˚“ rπ˚
js, where the limiting probabilities πj˚ for each jPS are given by πj˚“ lim
Markov processes 29
Definition 34 (Ergodicity).
We say that a CTMC is ergodic when the limiting distribution π˚ exists.
The following proposition is established in Section 6.5 of Ross [100]. Proposition 8.
Given an irreducible, positive recurrent CTMC, the limiting distribution exists, and is equal to the unique stationary distribution.
2.1.3 Absorbing CTMCs