Simulations

When a numerical solution is not possible, we often rely on simulations. Fortunately, due to the special structure of the continuous-time Markov-chain together with a certain property of the

Poisson process called PASTA (Poisson Arrivals See Time Averages), simulations of continuous time Markov-chain models can be simplified and expedited so they lead to accurate results. To explain the PASTA property, consider a stochastic process for which steady-state probabilities exist. If we are interested in obtaining certain steady-state statistical characteristics of the process (like the πi in a continuous-time Markov-chain), we could inspect the entire evolution of the process (in practice, for a long enough time period), or we could use an independent Poisson inspector. (We already discussed the property of the Poisson process to see time- averages.) The PASTA principle means that if the arrivals follow a Poisson process, we do not need a separate Poisson inspector, but we could inspect the process at occurrences of points in time just before points of arrivals.

Note that in practice, since we are limited to a finite number of inspections, we should choose a Poisson process that will have sufficient number of occurrences (inspections) during the sim- ulation of the stochastic process we are interested in obtaining its steady-state statistics. In many cases, when we are interested in steady-state statistics of a continuous time Markov- chain, we can conveniently find a Poisson process which is part of the continuous-time Markov- chain we are interested in and use it as a Poisson inspector. For example, if we consider a queueing system in which the arrival process follows a Poisson process, such process could be used for times of arrivals of the inspector if it, at any inspection, does not count (include) its own particular arrival. In other words, we consider a Poisson inspector that arrives just before its own arrival occurrences.

See Chapter 4 for more information and examples on simulations.

2.5.9 Reversibility

We have discussed the time reversibility concept in the context of discrete-time Markov- chains. In the case of a continuous-time Markov-chain the notion of time reversibility is similar. If you observe the process Xt for a large t (to ensure stationarity) and if you cannot tell from its statistical behavior if it is going forward or backward, it is time reversible.

Consider stationary continuous-time Markov-chain that has a unique steady-state solution. Its [Pij] matrix characterizes a discrete-time Markov-chain. This discrete-time Markov-chain, called the embedded chain of our continuous-time Markov-chain, has [Pij] as its transition probability matrix. This embedded chain is in fact the sequence of states that our original continuous-time chain visits where we ignore the time spent in each state during each visit to that state. We already know the condition for time reversibility of the embedded chain, so consider our continuous-time chain and assume that it has been running for a long while, and consider its reversed process going backwards in time. In the following we show that also the reversed process spends an exponentially distributed amount of time in each state. Moreover, we will show that the reverse process spends an exponentially distributed amount of time with parameter δi when in state i which is equal to the time spent in state i by the original process. P{X(t) =i, for t∈[u−v, u]|X(u) =i} = P{X(t) = i, for t ∈[u−v, u]∩X(u) =i} P[X(u) = i] = P[X(u−v) =i]e −δiv P[X(u) = i] = e −δiv_.

The last equality is explained by reminding the reader that the process is in steady-state so the probability that the process is in stateiat time (u−v) is equal to the probability that the process is in state i at timeu.

Since the continuous-time Markov-chain is composed of two parts, its embedded chain and the time spent in each state, and since we have shown that the reversed process spends time in each state which is statistically the same as the original process, a condition for time reversibility of a continuous-time Markov-chain is that its embedded chain is time reversible.

As we have learned when we discussed reversibility of stationary discrete-time Markov-chains, the condition for reversibility is the existence of positive ˆπi for all states ithat sum up to unity that satisfy the detailed balance equations:

ˆ

πiPij = ˆπjPji for all adjacent i, j. (247)

Recall that this condition is necessary and sufficient for reversibility and that if such a solution exists, it is the stationary probability of the process. The equivalent condition in the case of a stationary continuous-time Markov-chain is the existence of positive πi for all states i that sum up to unity that satisfy the detailed balance equations of a continuous-time Markov-chain, defined as:

πiQij =πjQji for all adjacent i, j. (248)

Homework 2.25

Derive (248) from (247).

It is important to notice that for a birth-and-death process, its embedded chain is time- reversible. Consider a very long time L during that time, the number of transitions from stateito statei+ 1, denotedTi,i+1(L), is equal to the number of transitions, denotedTi+1,i(L), from state i+ 1 to i because every transition from i to i + 1 must eventually follow by a transition from i+ 1 to i. Actually, there may be a last transition from i to i+ 1 without the corresponding return from i+ 1 to i, but since we assume that L is arbitrarily large, the number of transitions is arbitrarily large and being off by one transition for an arbitrarily large number of transitions is negligible.

Therefore, for arbitrary largeL,

Ti,i+1(L)

L =

Ti+1,i(L)

L . (249)

Since for a birth-and-death processQij = 0 for|i−j |>1 and fori=j, and since for arbitrarily large L, we have πiQi,i+1 = Ti,i+1(L) L = Ti+1,i(L) L =πi+1Qi+1,i, (250)

so our birth-and-death process is time reversible. This is an important result for the present context because many of the queueing models discussed in this book involve birth-and-death processes.