Stochastic Processes
6.4 Simulation of stochastic processes
A number of important characteristics of stochastic processes require lengthy complex com-putations unless they are estimated by means of Monte Carlo methods. One may be inter-ested to explore the time it takes a process to attain a certain level, the time the process spends above some level or above another process, the probability that one process reaches a certain value ahead of another process, etc. Also, it is often important to predict future behavior of a stochastic process.
Discrete-time processes
Sample paths of discrete-time stochastic processes are usually simulated sequentially. We start with X(0), then simulate X(1) from the conditional distribution of X(1) given X(0),
PX1|X0(x1|x0) = PX0,X1(x0, x1)
PX0(x0) , (6.6)
then X(2) from the conditional distribution of X(2) given X(0) and X1, PX2|X0,X1(x2|x0, x1) = PX0,X1,X2(x0, x1, x2)
PX0,X1(x0, x1) , (6.7) etc. In the case of continuous-state (but still, discrete-time) processes, the probability mass functions PXj(xj) in (6.6) and (6.7) will be replaced with densities fXj(xj).
Markov chains
For Markov chains, all conditional distributions in (6.6) and (6.7) have a simple form given by one-step transition probabilities. Then, the simulation algorithm reduces to the following.
Algorithm 6.1 (Simulation of Markov chains)
1. Initialize: generate X(0) from the initial distribution P0(x).
2. Transition: having generated X(t) = i, generate X(t+1) as a discrete random variables that takes value j with probability pij. See Algorithm 5.1 on p. 106.
3. Return to step 2 until a sufficiently long sample path is generated.
Example 6.22 (Weather forecasts and water resources). In Example 6.8, the Markov chain of sunny and rainy days has initial distribution
( P0(sunny) = P0(1) = 0.2, P0(rainy) = P0(2) = 0.8,
and transition probability matrix P =
0.7 0.3 0.4 0.6
.
For simplicity, sunny days are denoted by X = 1, and rainy days by X = 2. The following MATLAB code will generate the forecast for the next 100 days.
N = 100; % length of sample path
X = zeros(N,1);
p = [0.2 0.8]; % initial distribution
P = [0.7 0.3; 0.4 0.6]; % transition probability matrix
U = rand(N,1); % N Uniform variables to start with
for t=1:N; % simulate X(1),...,X(N) sequentially
X(t) = 1*( U(t) <= p(1) ) + 2*( U(t) > p(1) );
% X(t)=1 with probability p(1) and
% X(t)=2 with probability 1-p(1)=p(2)
p = P(X(t),:); % prepare the distribution of X(t+1);
% its pmf is {X(t)}th row of matrix P end; X
This program returns a sequence of states that looks like this:
2211122212221121111111122222222112111222211111111112221211221...
Notice fairly long segments of sunny days (X = 1) and rainy days (X = 2), showing dependence among the generated variables. This is because a sunny day is more likely to be followed by another sunny day, and a rainy day is more likely to be followed by a rainy day.
Generating a large number of such sequences, we can estimate, say, the probability of 20 consecutive days without rain at least once during the next year, the average time between such droughts, the expected number of times this may happen during the same year, etc.
As an immediate application, based on this, water resources can be allocated accordingly.
♦
Binomial process
Simulation of a Binomial process is rather straightforward, as it is based on a sequence of independent Bernoulli trials. After simulating such a sequence, compute partial sums as follows.
N = 100; p = 0.4;
X = zeros(N,1); % initialization
Y = (rand(N,1) < p); % sequence of Bernoulli(p) variables X(1) = Y(1);
for t=2:N; % X(t) is the number of successes
X(t)=X(t-1)+Y(t); % in the first t trials end; X
It is a good illustration to look at the simulated stochastic process in real time. Animation of the generated discrete-time process can be created as follows.
plot(1,X(1),’o’); % Start the plot and allocate the box axis([0 N 0 max(X)]); % for the entire simulated segment
hold on; % Keep all the plotted points
for t=2:N;
plot(t,X(t),’o’); % Plot each point with a circle
pause(0.5); % A half-second pause after each point end; hold off
If you have access to MATLAB, try this code. You will see a real discrete-time process with half-second frames ∆.
Continuous-time processes
Simulation of continuous-time processes has a clear problem. The time t runs continuously through the time interval, taking infinitely many values in this range. However, we cannot store an infinite number of random variables in memory of our computer!
For most practical purposes, it suffices to generate a discrete-time process with a rather short frame ∆ (discretization). For example, a three-dimensional Brownian motion process, a continuous-time process with continuous sample paths and independent Normal increments, can be simulated by the following code.
N=5000; X=zeros(N,3); % Initialize X(t)
Z=randn(N,3); % N× 3 matrix of Normal increments
X(1,:)=Z(1,:);
for t=2:N; X(t,:)=X(t-1,:)+Z(t,:); end;
comet3(X(:,1),X(:,2),X(:,3)); % Animated 3D graph
The last command, comet3, creates a three-dimensional animation of the generated process.
Poisson process
Poisson processes can be generated without discretization. Indeed, although they are continuous-time, the value of X(t) can change only a finite number of times during each in-terval. The process changes every time a new “rare event” or arrival occurs, which happens a Poisson(λt) number of times during an interval of length t.
Then, it suffices to generate these moments of arrival. As we know from Section 6.3.2, the first arrival time is Exponential(λ), and each interarrival time is Exponential(λ) too. Then, a segment of a Poisson process during a time interval [0, M ] can be generated and “ani-mated” with the following MATLAB code.
M=1000; lambda=0.04;
S=0; T=0; % T is a growing vector of arrival times
while S<=M; % Loop ends when arrival time S exceeds M Y=-1/lambda * log(rand); % Exponential interarrival time
S=S+Y; % new arrival time
T=[T S]; % vector of arrival times extends
end; % by one element
N=length(T); % generated number of arrivals
X=zeros(M,1); % initialize the Poisson process
for t=1:M;
X(t)=sum(T<=t); % X(t) is the number of arrivals
end; % by the time t
comet(X); % Animation of the generated process!
Summary and conclusions
Stochastic processes are random variables that change, evolve, and develop in time. There are discrete- and continuous-time, discrete- and continuous-state processes, depending on their possible times and possible values.
Markov processes form an important class, where only the most recent value of the process is needed to predict its future probabilities. Then, a Markov chain is fully described by its initial distribution and transition probabilities. Its limiting behavior, after a large number of transitions, is determined by a steady-state distribution which we compute by solving a system of steady-state equations. Binomial and Poisson counting processes are both Markov, with discrete and continuous time, respectively.
Thus we developed an important tool for studying rather complex stochastic (involving uncertainty) systems. As long as the process is Markov, we compute its forecast, steady-state distribution, and other probabilities, expectations, and quantities of interest. Continuous-time processes can be viewed as limits of some discrete-Continuous-time processes, when the frame size reduces to zero.
In the next chapter, we use this approach to evaluate performance of queuing systems.
Exercises
6.1. A small computer lab has 2 terminals. The number of students working in this lab is recorded at the end of every hour. A computer assistant notices the following pattern:
If there are 0 or 1 students in a lab, then the number of students in 1 hour has a 50-50% chance to increase by 1 or remain unchanged.
If there are 2 students in a lab, then the number of students in 1 hour has a 50-50%
chance to decrease by 1 or remain unchanged.
(a) Write the transition probability matrix for this Markov chain.
(b) Is this a regular Markov chain? Justify your answer.
(c) Suppose there is nobody in the lab at 7 am. What is the probability of nobody working in the lab at 10 am?
6.2. A computer system can operate in two different modes. Every hour, it remains in the same mode or switches to a different mode according to the transition probability matrix
0.4 0.6 0.6 0.4
.
(a) Compute the 2-step transition probability matrix.
(b) If the system is in Mode I at 5:30 pm, what is the probability that it will be in Mode I at 8:30 pm on the same day?
6.3. Markov chains find direct applications in genetics. Here is an example.
An offspring of a black dog is black with probability 0.6 and brown with probability 0.4.
An offspring of a brown dog is black with probability 0.2 and brown with probability 0.8.
(a) Write the transition probability matrix of this Markov chain.
(b) Rex is a brown dog. Compute the probability that his grandchild is black.
6.4. Every day, Eric takes the same street from his home to the university. There are 4 street lights along his way, and Eric has noticed the following Markov dependence. If he sees a green light at an intersection, then 60% of time the next light is also green, and 40% of time the next light is red. However, if he sees a red light, then 70% of time the next light is also red, and 30% of time the next light is green.
(a) Construct the transition probability matrix for the street lights.
(b) If the first light is green, what is the probability that the third light is red?
(c) Eric’s classmate Jacob has many street lights between his home and the university. If the first street light is green, what is the probability that the last street light is red?
(Use the steady-state distribution.)
6.5. The pattern of sunny and rainy days on planet Rainbow is a homogeneous Markov chain with two states. Every sunny day is followed by another sunny day with probability 0.8. Every rainy day is followed by another rainy day with probability 0.6. Compute the probability that April 1 next year is rainy on Rainbow.
6.6. A computer device can be either in a busy mode (state 1) processing a task, or in an idle
mode (state 2), when there are no tasks to process. Being in a busy mode, it can finish a task and enter an idle mode any minute with the probability 0.2. Thus, with the probability 0.8 it stays another minute in a busy mode. Being in an idle mode, it receives a new task any minute with the probability 0.1 and enters a busy mode. Thus, it stays another minute in an idle mode with the probability 0.9. The initial state X0 is idle. Let Xn be the state of the device after n minutes.
(a) Find the distribution of X2.
(b) Find the steady-state distribution of Xn.
6.7. A Markov chain has the transition probability matrix
P =
0.3 ... 0 0 0 ...
1 ... ...
(a) Fill in the blanks.
(b) Show that this is a regular Markov chain.
(c) Compute the steady-state probabilities.
6.8. A Markov chain has 3 possible states: A, B, and C. Every hour, it makes a transition to a different state. From state A, transitions to states B and C are equally likely. From state B, transitions to states A and C are equally likely. From state C, it always makes a transition to state A. Find the steady-state distribution of states.
6.9. Tasks are sent to a supercomputer at an average rate of 6 tasks per minute. Their arrivals are modeled by a Binomial counting process with 2-second frames.
(a) Compute the probability of more than 2 tasks sent during 10 seconds.
(b) Compute the probability of more than 20 tasks sent during 100 seconds. You may use a suitable approximation.
6.10. The number of electronic messages received by an interactive message center is modeled by Binomial counting process with 15-second frames. The average arrival rate is 1 message per minute. Compute the probability of receiving more than 3 messages during a 2-minute interval.
6.11. Jobs are sent to a printer at the average rate of 2 jobs per minute. Binomial counting process is used to model these jobs.
(a) What frame length ∆ gives the probability 0.1 of an arrival during any given frame?
(b) With this value of ∆, compute the expectation and standard deviation for the number of jobs sent to the printer during a 1-hour period.
6.12. On the average, 2 airplanes per minute land at a certain international airport. We would like to model the number of landings by a Binomial counting process.
(a) What frame length should one use to guarantee that the probability of a landing during any frame does not exceed 0.1?
(b) Using the chosen frames, compute the probability of no landings during the next 5 minutes.
(c) Using the chosen frames, compute the probability of more than 100 landed airplanes during the next hour.
6.13. On the average, every 12 seconds a customer makes a call using a certain phone card. Calls are modeled by a Binomial counting process with 2-second frames. Find the mean and the variance for the time, in seconds, between two consecutive calls.
6.14. Customers of a certain internet service provider connect to the internet at the average rate of 3 customers per minute. Assuming Binomial counting process with 5-second frames, compute the probability of more than 10 new connections during the next 3 minutes. Com-pute the mean and the standard deviation of the number of seconds between connections.
6.15. Customers of an internet service provider connect to the internet at the average rate of 12 new connections per minute. Connections are modeled by a Binomial counting process.
(a) What frame length ∆ gives the probability 0.15 of an arrival during any given frame?
(b) With this value of ∆, compute the expectation and standard deviation for the number of seconds between two consecutive connections.
6.16. Messages arrive at a transmission center according to a Binomial counting process with 30 frames per minute. The average arrival rate is 40 messages per hour. Compute the mean and standard deviation of the number of messages arrived between 10 am and 10:30 am.
6.17. Messages arrive at an electronic message center at random times, with an average of 9 messages per hour.
(a) What is the probability of receiving at least five messages during the next hour?
(b) What is the probability of receiving exactly five messages during the next hour?
6.18. Messages arrive at an interactive message center according to a Binomial counting process with the average interarrival time of 15 seconds. Choosing a frame size of 5 seconds, compute the probability that during 200 minutes of operation, no more than 750 messages arrive.
6.19. Messages arrive at an electronic mail server at the average rate of 4 messages every 5 minutes. Their number is modeled by a Binomial counting process.
(a) What frame length makes the probability of a new message arrival during a given frame equal 0.05?
(b) Suppose that 50 messages arrived during some 1-hour period. Does this indicate that the arrival rate is on the increase? Use frames computed in (a).
6.20. Power outages are unexpected rare events occurring according to a Poisson process with the average rate of 3 outages per month. Compute the probability of more than 5 power outages during three summer months.
6.21. Telephone calls to a customer service center occur according to a Poisson process with the rate of 1 call every 3 minutes. Compute the probability of receiving more than 5 calls during the next 12 minutes.
6.22. Network blackouts occur at an average rate of 5 blackouts per month.
(a) Compute the probability of more than 3 blackouts during a given month.
(b) Each blackout costs$1500 for computer assistance and repair. Find the expectation and standard deviation of the monthly total cost due to blackouts.
6.23. An internet service provider offers special discounts to every third connecting customer.
Its customers connect to the internet according to a Poisson process with the rate of 5 customers per minute. Compute:
(a) the probability that no offer is made during the first 2 minutes (b) expectation and variance of the time of the first offer
6.24. On the average, Mr. Z drinks and drives once in 4 years. He knows that
Every time when he drinks and drives, he is caught by police.
According to the laws of his state, the third time when he is caught drinking and driving results in the loss of his driver’s license.
Poisson process is the correct model for such “rare events” as drinking and driving.
What is the probability that Mr. Z will keep his driver’s license for at least 10 years?
6.25. Refer to Example 6.9. Find the steady-state distribution for the number of users by writing X(t) as a sum of two independent Markov chains,
X(t) = Y1(t) + Y2(t),
where Yi(t) = 1 if user i is connected at time t and Yi(t) = 0 if user i is not connected, for i = 1, 2. Find the steady-state distribution for each Yi, then use it to find the steady-state distribution for X. Compare your result with Example 6.12.
6.26. (Computer mini-project) Generate 10,000 transitions of the Markov chain in Exam-ple 6.9 on p. 139. How many times do you find your generated Markov chain in each of its three states? Does it match the distribution found in Example 6.12?
6.27. (Computer mini-project) In this project, we explore the effect of Markov dependence.
Start with a sunny day and generate weather forecasts for the next 100 days according to the Markov chain in Example 6.7 on p. 136. Observe periods of sunny days and periods of rainy days.
Then consider another city where each day, sunny or rainy, is followed by a sunny day with probability (4/7) and by a rainy day with probability (3/7). As we know from Example 6.11 on p. 143, this is the steady-state distribution of sunny and rainy days, so the overall proportion of sunny days should be the same in both cities. Generate weather forecasts for 100 days and compare with the first city. What happened with periods of consecutive sunny and rainy days?
Each day in the first city, the weather depends on the previous day. In the second city, weather forecasts for different days are independent.
6.28. (Computer mini-project) Generate a 24-hour segment of a Poisson process of arrivals with the arrival rate λ = 2 hours−1. Graph its trajectory. Do you observe several arrivals in a short period of time followed by relatively long periods with no arrivals at all? Why are the arrivals so unevenly distributed over the 24-hour period?