CONCURRENT PROBABILISTIC PROGRAMS, OR:
HOW TO SCHEDULE IF YOU MUST*
SERGIU HARTt AND MICHA SHARIRTt
Abstract. Consider a finite set of processes, such that each one may use randomizations in its course of execution; these processes are running concurrently, under a fair interleaving schedule. We analyze the worstcase probability of termination, i.e., program convergence to a specified set of goal states. Several methods for computing this probability are presented, and characterizations of the special case where it is identically 1 are derived. Specializations of these characterizations to the case of deterministic and nondeter ministic programs, and t.o the case of programs with finite state spaces, are also discussed.
Key words. concurrent probabilistic program, scheduler, fairness, program termination, Markov chains
1. Introduction. This paper continues the study, begun in [HSP], of termination of concurrent probabilistic programs. The model that we assume is that of a finite set K of concurrent processes, each of which is allowed to use randomization, i.e., draw randomly according to probability di.stributions. These processes execute asyn chronously, and we can thus consider each process k E K as a discrete Markov chain (with stationary transition probability matrix pk) on the set I of common execution states. The overall execution behavior of these processes is described in terms of the interleaving pattern in which they are scheduled by some imaginary scheduler
^{fT.}Each process k scheduled at a state i can reach more than one subsequent state, so that to specify
^{fT}we need to consider all these transitions simultaneously. We may therefore represent
^{fT}as a tree (referred to as the executiontree or the transitiontree induced , by a) each of whose nodes is labeled by a pair (i, k), where i E I is the state reached at that node, "and where k E K is the process to be scheduled there next. A node (iI, kI) will be a son of (i, k) in the tree if there exists a positive transition probability of reaching i1 from i under ~ single execution step of process k, and if process k1 will be next scheduled at
ibprovided that this transition has indeed taken place.
Given such a
^{fT,}it induces in a standard manner a probability measure }.La on the space of all infinite sequences of states.
We consider here general schedules
fT, with the sole restriction that they be fair,meaning that no process stops being scheduled; i.e., that the }.Lameasureof the set of all tree paths on which each process k E K is scheduled infinitely often is t.
This model is discussed and justified more fully in [HSP]. We note that it coincides with the model assumed by Lehmann and Rabin in [LR], and also with that used by Dubins and Savage [DS] in their study of optimal gambling strategies (with the essential exception that they do not require fairness). It does differ, though, from various other models used in the literature (cf. [Rat], [Ra2], [RSt], [RS2]). The crucial distinction lies in the degree to which the imaginary scheduler can base its scheduling decisions on the outcome of random draws made by the processes, or, more generally, on their internal states. These more restrictive scheduling models usually correspond to situ ations in which the execution time of a single step of a process is independent of its current state and of the outcome of the random draws it has made. Our model is more general, and allows for such dependence, thereby being a more realistic model for
* Received by the editors September 15, 1982, and in revised form August 1, 1984.
t
School of Mathematical Sciences, Tel Aviv University, Ramat Aviv, 69978 Tel Aviv, Israel.tThe work of this author was supported in part by the BatSheva Fund at Tel Aviv University, and by the Office of Naval Research under grant NOOOI475CO571 at the Courant Institute.
991
L
general concurrent or distributed probabilistic execution. Moreover, properties estab lished for concurrent probabili~tic programs under our model will continue to hold under the more restrictive models mentioned above, but not necessarily viceversa (for example, Rabin's synchronization algorithm described in [Ral] is shown in [HSP] to fail in our model).
In the preceding paper [HSP], we have analyzed termination of concurrent prob abilistic programs having a finite state space. We have obtained there necessary and sufficient conditions for such a program to reach (with probability 1) a given set X of goal states from some initial state, under any fair schedule. These conditions can be checked mechanically, and are independent of the particular values of nonzero transi tion probabilities of the processes involved.
In this paper we generalize and extend these results to programs with infinite state spaces. As in the case of a single Markov chain, the analysis of program termination becomes much more complicated in the general case, and becomes dependent upon the actual values of the nonzero transition probabilities involved. The basic problem that we treat in this paper is the computation of the function ({)o'n the set of states I, where, for each i E I, ({)(i) is the minimum probability of program termination starting at state i, under any fair schedule. We establish various properties and characterizations of cp,and derive from them several techniques for the calculation of this function. This theory enables us to gain a better understanding of the structure of the (worstcase) convergence .of the program towards termination. For example, one can interpret this convergence process as a game between the program and the scheduler, in which each move of the program requires the scheduler to schedule one of the processes and the scheduler responds by scheduling this process eventually, but only after scheduling some other processes prior to it, in a way which would hurt as much as possible the
\ program's probability to terminate. We sQow that the optimal payoff for the program in this game is the function ({),provided that the game is long enough, where the length of such a game is measured by some (i~finite) ordinal.
The various characterizations of ({)are next used to obtain necessary and sufficient conditions for the special case ({)==1 (i.e. for worstcase almostsure termination from any initial state) to hold. Some of these conditions generalize similar conditions given in the preceding paper [HSP] for programs with finite state spaces. These characteriz ations of program termination are next sp.ecialized to the case in which the processes are deterministic or nondeterministic.
^{1}Some of these characterizations are shown to reduce to the conditions given by Lehmann, Pnueli and Stavi[LSP] for the termination of nondeterministic programs, while others are new. Finally, the special case of probabilistic programs with finite state spaces is reconsidered from the viewpoint of the general theory developed in this paper, enabling us to obtain the decomposition of the state space described in [HSP] in a different manner. The results of this paper are exemplified on several running example programs. The techniques developed in this paper can be immediately interpreted as (sound and complete) proof methods for almost sure program termination.
This paper is organized as follows: Section 2 presents the notations and terminology used in the paper, and begins the analysis of ({)by establishing some more elementary properties of this function. Section 3 develops the main technical tools for the analysis
1A nondeterministic program is one where each execution step of any of its processes may lead from a state i E I to several succeeding states, but where there is no probability distribution associated with these states; instead, each of these succeeding states must be considered as being potentially the sole successor of i. Such a program is said to terminate if every execution sequence terminat.es.
j
and characterizations of cp,and obtains cp as the limit of a certain transfinite sequence of functions. Section 4 gives further characterizations of cpoSection 5 treats the special case cp==1 (i.e. of almostsure worstcase termination), and derives various characteriz ations of this property. Section 6 specializes the preceding results to the case of deterministic and nondeterministic programs. The new characterization of termination of such programs is also given a direct proof. Section 7 treats the special case of probabilistic programs with finite state spaces. Some concluding remarks are presented in § 8.
2. Preliminaries. In this section we present our model of probabilistic concurrent programs in more precise terms, introduce some notations, and establish several preliminary properties of the worstcase termination probabil~ty of the program.
A concurrent probabilistic program consists of a finite set K of processes acting on a state space 1; each i E 1 is a common execution state of the processes, and is specified by the program location at each process, by the values of all variablesshared and privateetc. Each k E K can be regarded as a stationary discrete Markov chain on 1. (This extra restriction of discreteness, which is quite adequate for actual programs, simplifies the analysis considerably, by avoiding the technical difficulties of treating non O"additive measures, which would be otherwise necessary as in Dubins and Savage [DS].) Under this assumption, each process k E K is specified in terms of its transition probability matrix p\ that is, for each i,j E 1, P~j is the probability of reachi~g state j from state i in a single (indivisible) execution step of process k. The nonnegative
matrix pk is stochastic: for each i, P~j > 0 for at most countably many j, and
LjEl P~j= 1.
As already stated, program execution is assumed to consist of interleaving execu tion steps of the processes, each executing in its turn one indivisible step. Let i E 1 be an initial execution state. Let H (i) denote the set of all finite execution histories sta:r;ting at i; formally,
H(i)={i}X(Qo 1").
An (infinite) schedule 0" starting at i is simply defined as a function 0":H (i)
~K, that
is, for each finite history hE H(i), O"(h) is the next process to perform an execution step, given that execution has proceeded so far through the states in h. The set of all schedules starting at i will be denoted by ~ (i).' To each such schedule 0" there corresponds an execution tree, defined inductively as follows. Each node of this tree is labelled by a pair (j, k) where j is the current execution state, and k the next process to be scheduled in this node. The root of the tree is labelled by (i, O"(i)). For each node v in the tree, let hE H(i) be the sequence of states along the path from the root
to v, let j be the last state in h, and let k =
^{0"(}h ); then v is labelled by (j, k), and its
sons are nodes labelled by (j', 0"(h, j')), (where (h, j') is the concatenation of j' to h) for j' E 1 such that pfj' > O.Let H*(i) denote the set of all infinite execution histories starting at i, that is, H*(i) = {I} x J':ta (where 100
=
Jt
^{1).}Each schedule 0" E ~ (i) induces a probability measure ILcr on the cylindrical 0"field
on H*(i), such that for each cylinder (i')1> i2, . . . , in), consisting of all histories whose initial n + 1 states are i, i1>. . . , in,
n]
ILcr{(i, i1>. . . , in)} = IT P~~is+1' s=o
where io= i, ks = 0"(io, i], . . . , is). Expectation with respect to ILcrwill be denoted by EfT"
Let H*
= U
iEl H*(i). Throughout the paper we will use the following notational convention: Elements of H*which we call paths or historieswill be denoted by 7T; for each such 7Tand each n ~ 0, the (n + 1)th state along ^{7T} will be denoted by in,and the subpath consisting of the first n + 1 states in
^{7T}will be denoted by
^{7Tn}==
00, it. . . . , il1)' A path ^{7T} is a fair path with respect to a given schedule U if each k E K appears infinitely often in the sequence {u( 7Tn)}~=O;the schedule U is a fair schedule if fLu {7T: 7T is fair}
= 1. For each i E I we denote
LF(i) =
{u EL (i):
Uis fair}.
Let X
cI be a given set of goal states, fixed henceforth, Our aim is to study the convergence of program execution to states in X; we will therefore assume in the sequel, without loss of generality, that all states in X are absorbing for each k E K;
i.e., that P~j = 1 for each i E X and each k E K.
The basic problem studied in this paper is that of analyzing and computing the worstcase probability of the program to reach X (Le., to terminate) when executed from a given initial state under a fair schedule. To formalize this notion, let Xx 'be the characteristic function of X (defined on 1); we extend this function to ]{* by putting Xx(7T)=limn>ooXx(il1) (recall that 7T=(iI1)n~O)' Since X is absorbing, XX(7T) = 1 if X is ever reached along
^{7T,}and 0 otherwise. The probability of reaching X under U is then simply Bu(Xx). The following standard observation, which also establishes the measurability of the extended Xx, will be quite useful in the sequel: For each n ~ 0
define a "truncated" extension X~) of Xx by putting X~)( 7T)
= 1, if X is reached during the first n steps of
^{7T,}and 0 otherwise. Then Bu(X~») is the probability of reaching X during the first n steps of u, and we h~ave
lim B()"(X~»)= sup Bu(X~») = Eu(Xx).
n>oo 11
The worstcase termination probability that we seek is defined, for each initial state i E I, as
cp(i) = inf Bu(Xx).
uE:I.F(f) ,
We will shortly establish several preliminary properties of the function cp,but first we introduce additional notations concerning finite portions of program execution.
Let N denote the set of nonnegative integers, and out N= N U{oo}. A stopping time N is a mapping from H* into N
such that if N( 7T)= m then N( 7T') = m for each path
^{7T'}which coincides with ^{7T}
at all steps up to, and including m. In other words, N( 7T)may
depend only on io, it,. . . , iN~i.e., on states visited before this step, but not on future steps (i.e., on iN+b . . .). A finite subschedule at i E I is a pair T ~ (c:,.,N) where U E L (i), and where N is a stopping time on H*( i) such that fLu( N < 00) = 1 (this corresponds to the notion of "policy" of Dubins and Savage [DS]). The intuitive meaning of such a pair is the initial portion of U up to, and including N; in particular, the actual value
of
^{(J"}is relevant only up to the stopping time N. The set of all finite subschedules at i
will be denoted by T( i) (note that the empty subscheduleLe., when N ==ais included in T(i)).
In the sequel we will occasionally use the following standard decomposition of an infinite schedule (J"EL(i): Let N be a stopping time, fLu(N<oo) = 1; then (J"is equivalent to its initial portion T = ((J",N), followed by the collection of continuation schedules; that is, for each
7TE H*(i) (with N<
00), the continuation U7TNE L (iN) of if after the end state iN of T. Note in particular that ^{(J"}is fair if and only if each of the
continuations (J"7TNis fair.
Let a be a real function on 1. Then for each finite subschedule ^{T}
= (CT,N)
E T( i), the expectation of a with respect to T is defined asB,(a) = Eer(aON))'
For example, let CTE I 0), and define, for each n:> 0, Tn = (CT,n) E TO) (i.e., with the stopping time N=n). Then, as already noted, Eer(Xx) =limn>eo ET,,(Xx). Note also that ETo(a) = a (i), and that ETI(a) = (pka )0), where k = CT(i).
Having introduced all the required terminology, we begin by establishing a few elementary properties of the function <po
PROPOSITION 2.1. (a) <p~ 0; <pIx==1.
(b) <p(i)=minkEK (pk<p)(i) for each iE1.
Proof (a) is trivial, since X is absorbing; note also that <p::::1.
(b) To show that <p(i)~(pk<p)(i) for kE K, iE I, use a schedule CTEIF(i) which starts by scheduling k at i, and then continues so as to approximate <pat each of the resulting states. For the converse inequality, take a sequence of schedules CTnin "2.F(i) such that Eern(Xx) converges to <p(i) and such that they all start by scheduling the same process k E K (since K is finite this is always possible); then it is easily seen that (pk<p)(i)~<p(i). More details can be found in [HS]. Q.E.D.
Extending standard notations in Markov chain theory, we say that a real function a on I is subharmonic if a ~ p\:r for each k E K. Similarly a will be called minharmonic if a
=
minkEK pka (note that each minharmonic function is subharmonic).In the special case where K contains a single process k, the function <pis harmonic (i.e., <p= pk<p). Moreover, it is wellknown (cL [SPH] for example) that <pis the smallest nonnegative harmonic function which is 1 on X. This might lead us to conjecture that for a general (finite) K, <pis also the smallest nonnegative minharmonic function which is 1 on X. This, however, is not true in general, as can be seen from the following simple example: Let 1= {O, 1}, X = {a}, and K = {1, 2}, with the nonzero transition probabilities p~,o
=
^{pi,l}=
1. Obviously, any fair execution of this program brings it into X with certainty, so that <p==1, yet the function 1/1(0)= 1, 1/1(1)= 0 is a smaller
nonnegative minharmonic function which is 1 on X. The reason for this phenomenon is that fairness is not directly connected to the minharmonicity of <poIndeed, let usdefine a function
^{1/1}on I by
I/I(i) =
inf Eu(Xx),UE~ (i) i E 1.
(i.e., infimum over all schedules, not necessarily fair). Then it can be shown that PROPOSITION
2.2.
^{1/1}is the smallest nonnegative minharmonic function which is 1
on X.
Our next result is a strong form of a "zeroone law" for <p,which generalizes the zeroone law established in [HSP] for finite. state spaces.
THEOREM 2.3 (zeroone law). inCE! <pO) is either 0 or 1. Moreover, for each i E I and CTEIF(i) define a sequence {f,.}n~o offunctions on H*(i) by puttingf,.(1T) = <p(in), 1TE H*(i), n ~ O. Then {in} converges /Lera.s. to Xx (extended to H*(i)).
Proof Let i E I and CTE IF( i) be given. The subharmonicity of <pimplies that the sequence {In} is a submartingale2 on H*(i). Since O<fn < 1 for each n ~ 0, it follows from the (sub )martingale convergence theorem that {In} converges /Lua.s. to a limit fee. Put Tn = (CT,n), n~ O. Then
Eer(Xx) = ETn(Eu,," (Xx)):> ET"«p) = Eu(fn)
2i.e., for all n ~ 0, ECT(fn+ll'ITn)~fn, where 'lTnis any history of length n with J..L(T('ITn) > O.
(since each unn is fair). Letting n ~ 00, we obtain
E<T(Xx) ~ Eo(/oo) ~ 1
.
,uo{ 1T: /00 ( 7T) = I}.But for each 'lTEH(i), if X is ever reached along 'IT then/n('IT)=cp(in)=1 for all sufficiently large n, so that foo( 'IT)= 1. Thus
,uo{ 'IT: /oo( 'IT) = 1} ~ /L<T{1T: X reached along 'IT}
=
Eo(Xx).Therefore we must have equalities throughout; that is
Eo(Xx) = Eo(/oo) = ,uo{ 'IT: /00 ( 1T), = 1}.
This, however, implies that foo is almost everywhere either 0 or 1, and that /oo('IT)= 1 if and only if X is ever reached along
'IT.The zeroone law is now immediate, because if ({lis not identically 1, take iEI, UE2.F(i) such that Eo(Xx)=c<1. Then cp(in)~O on a set of paths whose /Lomeasure is 1 c> 0, thus there exists states with arbitrarily small ({l,or infiEI ({l(i) = O. Q.E.D.
As a final preliminary note, we would like to point out that, unlike the case of a finite state space, the actual values of nonzero transition probabilities of the processes involved can have significant influence on the termination probabilities cpoThis is indeed well known even for a single Markov chain. (Consider e.g. the case of a random walk on the nonnegative integers, where the "leftward" transition probability is p.
Then the probability of converging towards 0 is identically 1 if p ;;::
1,and is exponentially decreasing otherwise; cr. [Ch] for details). Thus, for infinite state spaces there is no hope to obtain purely combinatorial analysis techniques (as have been developed in [HSP] for finite state spaces), and more complex techniques are needed. Development of such techniques is indeed the main purpose of the present paper.
3. 'P.iterates. Direct calculation of the function 'P from its definition is rather complicated. The purpose of this section is to develop machinery needed for a simpler calculation and characterization of 'P. Specifically, we will show that 'P is the limit of a transfinite sequence of iterates of a certain operator. We will call these cpiterates.
DEFINIT1ON. We define an 9perator Q, and an auxiliary set of operators {QkheK, on the space of all bounded real functions on I, as follows: For each bounded real function a on 1, each i E1, and each k E K, put
(Qka)(i)= inf E.r(a),
TET(i,k)
where T( i, k) is defined as
{(O",N+1): O"E2. (i), Nan a.s. finite stopping time with U(1TN)=k}c T(i);
i.e., T( i, k) is the set of all subschedules which start at i, schedule k eventually almost surely, and stop right after scheduling k. Q is then defined as
(Qa )(i) = max (Q~a)(i).
kEK
Let R be any of the operators Qk or Q; then plainly R is monotone (Le., at ~ a2
implies Ra)::: Raz), RO
= 0, and R1 = 1. The following lemma gives two characteriz
ations of the operators Qk, one of which is constructive while the other is not.LEMMA 3.1. Let a be a bounded real function on 1. For each k E K, Qka is the largest subharmonic function which does not exceed pka (i.'e., (1) Qka ~ pka, (2) Qka is subharmonic, and (3) if f3 ~ pka is subharmonic, then f3 ::::Qka). Furthermore, Qka
is the limit. (or infimum) of the following nonincreasing sequence of functions:
/31(i)
= (pka)(i), iEI,
/3n+1(i) = min
{
/3n(i), min (pI/3n)( i)} ,
IEK i E I, n2:1.
Proof. Let /3 be the limit of the nonincreasing sequence {/3n}; then /3 is the largest sub harmonic function ~pka. The rest follows by noting that, for each n 2: 1
/3n(i) = inf Er(a),
TE Tn (i,k)
where Tn(i, k) consists of those subschedules in T( i, k) which stop after at most n steps. Q.E.D.
LEMMA 3.2. For each subharmonic function a and each k E K we have a~Qka~pka.
Proof. By Lemma 3.1, Qka is the 1<irgest sub harmonic function which is <.pka.
Since a itself is subharmonic w~ have a ~ pka, so that a ^{<.}
Qka. Q.E.D.
DEFINITION.For each ordinal a we define on I real functions
'Yaand
'Y~,k
EK, by the following transfinite inductive process:
k
'Yo
=
^{'Yo}= Xx,kEK,
'Y~= sup Qk'Yb for each ordinal a> 0 and k E K,
b<a
'Ya
= max 'Y~ for each ordinal a.
kEK
,j'
The functions 'Y~and 'Ya are called the cpiterates of order a of the program (the reason for this terminology will be apparent at the end of this section).
Since X is absorbing, QkXx ~ Xx, thus 'Y~::> 'Y~ for each k E K, hence 'Y1::>'Yo. Also,
by definition, 'Y~~ 'Y~for each pair of ordinals a> b > O. Thus, for each k E K the
transfinite sequence {'Y~}^{a~O} is nondecreasing, and so is the sequence {'Ya}a~O' From this it follows that _{'Y~+1}=
Qk'Ya for each ordinal a, and that 'Ya= SUPb<a'Yb for limitordinals a.
Since the transfinite sequence {'Ya}a~Ois nondecreasing, and each of its elements is obviously bounded between 0 and 1, this sequence must converge to a limit function 'Y,and there must exist an ordinal c such that'Yc = 'Y. (Indeed, for each i E I the transfinite sequence {{'a(i)} is a nondecreasing and bounded sequence of real numbers, and so must attain its supremum at some ordinal Ci; the required ordinal c is simply
SUPiEl Ci') Obviously Q'Y
=
^{Q'Yc}=
^{'Yc+l}=
'Yc= 'Y.Moreover, using standard fixpoint arguments, it is easily seen by transfinite induction that y is the smallest fixpoint of Q which is 2:Xx.
Remarks. (1) To motivate these definitions, it is helpful to consider the following interpretation of the functions 'Ya and 'Y~: 'Yo(i) is just an indication whether i E X.
'Y~(i) is the smallest probability of reaching X by any subschedule which starts execution at i, and is forced to schedule k eventually (a.s.). Thus 'Yl(i) is the smallest probability of reaching X that must be yielded by any subschedule starting at i which is forced to schedule anyone of the processes at least once. Arguing inductively, 'Yn(i) is the smallest probability of reaching X that must be yielded by any subschedule starting at i which has to schedule any sequence of n processes one after the other.
(N ote that this sequence need not be specified in advance; rather the first process k1
,.'","to be scheduled is specified, then the second process to be scheduled is specified, but it may' depend on the state reached after scheduling k" and so on.)
(2)
_{'Yn}(i) can be viewed as the minmax value of a twoperson zerosum game
I'n
(i). In this game, the aim of the first player, called "player X," is to reach X during program execution with the highest possible probability, whereas the aim of the second player, called "the scheduler," is to prevent the program from reaching X as much as possible. The game I'
^{n}(i) consists of n stages. Each stage starts at some state j E I (stage 1 starts at i). Player X chooses some k E K, and then the scheduler chooses some T E T(j, k). The program is then run according to T; when it stops, the next stage is played. After n such stages, player X receives a payoff of one unit from the scheduler if a state in X has been reached, and zero otherwise.
This interpretation can be extended to higherorder ordinals. Specifically, for each ordinal a we define a collection of games f a(i), for each i E I, in the following transfinite
inductive manner:
.
(i) I' o(i) is the "empty" game; player X receives a payoff of 1 from the scheduler if i E X, and zero otherwise.
(ii) If a is not a limit ordinal, say a = b + 1, player X' first chooses a process k and then the scheduler chooses a subschedule T E T(i, k), and the program is run according to T; for each end state j of T, the game continu~s as fb(j).
(iii) If a is a limit ordinal, player X first chooses an ordinal b < a, and then the game continues as fb(i).
The definitions (ii) and (iii) imply that after each stage, games with smaller ordinals are played; since every strictly decreasing sequence of ordinals is finite, it follows that every play of any of these games is finite, so that fo is reached eventually, and the payoff is therefore well defined. Moreover, by the definition of the sequence
^{{ 'Ya}}^{a;;;;O,}one easily obtains by transfinite induction, that 'Ya(i) is precisely the value of f aU).
Indeed, an Boptimal strategy for.player X is constructed as follows (for each
^{B}> 0):
If a = b + 1, player X first chopses k E K for which 'Ya(i) = 'Y~(i), and from each end state j of the subschedule
^{T E}T( i, k) subsequently chosen by the scheduler, he continues with an Boptimal strategy of fj (b). If a is a limit ordinal, player X first chooses an ordinal b < a such that 'Ya(i)  'Yb(i)< B/2, and then continues with an el2optimal strategy of f
b( i)..As for the scheduler, at the first ordinal b + 1~ a where he is called upon to move, he chooses T E T(i, k) such that Er( 'Yb)  8/2 < (Qk'Yb)(i) = 'Y~+I(i), and then he continues with an B/2optimal strategy in the corresponding f b(j).
Furthermore, the ordinal c (at which 'Yc+l= 'Ycis fi1:stobtained) is such that the expected payoff that player X can guarantee in the game f c(i) is the largest possible among all games {fa (in a""O~uniformly in the initial state i. As we shall see later in this section, this maximum payoff is exactly q>(i).
(3) Note that if Q were (Torder continuous, Le., if for any nondecreasing sequence {l1n} of uniformly bounded functions we had
Q( S~Phl1) = s~~ Qhm
then convergence of the 'Ya's would be attained at c = w (the first infinite ordinal) or earlier. This is indeed so when I is a finite set, since then each such sequence {hl1}
converges uniformly to its supremum, in which case Q is clearly continuous. However,
this does not hold in general, and so higher ordinals may be needed. (A similar
phenomenon is noted by Lehmann, Pnueli and Stavi [LPS] concerning nondeterministic
concurrent programs; see § 6 for a detailed comparison between their technique and
ours.)
To illustrate the possible discontinuity of Q (and hence the need for higher ordinals), consider the following example (in which both processes involved are actually deterministic) .
Example 1. Let K = {I, 2}, and let I = II U I2, where II = N x {I}, 12 = N x{2}, and X = {CO,I)}. The nonzero transition probabilities are
pI(n,I),(nl,l)
=
^{p2}(n,I),(n,1)= 1
, n>O,pI(n,2),(n+l,2)

 ^{p2}(n,2),(n,1) 1  , n:>O.
These transitions are displayed in the following diagram:
1,:
I, :
2 2 2
It is easily seen that
'Yn(i,l)= { O, 1,
i:> n,
i < n, iEN.
By definition of 1'", we thus have
'Y",(i, 1) = 1, iEN.
On the other hand, 'Yn(i,2)=0 for each i, nEN (to obtain (Q2Yn)(i,2), schedule process 1 sufficiently many!imes so as to reach a state (j,2) with j 2: n, and then schedule process 2). Thus
'Y",(i, 2) = 0, iEN.
But 1'",+1= Q'Y",
>
y",. Indeed, for each (i, 2) E 12 we haveY"'+I(i, 2) = (Q'Y",)(i, 2) = (Q2y",)(i, 2)= Yw(j, 1) = 1 (where
j:>i).
Thus 1""+1==1, and convergence of the qJiterates is attained at the ordinal w + 1.
Remarks. (1) In the gametheoretic terminology established earlier, player X cannot achieve a nonzero payoff in any of the games r n(i, 2), n EN, or even r ",(i, 2), because if the number of rounds n is fixed in advance, the scheduler will initially schedule process 1, n + 1 times, and this will prevent player X from reaching Xin n moves. On the other hand, a payoff of 1 is guaranteed in r w+l0, 2) as follows: Player X first chooses process 2; no matter what subschedule in T((i, 2), 2) will be chosen by the scheduler, it will end at some state (j, 1) in II. and the game continues from
there as r
'"
(j, 1). Now player X chooses the ordinal j < w, and this guarantees its entry
into X after j additional moves, by requiring to schedule process 1 in each of these
moves.
(2) One can easily obtain along similar lines examples where higher and higher ordinals are needed to attain convergence.
(3) If we take in Example 1, pko,(o,O =! (instead of 1) and pkl)(0,2)
=! (instead of 0), it can be verified that the first ordinal c where Yc= cP==1 is c = CU2.
The main purpose of this section is to prove that Y
=
^{cpo}The proof of this assertion
is quite involved and will be split into proving both inequalities Y ~ cp and Y;;:;cpoIt consists of the following sequence of .lemmata. .LEMMA3.3. Each of the cpiterates Y~ and Ya is subharmonic.
Proof Lemma 3.1 and the fact that the supremum of subharmonic functions is subharmonic. Q.E.D.
LEMMA3.4. For each k Ek, Qkcp~ cpo
Proof For each (J"E 1,F( i) let N be the first time k is scheduled, and let T =
((J",N + 1) E T( i, k). Then, if (J" 7TN+l denotes the continuation of (J"after the end of T, we have
E(T(Xx) = Er[E(T" (Xx)] (because N + 1 is a stopping time)
N+l
;;:;Er[CP(iN+l)] (because (J"7TN+1E 1,F(iN+l))
;;:;(Qkcp)(i) (by definition of Qk).
Since this holds for each (J"E !.F(i), we have cp(i)?; (Qkcp)( i). Q.E.D.
PROPOSITION 3.5. cp
=
^{Qcp}=
^{QKcp, for} ^{each} ^{k E K.}Proof By the preceding lemma, Qcp = maXkeK Qkcp ~ cp: On the other hand, for each k E K, Qkcp?; cp by Lemma 3.2, since cp is subharmonic by Proposition 2.1(b). Q.E.D.
LEMMA 3.6. Y ~ cpo
Proof We will show, using transfinite induction, that Ya;2 cp for each ordinal a.
For a = 0,
cp;;:; Xx= Yo (see Proposition 2.1 (a)). Assume Yb ~ cp for each
b<
a;then
QkYb ~ Qkcp
= cp by the preceding proposition, thus oY~~ cp for each k E K, so that
Ya ~. cpoThus Y= Yc~ cpo Q.E.D.
LEMMA 3.7. cp~ y.
Proof Note that, since Y =
maXkeK Qky, we have(*)
^{y(i)?;} ^{re TO,k)}^{inf} ^{Er( y),} ^{i Eo/,}kEK,
(actually, with equality holding for at least one k, although we will not make use of this fact), Let iE/ be given. Choose 8> 0 and a sequence 8nto such that Ln 8n = 8.
Let {kn}ni5;lbe a fixed sequence of processes in whi'ch each k E K appears infinitely many times. We will use (*) to construct a fair schedule (J"starting at i by building it layerbylayer from subschedules, as follows: Suppose that the first n layers of (J"have already been constructed, the union of which being some subschedule
Tnstarting at i (initially,
^{TO} is "empty"). The (n + l)th layer of (J"is defined by appending to ^{Tn}at each of its end nodes j a subschedule Pj E T(J, kn+l) such that
y(j) ;;:;Ep/ y) ~
8n(such a subschedule exists by (*)). Repeating this process inductively, we obtain the required (infinite) schedule (J",which is fair by our choice of the sequence {kn}ni5;l'
Let {N"},,i5;o be the increasing sequence of stopping times defined by our construc tion; namelythe nth layer (i.e., T,J ends at N" (in particular No ==0). For each n?; 0 define the function
gl1(7T)=Y(7TNJ, 7T E H*(i) ;
in particular, go
=
'YO). By the choice of the subschedules Pjwe have
(**)
gn ~ Eu(gn+111TNJ  En+1> n>O.Hence, the sequence of functions {g~}n~O given by
1
n"
g n = gn  L. Em,
m=1
n;?:O
forms a supermartingale, which is bounded between 1 and E. Hence it converges almost surely to a limit g:x" so that {gn} converges almost surely to the function
ee
gee
=
^{g:x,+}I
Em = g:x,+ E.m=1
Note that 'Ylx ==1; thus, if X is reached along 1T,then gee(1T)= 1, because for all sufficiently large n we will have gn (1T) = 1. Hence, by (**),
'Y(i) = gb~ Eu(g:x,) = Eu(gee)  e
~ J.Lu(gee= 1)  e
;?:J.Lu(X is reached)  E
= Eu(Xx)
 e::: <p(i)  E.Since E was arbitrary, the proof is complete. Q.E.D.
Thus we have shown THEOREM 3.8. ^{<p}
=
'Y.Next, we give an example of explicit calculation of <pas the limit of the <piterates.
Example 2. Let 1= N, X = {O},K = {1, 2} such that each process is a random walk on I (with X absorbing). It turns out that a fair interaction of two random walks, under the worst kind of schedules, yields essentially the same absorption probabilities as those yielded by the "worse" of the two walks alone. We exhibit here one simple case:
P
^{I} ^{_1}. i,i1 3, ^{pI} _{' +1} = 32
1,1 , i;?:l,
ptI = 1, i;?:1.
It can be inductively shown that the <piterates for this program are (
^{.})
{
I;nJ /;n,
'YnI l  0,O<i<n,
i> n,
n:::1, where 1;;=2i_1, i~O, and
'Y",(i)= 'Y"'+1(i)= (j)(i) = 2i, 1 i>O.
, ,y.
Comparison with the iterates 1n and their limit 1", for the case in which only process 1 is activated shows that /"" = 1", but /'n > 1n for each finite n. Thus the fair interleaving of process 2 with process 1 increases the probability of convergence under any finite number of fairness constraints, but does not affect the overall (worstcase) convergence probability.
4. Characterizations of 'P. This section contains the main results of the paper,
Using the machinery developed in §§ 2 ~nd 3, we will derive several characterizations
of (j),which provide a variety of rather simple techniques for its calculation, or for
deriving various properties of this function. Obviously, the most important such
property is whether cp==1 (i.e., whether the program terminates almost surely from any initial state). Relaxation of the characterizations of cp given here will enable us to derive necessary and sufficient conditions for program termination, and these conditions are presented in § 5.
THEOREM 4.1. (a) cp is the smallest fixpoint of the equation cp
=
Qcpwhich
is ~Xx.
(b)
cpis the smallest simultaneous solution of the equations
cp= Qkcp for each k e K,
which is ~Xx.
Proof By Propositions 3.5 and 2.1 (a), cp =
^{Qcp}= Qkcp for each k e K, and
cp ~ Xx.To prove (a) we repeat the argument used in the proof of Lemma 3.6. That is, let t/J~ Xx be such that t/J =
^{Qt/J.}Then
t/JE;;'Yo, thus t/J= ^{Qt/J~Q'Yo} =
^{'Yt,}and by transfinite
induction t/J ~ 'Ya
for each ordinal a, thus
_{t/J ~ 'Y}=
cpoAs for (b), note that
t/J=
^{Qkt/J}^{for}
all k E K implies t/J
= Qt/J,and then use (a). Q.E.D.
Next we restate the second assertion of Theorem 4.1 in a manner which makes it more convenient for actual calculation of cpo
DEFINITION. Let a be a real function on 1. We say that a has property (A) if the
following are satisfied:
^{.}(AI) alx==l;
(A2) a is subharmonic;
(A3) for each k E K the only subharmonic function lying between a and pka is
a itself. ^{.}
(Note that the constant function 1 has property (A).)
THEOREM4.2.cp is the smallest nonnegative function on I having property (A) (i.e., ifa~O satisfies (A), then a(i)E;; cp(i) for each ieI).
Proof. By Lemmata 3.1 and 3.2, (A.2) and (A.3) imply a = Qka for all k E K, or
a= Qa. We now use Theorem 4.1. Q.E.D.
Theorem 4.2 suggests the following procedure for computing cp:Take any nonnega tive subharmonic function a ~ Xx. For each k E K compute the largest subharmonic function which is ~ pka, and require that it coincide with a. Find the general solution of these constraints, and obtain.cp as the smallest such solution. Later on in this section We will use this procedure to compute cp for several exemplary programs, and show that this technique is quite feasible in practice.
Put
(A2') a is minharmonic,
and let property (A') be defined as the conjunction of (AI), (A2') and (A.3). Then we also have
COROLLARY4.3. cpis the smallest nonnegative function having property (A').
Proof Immediate, since cpitself is minharmonic, by Proposition 2.I(b), and every 'minharmonic function is also subharmonic. Q.E.D.
Remark. In carrying out the calculations of the. procedure just outlined, it may sometimes be more convenient to employ the "Icomplement" version of Theorem
4.2; that is, instead of computing cp we compute the function t/J==1
 cp,which is then the largest function 31 which is a fixpoint of the equation
t/J(i)
=
min supEA t/J)keK TeT(i.k)
or, alternatively, is the largest function f3 s; 1 having property (B), defined as (B.l) f3lx ==0;
(B.2) {3 is superharmonic, i.e., {3:::::;pkf3 for each k E K;
(B.3) for each k E K, the only superharmonic function between pk{3 and {3 is f3 itself.
(Again, we can replace (B.2) by (B.2'), namely require that f3 be maxharmonic, that is f3 = maxkE~ pkf3.)
The usefulness of this complementation lies in the fact that property (B) is positively homogeneous (i.e., f3 satisfies (B) implies ".8, satisfies (B) for every A> 0, where (".8)(i)==" '/30)); note that (A) was not such (due to (A.1)). For example, we 0btain
COROLLARY 4.4.cp ==1
if and only if no bounded function having some positive
entries has property (B).
^{}Proof Assume
.8satisfies (B) so that
" ==SUPiEI.8(i) < co and is positive. Then
(1/ ")f3 also satisfies (B) and is ~1. Q.E.D.We can also give now a second short proof of the ZeroOne Law for cp; namely, that inf;EI cpO) is either 0 or 1 (see Theorem 2.3; however, the original proof is more
elementary).
^{,}Second proof of the zeroone law (Theorem 2.3). Let I/J
= 1
cp and put"
= SUPiEII/J(i). If 0< "< 1, then the function t/J'= (1/ ")I/Jis larg~r than I/J,satisfies (B), and is ~ Icontradicting the fact that I/Jis the largest such function.
Examples. We will now apply the techniques presented in this section to several programs, to compute the function cp for each of these programs. These examples include two programs with finite state spaces (whicn had already been analyzed in a preceding paper [HSP] by different sp'ecial techniques developed there for finitestate programs), and another program having an infinite state space.
Example 3. Let K = {I, 2}. The following program arises in an analysis of freedom from lockout in a simple synchronization protocol (cf. [HSP, Example 1] for details).
Using a notation slightly different from that of [HSP], we have I = X U {ib i2, i3, i4},
with nonzero transition probabilities 
P}hX =P7hil = 1,
P
^{l.'}i2,it= P^{'l}i2.i4= P^{2}i2,i1= P^{2}i2,i4= 2,^{I}P}3,i3 = P73,i2 = 1,
P}4,i4= Pt,i3 = 1.
To compute cp,we first write down the form of the general sub harmonic function which is 1 onX Such a function a= (ab a2, a3, a4) (where at is a shorthand for a(it), 1 ::: t ~ 4) must satisfy
al ~
1,
<1
+
^{1}a2 =2al 2a4,
,
"'
a3
:::
a2,a4 ~ a3'
Next, we spell out condition (A.3) for such an a: First consider k = 1. It is easily checked that the function
pIa = (1,
!al +!a4, a3, a4)pi p2
11 12 13 14 Is 16 X i 1 i2 i3 i4 IS i6 X
II I 1 1 1
'2 '2 '2 '2
12 1 1 1 1
'2 '2 2: 2:
13 1 I 1 1
:2 2: 2: _{'2}
14 1 1 1 1
'2 '2 '2 '2
Is 1 1
2: :2 1
16 1 1
'2 '2 1
is also subharmonic. Hence we must have a ^{:;:::}
pi a, i.e.,
0'1:;:::1; ^{a2:;:::} '2al^{I}
+
'2a4:;:::^{1} ^{1}+
^{1}'2 '2a4'
Similarly, for k:;:::2 we have
p2a:;::: (at, ~al +~a4' a2, a3), which is also seen to be subharmonic. H~nce a:;:::p2a, i.e.,
a2 :;:::a3, a3 :;:::a4'
Thus we have
al :;::: a2:;::: a3:;::: a4:;:::1. That is, the onlyand thus, the smallestfunction s<\tisfying (A) is 'P==1.
Example 4. This example is also taken from CHSP], and arises in the analysis of
another synchronization protocol. Here K:;::: {I, 2}, I:;:::X U {ib
.. .
, i6}, and the transition probability matrices are
It is straightforward to check that a general subharmonic function a:;:::(ab . . . , a6) which is 1 on X must satisfy
al :;::: a2:;:::
. . .
^{:;:::}_{a6}_{~ 1.}
It now follows that (A.3) holds for each such function a, because any function constant on I  X and lying between a and pi a (resp. p2a) must coincide with a
(since pia (resp. p2a) coincides with a at some of these states). Thus
_{'P,}which is the
smallest nonnegative such function, is Xx. '
Example 5 ("The Two Combs"). I.::et K:;::: {1, 2}, I:;:::X
U Z (where Z denotes the
set of signed integers); the nonzero transition probabilities areP:"n+l :;:::
Pm P~,nI :;:::qm
P~,x :;:::p~ :;:::1
 p",
p~,x :;:::q~ :;:::1  qn,
nEZ.
To avoid degeneracy, we assume that 0 < Pnqn+l< 1 for each n E Z. Denote, for nEZ,
00 n
QnF TI qmo
Pn:;::: TI Pm,
m=n ,"=00
Denote by (Cl) the condition
(C+) _{TI Pn}
> 0
11>0
and lim sup qn = 1,
n~oo
and by (C) the condition
(C) TI q"
> 0
11<0
and
liill sup Pn^{:;:::}1.
n+oo
 
PROPOSITION 4.5. (a) If neither (C+) nor (C) hold, then 'p ==1.
(b) If (C+) holds but (C) does not hold, then 'Pn= 1  Pm n E 7L (c) If (C) holds but (C+) does not hold, then 'Pn= 1  Qm n E 7L
(d) If both (C+) and (C) hold, then 'Pn= 1max {Pm Qn}, n
E71..Proof It will be more convenient to work in "1complement" mode, calculating
IjI==1 'P,and using property (B). The calculation of
^{IjI}proceeds through the following steps (for details, see [HS]).
(1) If IjIn= 0 for some n E 71.,then I/J==O.
(2) Put IjIl= pllj1, 1/J2= p2ifJ; if I/J==0 then it is impossible to have for some nElL., 1jI~= IjIn and 1jI~+1= I/Jn+l'
(3) IjIn> 1jI~~ I/Jm> 1jI;" for each m < n, and IjIn
>
I/J~~ IjIm> 1jI;" for each m ~ n.(4) Thus only the following four cases are possible:
/
(a)
1f/=ljIl=1jI2==0;(b) If/n = 1jI~
>
^{If/~} for each n ElL.;(c)
If/n=
^{1jI~}>
I/J~for each 11E 71.;(d) there exists no E 71.such that IjIn= If/~
>
^{If/~} for each n> no, and If/n= I/J~< If/~for each n < no.
(5) Suppose If/> O. If, for some no E 71.,If/n= If/~
for each n > no,then
TI n> noPn>
^{O.}Similarly, if IjIn
=
^{If/~} for each n < no, then TIn<no qn > O.(6) In particular, if TIn>oPn = TIn<o qn = 0, then If/ ==O.
(7) Suppose IjI> O. If, for some no E 71., IjIn
=
^{1jI~}for each n> no, then
lim SUPn~C()Pnqn+l = 1. Similarly, if If/n = I/J~ foreach n < no, then lim SUPn~C() Pnqn+l = 1.
(8) The following is a partial converse to (7): Let
_{0/}> 0 be any max~harmonic function satisfying
o/n=
o/~for each n> no, and suppose lim SUPn~C() Pnqn+l= 1. Then the unique superharmonic function lying between
0/and
0/2is
^{0/}itself. A similar statement holds if o/n=
I/J~for each n < no and lim sup Pnqn+l= 1.
(9) In case (b) condition (C+) holds; in case (c) condition (C) holds; and in case (d) both conditions (C+) and (C) hold.
(10) Conversely, if (C+) holds but (C) does not, then case (b) must occur.
Similarly, if (C) holds but not (C+), then case (c) must occur.
(11) Finally, if both (C+) and (C) hold, then case (d) mus( occur. Q.E.D.
5. Verification of program termination. The results developed in the two preceding
sections provide us with methods for calculating the function 'P for any concurrent probabilistic program. However, in many applications the only question of interest concerning 'Pis whether 'P==1, i.e., whether the program terminates almost surely from any initial state under any fair schedule. In this section we will present several characterizations of program termination, the first two of which are straightforward specializations of the general results of the preceding sections, while the third involves a somewhat different approach, generalizing that used in [HSP] for finite state spaces.
PROPOSITION5.1.
_{'P} _{==}1 if and only if no minharmonicfunction smaller than 1 has property (A).
Proof See Corollary 4.3.
PROPOSITION 5.2. 'P. 1
if
and onlyif
there exist an ordinal c and transfinite sequencesof functions {5~}
^{a;:;;c,}k
EK, and {5aL;:;;chaving the following properties:
(1) 8o=5~=Xx, kEK;
(2) 5~ is subharmonic for each a~ c and each k E K;
(3) 5a ~ maxkEK 5~, a <
c;
(4) 5~+1<pk5a, kEK, a<c;
(5) 5~ ~ SUPb<a 8~, for limit ordinals a, and k E K;
(6) infiEI5c(i»0.
,.~.
Proof If cp==1 then the cpiterates can be taken as the D'S. Conversely, if such sequences of functions are given, then by transfinite induction oa ~ cpfor each ordinal a. In particular Oc~ cp,so that inCEl cp(i) > 0, and by the zeroone law (Theorem 2.3) we must have cp==1. Q.E.D.
Our next characterization' of program termination generalizes one of the charac terizations given in [HSPJ for finitestate programs. Intuitively speaking, if the program does not always terminate, then there must exist some "ergodic structure" of nonter minating states, through which an "adversary" fair scheduler can iterate forever without reaching X. Unlike the case of a finite state space, where such a structure was a single
"Kergodic" set, ergodicity in general state spaces is a much more complex notion, and is defined as foll:ows.
DEFINITION. A Kergodic chain is a nonincreasing sequence {En}n;;;;;! of nonempty subsets of Xc ==I  X such that
lim sup (QxEJ(i)=O.
n>co iEE"
m;;;;;}
In other words, let n ~ 1, i E En' m ~ 1 and k E K be given. Then there exists a subschedule in T( i, k) which reaches Em with probability tending to 1 uniformly as 117 co. That is, without losing too much probability, we can'reach any of the sets Em from any state in Ell after scheduling any required process.
THEOREM 5.3. cp==1
if and only if I  X does not contain any Kergodic chain.
Before proving this theorem, we need two lemmata.
LEMMA5.4. Let 0> 0, and define D = {i
E I: cp(i) ~ 8}. Then cp~ QXD'Proof Let i E I, k E K, and 0 E IF( i). For each n ~ 1 define a stopping time Nn on H"'(i) so that Nn(1T) is the. nth time k has been scheduled along 1T; note that {NII}II;;;;;!is an increasing sequence of fLua.s. finite stopping times, whose limit is +co.
For each n ~ 1 the subschedule 7"n
= (0, Nn)
ET(i, k), so that
(QkXD)(i) ~ ET,,(XD) = fLO"{cp(iNJs 5}.Consider the sequence of functions
{fm}m;;;;;u, defined by fm(1T)=cp(im),msO, 1TEH*(i). By Theorem 2.3 {fm} converges a.s. to a limit foo, such that foo(1T) is 1 if X is reached, and is otherwise O. Therefore we also have cp(iNJ:fco a.s. as n7co, so that
fLO"{fco~ o}~ Jim fLO"{cp(iNJ ~ 5}~ (QkXD)(i).
,,>00
Since 0> 0, we have fco( 1T)~ 0 if and only if foo( 1T)
= 1, or, alternatively, if and only
if X is reached along 1T.Thus ^{.}
E(T(Xx) :;=fL(T{fco ~ o} ~ (QkXD)( i),
from which our assertion follows. Q.E.D.
LEMMA5.5. Let {G,J,,;;;;;!be a nondecreasing sequence of subsets of I, all of which contain X, and let {E,,},,;;;;;! be a sequence of positive. numbers converging to O.Suppose that
Qxo", (i) ~ E"
for each m, n ~ 1 and each i E G~. Then
cp~ sup QXo",.
m
Proof Put {3==sUPm QXo",. The above assumption concerning {Gn} can be restated as
Qxo", ~ E,,' Xo~+ 1.
Xo"
=
^{en}+ (1 en)Xo",
for each m, n ~ 1. This implies
t32En+(1En)XG,
.
^{M} ^{n ~ 1,} and thusQt3 2 Q(En + (1 En)XGJ, n:> 1.
However, it is easily checked that for any scalars a, b> 0 and any nonnegative function a we have
Q(a+ba)=a+bQa.
Hence,
Qt3 2 En+ (1 cn)QXGM~ Cn+ (1 8n)t3,
for each n ~ 1. Letting n ? co, we obtain Qf3 ::: 13.But f3 is subharmonic (as a supremum of subharmonic functions), thus
13=
Qt3 (see Lemmata 3.1 and 3.2), implying 'P 2 13by Theorem 4.1. Q.E.D.
Proof of Theorem 5.3. The theorem is now an easy consequence of the last two lemmata. For example, if 'P is not identically 1, then, by Lemma 5.4, the collection
{En}n~l
is a Kergodic chain, where
'",
En={iEI: 'P(i)<~}.
The converse statement follows similarly from Lemma 5.5 (for more details, see [HS]). Q.E.D.
Example 5 revisited. Consider the three cases in the example of "the two combs"
^{I}in which
_{'P}< 1. It is easily verified that in case (b) the chain E~ =
{i: i ~ n}, n:> 1, isKergodic; similarly, in case (c) the chain E~ = {i: i 2 n}, n ~ 1, is Kergodic; and in case (d) both these chain~ are Kergodic.
Remark In the case of a finite state space, ergodicity is manifested in a single set (see [HSP]). In an analogous manner, we could have considered here the set E =
{i E I: 'P(i) =
^{O}}as a natural candidate forbeing Kergodic (that is, consider the constant
chain En
= E,
n ~ 1). There are two problems, however, with this approach, whichmake it infeasible for general state spaces. One problem is that E may be empty (as is indeed the case in Example 5 just considered). Moreover, even if E is not empty, it may happen that, starting from some i E E, we never reach E again, but instead reach states j at which 'P(j) is arbitrarily small, but positive. Thus, for general state spaces ergodicity must be defined in terms of an infinite chain of sets rather than in terms of a single set. (In the finite case, though, any Kergodic chain must reduce to a constant set from a certain index on.) Note that this phenomenon occurs in Markov
chains as well.
^{.}We conclude this section with a further property of 'P.
PROPOSITION
5.6. There exists a nondecreasing sequence {Dn}~=l of subsets of I
such that(1) 'PIDc==0, where D = U~=l Dn, (2) 'P = limn~oo QXDM'
Proof Put Dn = {i: 'PO) ~ 1/ n}. Q.E.D.
Note that in the case 'P==1 we can take Dn ==1 for all n. Moreover, COROLLARY 5.7. If I is a finite set, then there exists DeI such that (1) 'PIDc==0,
(2) 'P = QXD'
6. Comparison with the deterministic and the nondeterministic cases. In this section we consider the special case in which each process is deterministic; programs with nondeterministic processes are also included, since any such program can be simulated by a deterministic one involving additional processes. (For example, suppose that one of the processes k1 makes a nondeterministic choice from some set A of alternatives;
the same behavior can be achieved by introducing a new shared variable v which k1 sets to some value in A prior to making the choice, and by introducing another process k2 whose only action is to iterate v over the set A. k1 then makes its choice deterministi cally, depending on the current value of v. Thus the nondeterminism is now transferred to the schedulerthe final choice depends on how many times k2 has been scheduled in between.) Thus, by specializing the various equivalent criteria for program termina tiondeveloped so far in this paper to the deterministic case, we can obtain similar criteria for the termination of deterministic (or nondeterministic) concurrent programs.
I As it turns out, the criterion obtained in this way from the characterization of <pas
the limit of the (f)iterates (Theorem 3.8, Proposition 5.2) essentially coincides with the known criterion of Lehmann, Pnueli and Stavi [LPS]. On the other hand, specialization of Theorem 4.2 leads to a new characterization for deterministic and nondeterministic termination. So as not to make this characterization appear too deep, we provide a direct non probabilistic proof of its validity. '
We begin by observing that in the deterministic case all transitions have probability 0 or 1, so that each of the operators Qk, k E K, and Q, when applied to a function which takes only the values 0, 1 (i.e., a characteristic function of some subset of 1) yields a similar function. Hence each of the (f)iterates 'Y~, k E K, (resp. 'Ya) is a characteristic function of the form XG~ (resp. XGJ. Note also that a characteristic function XA is subharmonic if and only if for each k E K and each i E A the (unique) ktransition from i is to a state in A, i.e., there are no transitions from states in A to states outside A. Hence, spelling out the conditions in Proposition 5.2 in terms of the subsets of I corresponding to the functions appearing there, we obtain the following.
COROLLARY6.1. A deterministic program terminates if. and only if there exist transfinite (increasing) sequences {G~}
^{a;;;;O,}k
EK, and {Ga} a;;;;O of subsets of I having
the following properties:(1) Go
= G~ = X, k
^{E}K;
(2) there are no transitions from states in
G~
to states outside G~, for each ordinal a and each k E K ;(3) Ga = U kEK G~, for each ordinal" a;
(4) for each k E K and each ordinal a, all ktransitions frqm states in G~+l are to states in Gu;
(5) G~ = U I,<a G~, for each limit ordinal and each k E K;
(6) there exists an ordinal c such that Gc
=
1. .These conditions, however, are merely a rephrasing of the characterization for termination of "just" programs given by Lehmann, Pnueli and Stavi in [LPS]. To see this, define a function p from I to the ordinals 'by
pO) = min {a: i E Ga}, i E I,
and a function h: I ^{~} K which maps each i E Gp(i) to some k E K such that i E G;(i)' Then it is easily checked that these functions satisfy the conditions in [LPS] for just termination, i.e.: the "ranking" map p never increases during execution; activating process h( i) at state i always strictly decreases the value of p; and h remains unchanged