IMCs strictly separate interactive from Markovian transitions; therefore, they can be seen as a fully orthogonal extension of labeled transition systems with exponentially dis-tributed delays. This enables compositional modeling with intermittent weak bisimula-tion minimizabisimula-tion [Her02] and even allows us to augment existing untimed process alge-bra specifications with random timing [HK00, BHH+09]. Moreover, the IMC formalism is not restricted to exponential delays but permits to encode arbitrary phase-type distribu-tions such as hyper- and hypoexponentials [Pul09]. An excellent and detailed discussion of the advantages of the IMC modeling formalism can be found in the paper [BHK06].
6.1.1 Preliminaries
Opposed to CTMDPs, interactive Markov chains (IMCs) disentangle the relation be-tween Markovian and nondeterministic behaviors: Therefore, IMCs strictly separate Mar-kovian from interactive transitions. We restate the definition of IMCs from [Her02]:
Definition 6.1 (Interactive Markov chain). An interactive Markov chain is a tuple M = (S, Act, IT, MT, ν) where S and Act are nonempty sets of states and actions, IT ⊆ S × Act × S is a set of interactive transitions and MT ⊆ S × R>0×S is a set of Markovian transitions. Further, ν ∈ Distr(S) is the initial distribution.
We distinguishexternal actions in Acte from internal actions in Acti and setAct = Acte⊍ Acti. The reason for this distinction is that IMCs may be composed via synchro-nization over the set of external actionsActe, while internal actions inActi are not ob-servable from the outside environment. For a detailed discussion of the compositional aspects of IMCs, we refer the reader to [Her02]. For the scope of this thesis, we consider closed IMCs [Her02, Joh07], that is, we focus on the IMCM that is obtained as the final outcome of the composition. Accordingly,M is not subject to any further synchroniza-tion and all remaining external acsynchroniza-tions can safely be hidden. In our analysis, we therefore assume thatActe = ∅ and identify the setsAct and Acti.
For Markovian transitions, we useλ and µ to denote rates of exponential distributions.
Moreover,IT(s) = {(s, α, s′) ∈ IT} is the set of interactive transitions that leave state s;
similarly, for Markovian transitions, we setMT(s) = {(s, λ, s′) ∈ MT}. A state s ∈ S is Markovian iff MT(s) /= ∅ and IT(s) = ∅; it is interactive iff MT(s) = ∅ and IT(s) /= ∅.
Further,s is a hybrid state iff MT(s) /= ∅ and IT(s) /= ∅; finally, s is a deadlock state iff MT(s) = IT(s) = ∅. We use MS ⊆ S and IS ⊆ S to refer to the sets of Markovian and interactive states inM.
s0
Figure 6.1: Example of an IMC with Markovian and interactive states.
For a Markovian states ∈ MS, we define R(s, s′) = ∑{λ ∣ (s, λ, s′) ∈ MT(s)} as the rate to move from states to state s′andE(s) = ∑s′∈SR(s, s′) as the exit rate of state s; further, postM(s) = {s′∈S ∣ R(s, s′) > 0} denotes the set of successor states of state s. The discrete branching probability to move from state s to state s′is P(s, s′) = RE(s)(s,s′).
Example 6.1. LetM be the IMC depicted in Fig. 6.1. The semantics of Markovian states equals that of a CTMC state: More precisely, consider the Markovian state s0and the tran-sition(s0, 0.3,s2) ∈ MT(s) (depicted by a solid line) that leads from state s0to state s2with rate λ = 0.3. The transition’s delay is exponentially distributed with rate λ; hence, it expires in the next z ∈ R≥0time units with probability∫0zλe−λtdt =(1 − e−0.3z). As state s0has two Markovian transitions, they compete for execution and the IMC moves along the transition whose delay expires first. Clearly, in such a race, the sojourn time in s0is determined by the first transition that executes. As the minimum of exponential distributions is exponentially distributed with the sum of their rates, the sojourn time in a state s is determined by the exit rate E(s) of state s. In general, the probability to move from a state s ∈ MS to a suc-cessor state s′ ∈S equals the probability that (one of) the Markovian transitions that lead from s to s′wins the race. Accordingly, for state s0of our example, we have R(s0,s2) = 0.3,
E(s0) = 0.3 + 0.6 = 0.9 and P(s0,s2) = 13. ♢
For interactive transitions, we adopt themaximal progress assumption [Her02, p. 71]
which states that internal transitions (i.e. interactive transitions labeled with internal ac-tions) trigger instantaneously. This implies that they take precedence over all Markovian transitions whose probability to execute immediately is 0. Therefore all Markovian tran-sitions that emanate a hybrid state can be removed without altering the IMC’s behavior.
This allows us to assume throughout this chapter thatMT(s) ∩ IT(s) = ∅ for all s ∈ S.
To ease the development of the theory, we assume w.l.o.g. that each internal action α ∈ Act has a unique successor state, denoted succ(α); note that this is no restriction, for if(s, α, u) , (s, α, v) ∈ IT(s) are internal transitions with u /= v, we may replace them by new transitions(s, αu,u) and (s, αv,v) with fresh internal actions αuandαv.
Theinternal successor relation↝i ⊆S × S is given by s ↝i s′iff(s, α, s′) ∈ IT; further-more, the internal reachability relation ↝∗i is the reflexive and transitive closure of↝i. Accordingly, we defineposti(s) = {s′∈S ∣ s ↝i s′} and Reachi(s) = {s′∈S ∣ s ↝∗i s′}.
Finally, entering a deadlock state results in a time lock, as neither internal nor
Marko-vian transitions are available. Therefore, we equip deadlock statess ∈ S with interactive self-loops(s, α, s). Note that the occurrence of time locks breaks compositionality; how-ever, note that our analysis takes place on the closed model which is the monolithic result that is obtained after all compositions.
We justify the modification of deadlock states as follows: Whereas each interactive or Markovian state has an associated sojourn time distribution (which is either 0 or an exponential distribution), the sojourn time in deadlock states remains unquantified. In this case, we encounter atime lock situation where the global time does not proceed any further: If a deadlock state is reached at global timetdead, the probability distribution of the associated stochastic process {Xt}t∈R≥0 is undefined for time-points t > tdead. The same phenomenon occurs if a closed IMC eventually remains in a cycle of interactive transitions. In this case, the global time also stops, resulting in a time lock. Hence, the two situations are semantically equivalent which justifies to equip any deadlock state with an interactive self-loop.
Note however, that our approach also allows for a different deadlock state semantics, where the global clock continues; in this case, we would add a Markovian instead of an internal self-loop.
6.1.2 Paths in interactive Markov chains
To unify the notation for interactive and Markovian transitions, we introduce a special action ∉Act and let σ range over Act =Act ⊍{}. In this way, we can denote a finite
refer to the(k+1)-th state on π and its associated sojourn time. Accordingly, ∆(π, i) =
∑i−1k=0tkis the total time spent onπ (where ∆(π, 0) = 0) when reaching state π[i]. If π is finite with∣π∣ = n, then ∆(π) = ∆(π, n) is the total time spent on π; similarly, π↓ = sn
is the last state onπ. The path infix between the(i+1)-th and the (j+1)-th state of π is denotedπ[i..j].
Because internal transitions occur immediately in IMCs, an IMC can traverse several states at once. Therefore, we modify the definition ofπ@t such that π@t ∈ (S∗⊍Sω) denotes the sequence of states that are traversed onπ at time point t ∈ R≥0.
The formal derivation ofπ@t is slightly involved: Let i be the smallest index such that t ≤ ∆(π, i). Then π[i] is the first state on π that is visited at or after time t; if no such state exists, we setπ@t =⟨⟩. Otherwise we distinguish two cases: If t < ∆(π, i), we define π@t =⟨si−1⟩; if t = ∆(π, i), let j be the largest index (or +∞, if no such finite index exists)
Intuitively, the (i+1)-th state on path π (i.e. π[i]) is entered at time ∆(π, i). To find the first state of the sequence π@t, let i be the first index on π where at least t time units have passed. Formally, we have to choose the minimal i that satisfies t ≤ ∆(π, i). For such a minimal i, t < ∆(π, i) implies that time has passed in the previous state π[i−1] and that we have been in that state at time point t. Hence, π[i−1] must be a Markovian state and we set π@t =⟨π[i−1]⟩. Otherwise t = ∆(π, i), implying that state π[i] is entered at time point t.
If it is an interactive state, further transitions can occur immediately. Hence, we look for the maximal index j, for which ∆(π, j) still equals t and define π@t = ⟨π[i] . . . π[j]⟩. ♢ We write s ∈ ⟨si. . .sj⟩ if s ∈ {si, . . . ,sj}; further, for states s ∈ ⟨si. . .sj⟩ we define Pref(⟨si. . .sj⟩, s) = ⟨si, . . .sk⟩, where s = sk and k is minimal. If s ∉ ⟨si. . .sj⟩, we set Pref(⟨si. . .sj⟩, s) = ⟨⟩. The definitions for time-abstract paths are similar.
6.1.3 Events and measurable spaces
A path π (time-abstract path π′) as defined in Sec. 6.1.2 is a concatenation of a state and a sequence of combined transitions (time-abstract combined transitions) from the set Ω = R≥0× Act×S (Ωabs = Act ×S); hence, π = s0 ○ m0○ m1 ○ . . . ○ mn−1 with mi = (ti,σi,si+1) ∈ Ω (mi = (σi,si+1) ∈ Ωabs). Thus Pathsn(M) = S × Ωn is the set of paths of lengthn in an IMCM; further, Paths⋆(M), Pathsω(M) and Paths(M) are the sets of finite, infinite and all paths in M. To refer to time-abstract paths, we add the subscriptabs; further the reference toM is omitted wherever possible. The measure-theoretic concepts are mentioned only briefly, as they directly carry over from the defi-nitions for the CTMDP case (cf. Sec. 3.3.2 on page 76): Events inM are measurable sets of paths; as paths are Cartesian products of combined transitions, we define theσ-field F= σ(B(R≥0) × FAct× FS) on subsets of Ω where FS= 2S and FAct = 2Act.
The productσ-field FPathsn of measurable subsets of Pathsn is defined as usual, that is, FPathsn = σ({S0× M1× ⋯ × Mn ∣ S0∈ FS,Mi ∈ F}). As for CTMDPs, the cylinder-set construction [ADD00] extends this to infinite paths: A setB ∈ FPathsnis called abase of an infinitecylinder C where C = Cyl(B) = {π ∈ Pathsω∣ π[0..n] ∈ B}. Finally, the cylinders generate theσ-field FPathsω = σ(⋃∞n=0{Cyl(B) ∣ B ∈ FPathsn}).
t ≤ ∆(π, i) 0 1 2 3 4 5 6 min i max j π@t
0 ✓ ✓ ✓ ✓ ✓ ✓ ✓ 0 2 ⟨s0s1s2⟩
t2−ε ⨉ ⨉ ⨉ ✓ ✓ ✓ ✓ 3 NA ⟨s2⟩
t2 ⨉ ⨉ ⨉ ✓ ✓ ✓ ✓ 3 5 ⟨s3s4s5⟩
t2+ε ⨉ ⨉ ⨉ ⨉ ⨉ ⨉ ✓ 6 NA ⟨s5⟩
Table 6.1: An example for the derivation ofπ@t for interactive Markov chains.
6.1.4 Resolving nondeterminism by schedulers
An IMC M is nondeterministic iff for some s ∈ IS, there exist interactive transitions (s, α, u) , (s, β, v) ∈ IT(s) with u /= v: For example, nondeterminism arises in the IMC in Fig. 6.1: In state s2, two internal transitions (with actions α and β) lead to states s1
and s4, respectively. By the maximal progress assumption, they both execute instanta-neously at time point 0. Hence, no order of execution can be fixed, which leads to the situation that the successor state of states2(eithers1ors4) is not uniquely determined. To resolve this nondeterministic choice, we useschedulers: IfM reaches state s2along a his-tory π ∈ Paths⋆, a scheduler yields a probability distribution over the setAct(π↓) = {α, β}.
Formally, we define the set ofenabled actions in an interactive state s ∈ IS of an IMC as follows:
Act(s) = {α ∈ Act ∣ ∃s′∈S.(s, α, s′) ∈ IT} .
IMC schedulers are closely related to CTMDP schedulers and most of the concepts from Sec. 3.3.2 and Chapters 4 and 5 apply analogously. The only notable difference is the distinction between interactive and Markovian states: Nondeterminism does not occur in the latter, as the successor states are probabilistically quantified. Hence, the only source of nondeterminism are competing internal transitions in interactive states.
Definition 6.2 (Generic measurable scheduler). A generic scheduler on an IMCM = (S, Act, IT, MT, ν) is a partial mapping D ∶ Paths⋆× FAct ↣ [0, 1] such that D(π, ⋅) ∈ Distr(Act(π↓)) for all π ∈ Paths⋆with π↓ ∈ IS. A generic scheduler D is measurable (that is, a GM scheduler) iff for all A ∈ FAct, D−1(A) ∶ Paths⋆→[0, 1] is measurable.
Measurability states that {π ∣ D(π, A) ∈ B} ∈ FPaths⋆ holds for all A ∈ FAct and B ∈ B([0, 1]); intuitively, it excludes schedulers which resolve the nondeterminism in a way that induces non-measurable sets. Recall that no nondeterminism occurs if π↓ ∈ MS.
However, we slightly abuse notation and assume that D(π, ⋅) = { ↦ 1} if π↓ ∈ MS so thatD yields a distribution over Act. In this way, we can treat aGM-scheduler D as a total functionD ∶ Paths⋆× FAct →[0, 1].
A GM scheduler D is deterministic iff D(π, ⋅) is degenerate for all π ∈ Paths⋆. We use GM (and GMD) to denote the class of generic measurable (deterministic) sched-ulers. Further, aGM scheduler Dabsistime-abstract (GMabs) iffabs(π) = abs(π′) implies Dabs(π, ⋅) = Dabs(π′, ⋅).
Example 6.3. If state s2in Fig. 6.1 is reached along path π = s0 0.4,
ÐÐ→ s2, then D(π) might yield the distribution{α ↦ 12,β↦ 12}, whereas for history π′ =s0
1.5,
ÐÐ→ s2, it might return
a different distribution, say D(π) = {α ↦ 1}. ♢
6.1.5 Probability measures for IMCs
In this section, we define the probability measure [Joh07] induced byD on the measur-able space(Pathsω, FPathsω). We first derive the probability of measurable sets of com-bined transitions, i.e. of subsets of Ω:
Definition 6.3 (Probability of combined transitions). Let M = (S, Act, IT, MT, ν) be an IMC and D ∈ GM. For π ∈ Paths⋆, we define the probability measure µD(π, ⋅) ∶ F→[0, 1]:
µD(π, M) =⎧⎪⎪
⎨⎪⎪⎩
∑α∈Act(π↓)IM(α, 0, succ(α)) ⋅ D (π, {α}) if π↓ ∈ IS
∫R≥0E(s)e−E(s)t⋅∑s′∈SIM(, t, s′) ⋅ P(s, s′) dt if π↓ ∈ MS. (6.1)
As usual, IM denotes the indicator function for the setM. Intuitively, µD(π, M) is the probability to continue along one of the combined transition in the setM. For an interac-tive states ∈ IS, it is the probability of choosing α ∈ Act(π↓) such that (α, 0, succ(α)) is a transition inM. Stated differently, we sum up the probabilities of all combined transi-tions inM that lead immediately with an interactive transition to a successor state of π↓.
If s ∈ MS, µD(π, M) is given by the density for the Markovian transition to trigger at timet and the probability that the IMC moves to a successor state s′according to a com-bined transition inM. As paths are inductively defined using combined transitions, we can lift the probability measureµD(π, ⋅) to FPathsn as usual:
Definition 6.4 (Probability measure). Let M = (S, Act, IT, MT, ν) be an IMC and D ∈ GM. For n ≥ 0, we define the probability measures Prnν,D inductively on the measurable space(Pathsn, FPathsn):
Pr0ν,D∶ FPaths0 →[0, 1] ∶ Π ↦ ∑
s∈Π
ν(s) and
Prn+1ν,D ∶ FPathsn+1→[0, 1] ∶Π ↦
∫
PathsnPrnν,D(dπ)∫
ΩIΠ(π ○ m) µD(π, dm).6.1.6 Interactive probabilistic chains
In this section, we introduceinteractive probabilistic chains (IPCs) [CHLS09] which serve as the discrete-time analogon of IMCs. In an IPC, Markovian transitions are replaced by probabilistic transitions. As a consequence, no delay time distribution is associated with probabilistic states. Therefore, taking a probabilistic transitions corresponds to a discrete time step in the IPC.
The semantics of interactive transitions remains the same as in the IMC case. Open IMCs can synchronize over the set ofexternal actions, whereas internal actions are unob-servable for the environment.
Definition 6.5 (Interactive probabilistic chain). An interactive probabilistic chain (IPC) is a tuple P = (S, Act, IT, PT, ν), where S, Act, IT and ν are as in Def. 6.1 and PT ∶S × S →[0, 1] is a transition probability function s.t. ∀s ∈ S. PT(s, S) ∈ {0, 1}.
A states in an IPCP is probabilistic iff ∑s′∈SPT(s, s′) = 1 and IT(s) = ∅; PS denotes the set of all probabilistic states. The sets of interactive, hybrid and deadlock states are defined as for IMCs, with the same assumption imposed on deadlock states. Further, we assume any IPC to be closed, that is(s, α, s′) ∈ IT implies α ∈ Acti. Hence,Acte = ∅ and we identify the setsActi andAct.
As for IMCs, we adopt themaximal progress assumption [Her02, p. 71]; hence, internal transitions take precedence over probabilistic transitions and their execution takes 0 dis-crete time steps. In this way, we obtain a full correspondence between IMCs and IPCs, as in both cases internal transitions consume no time.
Definition 6.6 (IPC scheduler). LetP = (S, Act, IT, PT, ν) be an IPC. A partial func-tion D ∶ Paths⋆abs↣ Distr(Act) with D(π) ∈ Distr(Act(π↓)) is a time-abstract history-dependent randomized(GMabs) scheduler.
Note that in the discrete-time setting, measurability issues do not arise. Moreover, we extend D ∈ GMabs to a complete functionD ∶ Paths⋆abs → Distr(Act) and assume that D(π) = { ↦ 1} iff π↓ ∈ PS. To define a probability measure on sets of paths in P, we define the probability of a single transition:
Definition 6.7 (Combined transitions in IPCs). Let P = (S, Act, IT, PT, ν) be an IPC, s ∈ S, σ ∈ Act, π ∈ Paths⋆abs and(σ, s) ∈ Ωabs a time-abstract combined transi-tion. For scheduler D ∈ GMabs, we define
µabsD (π, {(σ, s)}) =⎧⎪⎪⎪⎪
⎨⎪⎪⎪⎪⎩
PT(π↓, s) if π↓ ∈ PS ∧ σ = D(π, {σ}) if π↓ ∈ IS ∧ succ(σ) = s
0 otherwise.
For a set of combined transitions M ⊆ Ωabs, we set µabsD (π, M) =
∑(σ,s)∈MµabsD (s, {(σ, s)}).
s0 s1 s2 s3
Figure 6.2: An example for an IMC and its embedded IPC.
The measuresµabsD extend to a unique measure on sets of paths inP in the same way as it was shown for the IMC case in Sec. 6.1.5.
Example 6.4. Each IMC induces an embedded IPC: Consider the IMCM in Fig. 6.2(a), with initial state s0and interactive states s1and s3. A scheduler D has to resolve the nondeter-minism in state s1: If π = s0
,t0
ÐÐ→ s0
,t1
Ð→ s1is the path that led into state s1, then D(π)(α) is the probability that α is chosen in s1. In Fig. 6.2(b), we depict the embedded IPC emb(M) ofM: It is obtained by disregarding M’s timed behavior and considering the IMC’s discrete branching probabilities P(s, s′) only. Hence emb(M) is the IPC (S, Act, PT, IT, ν), where PT(s, s′) = RE(s)(s,s′)if s ∈ MS and PT(s, s′) = 0, otherwise. ♢