Compositional Abstractions of Stochastic
Systems
?Abolfazl Lavaei1, Sadegh Soudjani2, and Majid Zamani3,4 1
Department of Electrical Engineering, Technical University of Munich, Germany
2
School of Computing, Newcastle University, United Kingdom
3
Department of Computer Science, University of Colorado Boulder, USA, and Ludwig Maximilian University of Munich, Germany
Abstract. In this paper, we propose a compositional approach for con-structing abstractions of general Markov decision processes (gMDPs) using approximate probabilistic relations. The abstraction framework is based on the notion of δ-lifted relations, using which one can quantify the distance in probability between the interconnected gMDPs and that of their abstractions. This new approximate relation unifies composition-ality results in the literature by allowing abstract models to have either finite or infinite state spaces. To this end, we first propose our composi-tionality results using the new approximate probabilistic relation which is based on lifting. We then focus on a class of stochastic nonlinear dy-namical systems and construct their abstractions using both model or-der reduction and space discretization in a unified framework. Finally, we demonstrate the effectiveness of the proposed results by consider-ing a network of four nonlinear dynamical subsystems (together 12 di-mensions) and constructing finite abstractions from their reduced-order versions (together 4 dimensions) in a unified compositional framework. Keywords: Approximate Probabilistic Relations· Compositional Ab-stractions·General Markov Decision Processes·Policy Refinement.
1
Introduction
Control systems with stochastic uncertainty can be modeled as Markov decision processes (MDPs) over general state spaces. Synthesizing policies for satisfying complex temporal logic properties [1] over MDPs evolving on uncountable state spaces is inherently a challenging task due to the computational complexity. Since closed-form characterization of such policies is not available in general,
?
This work was supported in part by the H2020 ERC Starting Grant AutoCPS (grant agreement No. 804639).
a suitable approach is to approximate these models by simpler ones possibly with finite or lower dimensional state spaces. A crucial step is to provide formal guarantees during this approximation phase, such that the analysis or synthesis on the simpler model can be refined back over the original one.
Similarity relations over finite-state stochastic systems have been studied, ei-ther via exact notions of probabilistic (bi)simulation relations [2], [3] or approx-imate versions [4], [5]. Similarity relations for models with general, uncountable state spaces have been also proposed in the literature. These relations either depend on stability requirements on model outputs via martingale theory or contractivity analysis [6], [7] or enforce structural abstractions of a model [8] by exploiting continuity conditions on its probability laws [9], [10].
There have been also several results on the construction of (in)finite ab-stractions for stochastic systems. Construction of finite abab-stractions for formal verification and synthesis is presented in [11]. An adaptive gridding approach is proposed in [14] with the dedicated tool FAUST2 [15]. Extension of such
tech-niques to automata-based controller synthesis and infinite horizon properties is studied in [12] and [13], respectively. Compositional construction of finite ab-stractions using dynamic Bayesian networks is discussed in [16]. Compositional construction of infinite abstractions (reduced-order models) is respectively pro-posed in [17, 18] using small-gain type conditions and dissipativity-type proper-ties of subsystems and their abstractions. Compositional construction of finite abstractions is studied in [19, 20]. Recently, compositional synthesis of large-scale stochastic systems using a relaxed dissipativity approach is proposed in [21].
In this paper, we provide conditions under which the proposed similarity re-lations between individual gMDPs can be extended to rere-lations between their respective interconnections. These conditions enable compositional quantifica-tion of the distance in probability between the interconnected gMDPs and that of their abstractions. Our compositional scheme allows constructing both infinite and finite abstractions in a unified framework.
Similarities between two gMDPs have been recently studied in [22] using a notion ofδ-lifted relation. This notion is used in [23] for temporal logic control of gMDPs. These two works are focused on single gMDPs. One of the main contributions of this paper is to extend this notion such that it can be applied to networks of gMDPs. Furthermore, we provide an approach for the construction of finite MDPs in a unified framework for a class of stochastic nonlinear dynamical systems, considered as gMDPs, whereas the construction scheme in [22, 23] only handles the class of linear systems.
2
General Markov Decision Processes
In our framework, we consider the class of general Markov decision processes (gMDPs), as in the next definition, that evolves over continuous or uncountable state spaces. This class of models generalizes the usual notion of MDP [1] by including internal inputs that are employed for composition [19], and by adding an output space over which properties of interest are defined [22].
Definition 1. A general Markov decision process (gMDP) is a tuple
Σ= (X, W, U, π, T, Y, h), (1)
where
– X ⊆ Rn is a Borel space as the state space of the system. We denote by (X,B(X))the measurable space withB(X)being the Borel sigma-algebra on the state space;
– W ⊆Rp is a Borel space as the internal input space of the system; – U ⊆Rm is a Borel space as the externalinput space of the system; – π=B(X)→[0,1]is the initial probability distribution;
– T :B(X)×X×W×U →[0,1]is a conditional stochastic kernel that assigns to any x ∈ X, w ∈ W, and ν ∈ U, a probability measure T(·|x, w, ν) on the measurable space(X,B(X)). This stochastic kernel specifies probabilities over executions{x(k), k∈N} of the gMDP such that for any setA ∈ B(X) and anyk∈N, P(x(k+ 1)∈ A x(k), w(k), ν(k)) = Z A T(dx(k+ 1)|x(k), w(k), ν(k)).
– Y ⊆Rq is a Borel space as the output space of the system;
– h:X →Y is a measurable function that maps a state x∈X to its output
y=h(x).
Evolution of the state of a gMDPΣ, can be alternatively described by
Σ:
x(k+ 1) =f(x(k), w(k), ν(k), ς(k)),
y(k) =h(x(k)), k∈N, x(0)∼π, (2)
for input sequences w(·) : N → W and ν(·) : N → U, where ς := {ς(k) :
Ω→Vς, k∈N} is a sequence of independent and identically distributed (i.i.d.) random variables on a setVς with sample spaceΩ. Vector fieldf together with the distribution ofς give the stochastic kernelT. IfX, W, Uare finite sets, system
Σ is called finite, and infinite otherwise.
Next section presents approximate probabilistic relations that can be used for relating two gMDPs while capturing probabilistic dependency between their executions.
3
Approximate Probabilistic Relations based on Lifting
Definition 2. LetX,Xˆ be two sets with associated measurable spaces(X,B(X)) and ( ˆX,B( ˆX)). Consider a relation Rx ∈ B(X ×Xˆ). We denote by Rδ¯ ⊆ P(X,B(X))× P( ˆX,B( ˆX)), the corresponding δ-lifted relation such that ΦRδ¯ Θ
if there exists a probability space(X×X,ˆ B(X×Xˆ),L) (equivalently, a lifting L) satisfying
– ∀A ∈ B(X), L(A ×Xˆ) =Φ(A), – ∀A ∈ Bˆ ( ˆX), L(X×Aˆ) =Θ( ˆA),
– for the probability space (X ×X,ˆ B(X ×Xˆ),L), it holds that xRxxˆ with probability at least1−δ, equivalently, L(Rx)≥1−δ.
For a given relationRx⊆X×Xˆ, the above definition specifies required prop-erties for lifting relation Rx to a relation ¯Rδ that relates probability measures overX and ˆX.
We are interested in usingδ-lifted relation for specifying similarities between a gMDP and its abstraction. Therefore, internal inputs of the two gMDPs should be in a relation denoted by Rw. Next definition gives conditions for having a stochastic simulation relation between two gMDPs.
Definition 3. Consider gMDPsΣ= (X, W, U, π, T, Y, h)andΣb = ( ˆX,W ,ˆ U ,ˆ ˆπ,
,T , Y,ˆ hˆ)with the same output space. SystemΣb is (, δ)-stochastically simulated by Σ, i.e. Σb δ Σ, if there exist relationsRx⊆X×Xˆ andRw ⊆W ×Wˆ for which there exists a Borel measurable stochastic kernel LT(· | x,x, w,ˆ w,ˆ ˆν) on
X×Xˆ such that
– ∀(x,xˆ)∈Rx, kh(x)−ˆh(ˆx)k ≤,
– ∀(x,xˆ)∈Rx,∀wˆ ∈Wˆ,∀νˆ∈Uˆ, there exists ν ∈U such that ∀w∈W with (w,wˆ)∈Rw,
T(· |x, w, ν) ¯Rδ Tˆ(· |x,ˆ w,ˆ νˆ),with liftingLT(· |x,x, w,ˆ w,ˆ νˆ),
– πRδ¯ ˆπ.
Definition 3 can be applied to gMDPs without internal inputs that may arise from composing gMDPs via their internal inputs. For such gMDPs, we eliminate Rw, thus the definition reduces to that of [22].
Definition 3 enables us to quantify the error in probability between a concrete systemΣand its abstractionΣb. In any (, δ)-approximate probabilistic relation,
δ is used to quantify the distance in probability between gMDPs and for the closeness of output trajectories as stated in the next theorem. This theorem is adapted from [22] and provides the probabilistic closeness guarantee between interconnected gMDPs and that of their compositional abstractions which are discussed in Section 4.
Theorem 1. If Σb δ Σ and(w(k),wˆ(k))∈Rw for all k∈ {0,1, . . . , T}, then for all policies on Σb there exists a policy for Σ such that, for all measurable events A⊂YT+1,
P{{yˆ(k)}0:T ∈A−} −γ≤P{{y(k)}0:T∈A} ≤P{{yˆ(k)}0:T ∈A}+γ, (3)
with constant1−γ := (1−δ)T+1, and with the-expansion and -contraction of Adefined as
A:={y(·)∈YT+1∃y¯(·)∈Awithmaxk≤Tky¯(k)−y(k)k ≤},
A−:={y(·)∈Ay¯(·)∈A for ally¯(·)withmaxk≤Tky¯(k)−y(k)k ≤}.
In the next section, we define composition of gMDPs via their internal inputs and discuss how to relate them to a network of interconnected abstractions based on their individual relations.
4
Compositional Abstractions of Interconnected gMDPs
4.1 Interconnected gMDPs LetΣbe a network ofN ∈N≥1 gMDPs
Σi= (Xi, Wi, Ui, πi, Ti, Yi, hi), i∈ {1, . . . , N}. (4)
We partition internal input and output ofΣi as
wi= [wi1;. . .;wi(i−1);wi(i+1);. . .;wiN], yi= [yi1;. . .;yiN], (5)
and also output space and function as
hi(xi) = [hi1(xi);. . .;hiN(xi)], Yi= N
Y
j=1
Yij. (6)
The outputsyiiare denoted asexternal ones, whereas the outputsyij withi6=j as internal ones which are employed for interconnection by requiringwji=yij. This can be explicitly written as
wi=gi(x1, . . . , xN)=h1i(x1);. . .;h(i−1)i(xi−1);h(i+1)i(xi+1);. . .;hN i(xN). (7)
Now, we formally define theinterconnected gMDP Σ as follows.
Definition 4. Consider N ∈ N≥1 gMDPs Σi = (Xi, Wi, Ui, πi, Ti, Yi, hi), i ∈
{1, . . . , N}, with the input-output configuration as in (5) and (6). The inter-connection of Σi, i∈ {1, . . . , N}, is a gMDPΣ= (X, U, π, T, Y, h), denoted by
I(Σ1, . . . , ΣN), such that X := QNi=1Xi, U := QNi=1Ui, Y := QNi=1Yii, and
h=QN
i=1hii, with the following constraints:
∀i, j∈ {1, . . . , N}, i6=j: wji=yij, Yij⊆Wji. (8)
Moreover, one has conditional stochastic kernelT :=QN
i=1Ti and initial prob-ability distributionπ:=QN
i=1πi.
4.2 Compositional Abstractions of Interconnected gMDPs
We assume that we are given N gMDPs as in Definition 1 together with their corresponding abstractions Σbi = ( ˆXi,Wˆi,Uˆi,πˆi,Tˆi, Yi,ˆhi) such that Σbi δii Σi for some relation Rxi and constantsi, δi. Next theorem shows the main com-positionality result of the paper.
Theorem 2. Consider the interconnected gMDP Σ =I(Σ1, . . . , ΣN) induced by N ∈ N≥1 gMDPs Σi. Suppose Σbi is (i, δi)-stochastically simulated by Σi with the corresponding relations Rxi andRwi and liftingLi. If
with interconnection constraint mapsgi,ˆgidefined as in (7), thenΣb =I(Σb1, . . . ,
,ΣbN) is (, δ)-stochastically simulated by Σ =I(Σ1, . . . , ΣN) with relation Rx defined as x1 . . . xN Rx ˆ x1 . . . ˆ xN ⇔ x1Rx1xˆ1, . . . xNRxNˆxN, and constants=PN i=1i, andδ= 1− QN
i=1(1−δi). Lifting L and interface
ν are obtained by taking products L = QN
i=1Li and ν = QN
i=1νi, and then substituting interconnection constraints (8).
5
Construction of Abstractions for Nonlinear Systems
Here, we focus on a specific class of stochastic nonlinear dynamical systemsΣ
as
Σ:
x(k+1) =Ax(k)+Eϕ(F x(k))+Dw(k)+Bν(k)+Rς(k),
y(k) =Cx(k), (10)
whereς(·)∼ N(0,In), andϕ:R→Rsatisfies 0≤ ϕ(c)−ϕ(d)
c−d ≤b, ∀c, d∈R, c6=d, (11)
for someb∈R>0∪ {∞}. We use the tupleΣ= (A, B, C, D, E, F, R, ϕ)to refer to the class of nonlinear systems of the form (10).
Existing compositional abstraction results for this class of models are based on either model order reduction [17], [18] or finite MDPs [19], [20]. Our proposed results here combine these two approaches in one unified framework. In other words, our abstract model is obtained by discretizing the state space of a reduced-order version of the concrete model.
5.1 Construction of Finite Abstractions
Consider a nonlinear systemΣ = (A, B, C, D, E, F, R, ϕ) and its reduced-order versionΣbr= ( ˆAr,Bˆr,Cˆr,Dˆr,Eˆr,Fˆr,Rˆr, ϕ). Note that indexrin the whole paper
signifies the reduced-order version of the original model. Construction of a finite gMDP from Σbr follows the approach of [14, 24]. Denote the state and input
spaces ofΣbrrespectively by ˆXr,Wˆr,Uˆr. We construct a finite gMDP by selecting
partitions ˆXr =∪iXi, ˆWr =∪iWi, and ˆUr=∪iUi, and choosing representative points ¯xi ∈Xi, ¯wi ∈Wi, and ¯νi ∈Ui, as abstract states and inputs. The finite abstraction ofΣ is a gMDPΣb = ( ˆX,W ,ˆ U ,ˆ π,ˆ T , Y,ˆ ˆh), where
ˆ
Transition probability matrix ˆT is constructed according to the dynamics ˆx(k+ 1) = ˆf(ˆx(k),wˆ(k),νˆ(k), ς(k)) with
ˆ
f(ˆx,ν,ˆ w, ςˆ ) :=Πx( ˆArˆx+ ˆErϕ( ˆFrxˆ) + ˆDrwˆ+ ˆBrνˆ+ ˆRrς), (12) where Πx: ˆXr→Xˆ is the map that assigns to any ˆxr∈Xˆr, the representative
point ¯x∈Xˆ of the corresponding partition set containing ˆxr. The output map
ˆ
h(ˆx) = ˆCxˆ. The initial state of Σb is also selected according to ˆx0 :=Πx(ˆxr(0))
with ˆxr(0) being the initial state ofΣbr.
Abstraction mapΠx satisfies the inequality kΠx(ˆxr)−xˆrk ≤β for all ˆxr ∈
ˆ
Xr, where β is the state discretization parameter defined as β := sup{kxˆr−
ˆ
x0rk, xˆr,xˆr0∈Xi, i= 1,2, . . . , nx}.
5.2 Establishing Probabilistic Relations Here, we candidate relations
Rx= n (x,xˆ)|(x−Pˆx)TM(x−Pxˆ)≤2 o , (13) Rw= n (w,wˆ)|(w−Pwwˆ)TMw(w−Pwwˆ)≤2w o , (14)
where P ∈ Rn׈n and Pw ∈ Rm×mˆ are matrices of appropriate dimensions (potentially with the lowest ˆn and ˆm), and M, Mw are some positive-definite matrices.
Next theorem gives conditions for havingΣb δ Σ with relations (13) and (14).
Theorem 3. LetΣ= (A, B, C, D, E, F, R, ϕ)andΣbr= ( ˆAr,Bˆr,Cˆr,Dˆr,Eˆr,Fˆr,Rˆr, ϕ)
be two nonlinear systems with the same additive noise. Suppose Σb is a finite gMDP constructed from Σbr (cf. Subsection 5.1). Then Σb is (, δ)-stochastically simulated byΣ with relations (13)-(14)if there exist matricesK,Q,S,L1,L2 andR˜ such that
MCTC, (15) ˆ Cr=CP, (16) ˆ Fr=F P, (17) E=PEˆr−B(L1−L2), (18) AP =PAˆr−BQ, (19) DPw=PDˆr−BS, (20) P{(H+P G)TM(H+P G)≤2} 1−δ, (21) where H= ((A+BK)+ ¯δ(BL1+E)F)(x−Pxˆ)+D(w−Pwwˆ)+(BR˜−PBˆr)ˆν+(R−PRˆr)ς, G= ˆArxˆ+ ˆErϕ( ˆFrˆx)+ ˆDrwˆ+ ˆBrˆν+ ˆRrς−Πx( ˆArˆx+ ˆErϕ( ˆFrxˆ)+ ˆDrwˆ+ ˆBrνˆ+ ˆRrς).
6
Case Study
We applied our results to a network of four stochastic nonlinear systems (totally 12 dimensions). We constructed finite gMDPs from their reduced-order versions (together 4 dimensions). We guaranteed that the distance between outputs ofΣ
and ofΣbwould not exceed= 6 during the time horizonT = 10 with probability at least 97% (γ= 0.03).
References
1. C. Baier and J.-P. Katoen,Principles of model checking. MIT press, 2008. 2. K. G. Larsen and A. Skou, “Bisimulation through probabilistic testing,”
Informa-tion and computaInforma-tion, vol. 94, no. 1, pp. 1–28, 1991.
3. R. Segala and N. Lynch, “Probabilistic simulations for probabilistic processes,”
Nordic Journal of Computing, vol. 2, no. 2, pp. 250–273, 1995.
4. J. Desharnais, F. Laviolette, and M. Tracol, “Approximate analysis of probabilistic processes: Logic, simulation and games,” in Proceedings of the 5th international conference on quantitative evaluation of system, 2008, pp. 264–273.
5. A. D’Innocenzo, A. Abate, and J. Katoen, “Robust PCTL model checking,” in
Proceedings of the 15th ACM international conference on Hybrid Systems: Com-putation and Control, 2012, pp. 275–286.
6. A. A. Julius and G. J. Pappas, “Approximations of stochastic hybrid systems,”
IEEE Transactions on Automatic Control, vol. 54, no. 6, pp. 1193–1203, 2009. 7. M. Zamani, P. Mohajerin Esfahani, R. Majumdar, A. Abate, and J. Lygeros,
“Sym-bolic control of stochastic systems via approximately bisimilar finite abstractions,”
IEEE Transactions on Automatic Control, vol. 59, no. 12, pp. 3135–3150, 2014. 8. J. Desharnais, V. Gupta, R. Jagadeesan, and P. Panangaden, “Metrics for labelled
markov processes,”Theoretical computer science, vol. 318, no. 3, pp. 323–354, 2004. 9. A. Abate, “Approximation metrics based on probabilistic bisimulations for general state-space markov processes: a survey,”Electronic Notes in Theoretical Computer Science, vol. 297, pp. 3–25, 2013.
10. A. Abate, M. Kwiatkowska, G. Norman, and D. Parker, “Probabilistic model check-ing of labelled markov processes via finite approximate bisimulations,” inHorizons
of the Mind. A Tribute to Prakash Panangaden. Springer, 2014, pp. 40–58.
11. A. Abate, M. Prandini, J. Lygeros, and S. Sastry, “Probabilistic reachability and safety for controlled discrete-time stochastic hybrid systems,”Automatica, vol. 44, no. 11, pp. 2724–2734, 2008.
12. M. Kamgarpour, S. Summers, and J. Lygeros, “Control design for specifications on stochastic hybrid systems,” inProceedings of the 16th International Conference
on Hybrid Systems: Computation and Control, ser. HSCC ’13. New York, NY,
USA: ACM, 2013, pp. 303–312.
13. I. Tkachev and A. Abate, “On infinite-horizon probabilistic properties and stochas-tic bisimulation functions,” inProceedings of the 50th IEEE Conference on
Deci-sion and Control and European Control Conference (CDC-ECC), 2011, pp. 526–
531.
14. S. Soudjani and A. Abate, “Adaptive and sequential gridding procedures for the abstraction and verification of stochastic processes,” SIAM Journal on Applied Dynamical Systems, vol. 12, no. 2, pp. 921–956, 2013.
15. S. Soudjani, C. Gevaerts, and A. Abate, “FAUST2: Formal abstractions of uncountable-state stochastic processes,” inTACAS’15, ser. Lecture Notes in Com-puter Science. Springer, 2015, vol. 9035, pp. 272–286.
16. S. Soudjani, A. Abate, and R. Majumdar, “Dynamic Bayesian networks as for-mal abstractions of structured stochastic processes,” in Proceedings of the 26th
International Conference on Concurrency Theory, 2015, pp. 1–14.
17. A. Lavaei, S. Soudjani, R. Majumdar, and M. Zamani, “Compositional abstractions of interconnected discrete-time stochastic control systems,” in Proceedings of the
56th IEEE Conference on Decision and Control, 2017, pp. 3551–3556.
18. A. Lavaei, S. Soudjani, and M. Zamani, “Compositional construction of infinite abstractions for networks of stochastic control systems,” Automatica, accepted, 2019.
19. A. Lavaei, S. Soudjani, and M. Zamani, “From dissipativity theory to compositional construction of finite Markov decision processes,” inProceedings of the 21st ACM
International Conference on Hybrid Systems: Computation and Control, 2018, pp.
21–30.
20. A. Lavaei, S. Soudjani, and M. Zamani, “Compositional synthesis of finite abstrac-tions for continuous-space stochastic control systems: A small-gain approach,” in
Proceedings of the 6th IFAC Conference on Analysis and Design of Hybrid Systems, vol. 51, no. 16, 2018, pp. 265–270.
21. A. Lavaei, S. Soudjani, and M. Zamani, “Compositional synthesis of large-scale stochastic systems: A relaxed dissipativity approach,”arXiv:1902.01223v2, Febru-ary 2019.
22. S. Haesaert, S. Soudjani, and A. Abate, “Verification of general Markov decision processes by approximate similarity relations and policy refinement,”SIAM Jour-nal on Control and Optimization, vol. 55, no. 4, pp. 2333–2367, 2017.
23. S. Haesaert, S. Soudjani, and A. Abate, “Temporal logic control of general markov decision processes by approximate policy refinement,” IFAC-PapersOnLine, vol. 51, no. 16, pp. 73 – 78, 2018, 6th IFAC Conference on Analysis and Design of Hybrid Systems ADHS 2018.
24. S. Soudjani, “Formal abstractions for automated verification and synthesis of stochastic systems,” Ph.D. dissertation, Technische Universiteit Delft, The Nether-lands, 2014.
