Compositional Abstraction-based Synthesis of General MDPs via Approximate Probabilistic Relations

COMPOSITIONAL ABSTRA CTION-BASED SYNTHESIS OF GENERAL MDPS VIA APPR O XIMA TE PROBABILISTIC RE LA TIONS ABOLF AZL LA V AEI 1 , SADEGH SOUDJANI 2 , AND MAJID ZAMANI 3 , 4 Abstract. W e propose a compositional approac h for constructing abstractions of general Marko v decision processes using approximate probabilistic relations. The abstraction framewo rk i s based on the notion of δ - lifted relations, using which one can quan tify the distance in probabili ty betw een the intercon nected gMDPs and that of their abstractions. This new approximate relation unifies compositionality results in the literature b y incorp orating the dep endencies b etw een state transitions explicitly and by allowing abstract m odels to ha ve either finite or infinite state spaces. Accordingly , one can lev erage the proposed results to p erform analysis and synthe sis ov er abstract mo dels, and then carry the r esults ov er concrete ones. T o this end, we first prop ose our comp ositionality results using the new approximate probabilistic relation which i s based on lifting. W e then fo cus on a class of sto cha stic nonlinear dynamical systems and construct their abstractions using both m odel order reduction and space discretization i n a unified fr amew ork. W e provide conditions for simultane ous existence of r elations incorpor ating the structure of the netw ork. Finally , we demonstrate the effectiv eness of the proposed resul ts b y considering a netw ork of four nonlinear dynamical subsystems (toget her 12 dimensions) and constructing finite abstractions fr om their reduced-order v ersions (toge ther 4 dimensions) in a unified compositional framew ork. W e benc hmark our results against the comp ositional abstraction tec hniques that construct b oth infinite abstractions (reduced-order mo dels) and finite MDPs in t wo consecutiv e steps. W e show that our approac h i s m uch less conserv ative than the ones av ai l able in the literature. 1. In tr oduction Motiv ations. Co nt rol systems with stochastic uncertain ty can b e mo deled as Marko v decision proces s es (MDPs) ov er g eneral state spaces. Synt hesizing p o licies for sa tisfying complex temp or a l log ic prop erties ov er MDPs evolving on uncoun table state spaces is inher ently a challenging ta sk due to the co mputational complexity . Since clos ed-form characterization of s uch p olicies is not av a ilable in general, a suitable approach is to a pproximate these mo dels by simpler ones p oss ibly with finite or low er dimensional state spaces. A crucial step is to provide formal guara nt ees dur ing this approximation phase, such that the ana lysis or synthesis on the s impler mo del can b e refined back ov e r the origina l one. In other words, o ne can fir s t abstra ct the orig inal mo del b y a simpler one, and then carry the results from the simpler model to the concrete one using an int erface map, by pr oviding quantified error s on the appr oximation. Related li terature. Similar it y relations over finite-s ta te sto chastic systems hav e b een studied, either v ia ex- act notions of probabilistic (bi)simulation relatio ns [LS91], [SL95] or approximate versions [DL T08 ], [DAK12]. Similarity re la tions for mo dels with g eneral, uncountable s tate spaces hav e also b een prop osed in the liter- ature. These relatio ns either depend on sta bilit y requiremen ts o n model outputs via martingale theory or contractivit y analysis [JP09], [ZME M + 14] o r enfor ce structural abstractions o f a mo del [DGJP04] by explo it- ing contin uity conditions on its probability laws [Aba1 3], [AKNP14]. These similarity relations are then used to rela te the proba bilistic b ehavior of a concr e te mo del to that of its abstr action. Ther e hav e be e n also several results o n the c o nstruction of (in)finite abstractio ns for s to chastic sys tems. Co nstruction o f finite abstractio ns for forma l verification and synthesis is pres ent ed in [APLS08]. Ex tens ion of such techniques to a uto mata-based controller synthesis a nd infinite hor izon prop erties, and improv ement o f the construction algorithms in terms of scalability are prop o sed in [KSL1 3], [T A11], and [SA13], resp ectively . 1 2 ABOLF AZL LA V AEI 1 , S ADEGH S OUDJANI 2 , AND M AJID ZAMANI 3 , 4 In o rder to make the techniques applicable to netw orks o f interacting systems, comp ositio na l abstr a ction and po licy synthesis ar e studied in the literature. Compos itional co nstruction o f finite abs tractions using dy na mic Bay esian netw o rks is discussed in [SAM15]. Compo s itional construction o f infinite a bstractions (reduced-order mo dels) is prop osed in [LSMZ17, LSZ19a] using small-gain type conditions and dissipativity-t yp e pr op erties of subsys tems a nd their abs tractions, re sp ectively . Comp ositiona l co nstruction of finite abstra ctions is studied in [LSZ18a, LSZ18b]. Comp ositiona l mo deling and analysis for the sa fet y v e r ification o f sto chastic hybrid systems are inv estigated in [HHHK1 3] in which ra ndom b ehaviour o c c urs o nly ov er the discrete comp onents – this limits their applicability to systems with contin uous pr obabilistic evolutions. Compos itio nal mo deling of sto chastic h ybrid s ystems is discussed in [Sv06] using communicating piecewise deterministic Markov process es that are connected through a comp os ition o pe r ator. R ecently , comp ositio nal synthesis of large- s cale sto chastic systems using a relax ed dissipativity a pproach is prop osed in [LSZ19 b]. Our C o n tributions. In our pr op osed framework, we consider the class of ge ne r al Mar ko v decisions pro cesses (gMDPs), which evolves ov er contin uous or uncountable state spaces , equipped with an o utput space and an output map. W e enco de interaction b etw een gMDPs via internal inputs, as opp os ed to ext ernal inputs which are used for applying the syn thesized p olicie s enfor cing so me co mplex temp or al logic prop er ties. W e provide conditions under which the pr op osed similar ity r elations b etw een individual g MDPs can b e extended to relations betw een their re sp ective int erco nnections. These conditions ena ble co mpo sitional quantification of the distance in probability betw een the interconnected gMDPs and that of their abstra ctions. The prop osed notion has the a dv antage of enco ding prior knowledge on depe ndenc ie s betw een uncertainties of the two models. Our comp o s itional scheme allows constructing b oth infinite and fi nite abstractions in a unified framework. W e benchmark o ur r esults ag ainst the comp ositiona l a bs traction techniques o f [LSZ1 8b, LSZ19 a] which are based o n dissipativity-t yp e re asoning and provide a comp ositio nal metho dology for co nstructing b oth infinite abstractions (reduced-o rder mo dels) and finite MDPs in tw o consecutive steps. W e show that our appr oach is m uch less co nserv a tive than the ones prop os ed in [LSZ1 8 b, LSZ19a]. Recen t W orks. Similarities b etw een t wo gMDPs hav e b een rece ntly studied in [HSA17] using a notion of δ -lifted rela tion, but only for single g MDPs. The result is g eneralized in [HSA18 ] to a large r cla ss of tempo ral pro p erties and in [HS18] to s ynth esize policies for robus t satisfaction of s p e c ifications. One of the main contributions o f this pa p er is to extend this no tion such that it can b e applied to net works of gMDPs. This extensio n is inspired by the notio n of disturba nce bisimulation rela tion prop ose d in [MSSM16]. In particular, we extend the notion of δ -lifted relation for netw orks of gMDPs and show that under spe c ific conditions systems ca n be comp ose d while preserving the r elation. This type of relations enables us to provide the pro babilistic closeness gua rantee b etw een tw o interconnected gMDPs (cf. Theorem 3.5). F urthermore , we provide an approach for the constr uc tio n of finite MDPs in a unified framework for a class o f sto chastic nonlinear dynamical systems, cons ide r ed a s gMDPs, whereas the constr uc tio n scheme in [HSA17] o nly ha ndles the class of linear systems. Organization. The r est of the pap er is or ganized as follows. Section 2 defines the c la ss of general Markov decision pro cesses with internal inputs and o utput maps. Section 3 pr esents fir st the notion of δ -lifted r elations ov er pro bability spaces and then the notion of lifting for gMDPs. Section 4 provides comp o s itional conditions for ha ving the simila rity relation betw een net works of gMDPs based on rela tions betw een their individual comp onents. Section 5 provides details of constructing finite abs tractions for a netw ork of sto chastic no nlinear control systems, which is based on b oth mo del order r eduction a nd s pa ce discretization in a unified framework, together with the s imilarity r elations. Finally , Section 6 demonstrates the effectiveness of o ur appro ach on a nu merica l cas e study . 2. General Marko v Decision Processes 2.1. Preliminaries and Notations. In this pap er, we w ork on Borel measurable spac e s , i.e., ( X , B ( X )), where B ( X ) is the Bo rel s igma alg ebra on X , and restrict o urselves to Polish spaces (i.e., s eparable and completely metrizable spa ces). Giv en the meas urable spa ce ( X, B ( X )), a pro bability measure P defines the COMPOSITIONAL ABSTRACT ION-BASED SYNTHE S IS OF MDPS VIA APPRO XIM A TE P ROB ABILISTIC RELA TIONS 3 probability spa ce ( X, B ( X ) , P ). W e denote the set of all pro bability measures on ( X , B ( X )) as P ( X, B ( X )). A map f : S → Y is measurable whenever it is Borel measur able. The sets of nonnegative and po sitive in tegers, and real num b ers ar e denoted by N := { 0 , 1 , 2 , . . . } , N ≥ 1 := { 1 , 2 , 3 , . . . } , a nd R , resp ectively . F o r column vectors x i ∈ R n i , n i ∈ N ≥ 1 , and i ∈ { 1 , . . . , N } , we denote by x = [ x 1 ; . . . ; x N ] the cor resp onding column vector o f dimens io n P i n i . Giv en a vector x ∈ R n , k x k denotes the Euclidean norm of x . The identit y and zero ma trices in R n × n are denoted by I n and 0 n × n , resp ectively . The symbols 0 n and 1 n denote the column vector in R n with all elements equal to zero and one, resp ectively . A dia gonal matr ix in R N × N with diag onal entries a 1 , . . . , a N starting from the upper left co rner is denoted by diag ( a 1 , . . . , a N ). Given functions f i : X i → Y i , for any i ∈ { 1 , . . . , N } , their Cartesian pro duct Q N i =1 f i : Q N i =1 X i → Q N i =1 Y i is defined as ( Q N i =1 f i )( x 1 , . . . , x N ) = [ f 1 ( x 1 ); . . . ; f N ( x N )]. Given sets X and Y , a relation R ⊆ X × Y is a subset of the Cartesia n pro duct X × Y that rela tes x ∈ X with y ∈ Y if ( x, y ) ∈ R , which is equiv ale ntly deno ted by x R y . 2.2. General Mark ov Decisio n Pro cesse s. In our fra mework, we consider the class of g e neral Mar ko v decision pro cesses (g MDPs) tha t evolves o ver contin uous or uncountable state spaces. This cla ss of mo dels generalizes the usual notion of MDP [BKL08] by including internal inputs that a re employed for comp osi- tion [LSZ18b], and by adding an output spac e over which prop erties of int erest are defined [HSA17]. Definition 2 .1. A gener al Markov de cision pr o c ess (gMDP) is a tuple Σ = ( X , W, U, π , T , Y , h ) (2.1) wher e • X ⊆ R n is a Bor el sp ac e as the state sp ac e of the system. We denote by ( X , B ( X )) the me asur able sp ac e with B ( X ) b eing the Bor el sigma-algebr a on the s tate sp ac e; • W ⊆ R p is a Bor el s p ac e as the internal input sp ac e of t he syst em; • U ⊆ R m is a Bor el s p ac e as the ex ter nal input sp ac e of the s yst em; • π = B ( X ) → [0 , 1] is the initial pr ob ability distribution; • T : B ( X ) × X × W × U → [0 , 1] is a c onditional sto chastic kernel that assigns to any x ∈ X , w ∈ W , and ν ∈ U , a pr ob ability me asur e T ( ·| x, w, ν ) on the me asur able sp ac e ( X , B ( X )) . This sto chastic kernel sp e cifies pr ob abilities over exe cutions { x ( k ) , k ∈ N } of t he gMDP such that for any set A ∈ B ( X ) and any k ∈ N , P ( x ( k + 1) ∈ A    x ( k ) , w ( k ) , ν ( k )) = Z A T ( dx ( k + 1) | x ( k ) , w ( k ) , ν ( k )) . • Y ⊆ R q is a Bor el s p ac e as the output sp ac e of t he system; • h : X → Y is a me asur able funct ion t hat m aps a state x ∈ X to its output y = h ( x ) . Remark 2 .2. In this work, we ar e inter este d in networks of gMDPs that ar e obtaine d fr om c omp osing gMDPs having b oth internal and extern al inputs and ar e synchr onize d thr ough their internal inputs. The r esulting inter c onne cte d gMDP wil l have only external input and wil l b e denote d by t he tuple Σ = ( X , U, π , T , Y , h ) with sto chastic kernel T : B ( X ) × X × U → [0 , 1] . Evolution of the state of a gMDP Σ, can b e alterna tively descr ib ed by Σ :  x ( k + 1) = f ( x ( k ) , w ( k ) , ν ( k ) , ς ( k )) , y ( k ) = h ( x ( k )) , k ∈ N , x (0) ∼ π , (2.2) for input sequences w ( · ) : N → W and ν ( · ) : N → U , w her e ς := { ς ( k ) : Ω → V ς , k ∈ N } is a sequence o f independent a nd iden tically distr ibuted (i.i.d.) ra ndom v ar iables on a se t V ς with sample spa c e Ω. V ector field f together with the distributio n of ς provide the sto chastic k ernel T . The sets W a nd U are, resp ectively , a sso ciated to W and U , c o llections o f sequences { w ( k ) : Ω → W, k ∈ N } and { ν ( k ) : Ω → U, k ∈ N } , in which w ( k ) and ν ( k ) ar e indep endent of ς ( t ) for a ny k , t ∈ N a nd t ≥ k . F or 4 ABOLF AZL LA V AEI 1 , S ADEGH S OUDJANI 2 , AND M AJID ZAMANI 3 , 4 any initial s tate a ∈ X , w ( · ) ∈ W , ν ( · ) ∈ U , the r andom sequence y awν : Ω × N → Y satis fying (2.2) is called the out put tr aje ctory o f Σ under initial state a , internal input w , and external input ν . W e eliminate subsc ript of y awν wherever it is known from the context. If X , W, U are finite sets , system Σ is ca lled finite, and infinite otherwise. Next section presents appr oximate probabilis tic relations that can b e used for relating tw o g MDP s while capturing probabilistic dependency b etw een their executions. This new relation enables us to comp ose a s et of concrete g MDPs and that of their abstr actions while pr oviding conditions for preserving the relation after comp osition. 3. A ppr oxima te P robabilistic Rela tions based on Lifting In this section, we first introduce the notion of δ - lifted rela tions ov er g eneral state space s. W e then define ( ǫ, δ )- approximate proba bilis tic relations based on lifting fo r gMDPs with internal inputs. Finally , we define ( ǫ, δ )- approximate r elations for interconnected g MDPs without internal input res ulting from the int erco nnection of gMDPs having b o th internal and externa l inputs. First, we provide the notion of δ -lifted relation bo rrowed from [HSA17]. Definition 3.1. L et X , ˆ X b e two sets with asso ciate d me asur able sp ac es ( X , B ( X )) and ( ˆ X , B ( ˆ X )) . Consider a r elation R x ∈ B ( X × ˆ X ) . We denote by ¯ R δ ⊆ P ( X, B ( X )) × P ( ˆ X , B ( ˆ X )) , the c orr esp onding δ -lifte d r elation if ther e exists a pr ob ability sp ac e ( X × ˆ X , B ( X × ˆ X ) , L ) ( e quivalently, a lifting L ) such that (Φ , Θ) ∈ ¯ R δ if and only if • ∀A ∈ B ( X ) , L ( A × ˆ X ) = Φ( A ) , • ∀ ˆ A ∈ B ( ˆ X ) , L ( X × ˆ A ) = Θ( ˆ A ) , • for t he pr ob ability sp ac e ( X × ˆ X , B ( X × ˆ X ) , L ) , it holds t hat x R x ˆ x with pr ob ability at le ast 1 − δ , e quivalently, L ( R x ) ≥ 1 − δ . F or a given re lation R x ⊆ X × ˆ X , the ab ove definition sp ecifies requir ed prop erties for lifting r elation R x to a relation ¯ R δ that r elates probability meas ures over X and ˆ X . W e a re interested in using δ -lifte d relation for s pec ifying similarities b etw een a gMDP and its a bstraction. Therefore, internal inputs of the t wo gMDPs should b e in a rela tion deno ted b y R w . Next definition g ives conditions for having a sto chastic s im ulation relatio n b etw e en tw o gMDPs . Definition 3.2. Consider gMDPs Σ = ( X , W, U, π , T , Y , h ) and b Σ = ( ˆ X , ˆ W , ˆ U , ˆ π , ˆ T , Y , ˆ h ) with t he same output sp ac e. System b Σ is ( ǫ, δ )-sto chastic al ly simulate d by Σ , i.e. b Σ  δ ǫ Σ , if ther e exist r elations R x ⊆ X × ˆ X and R w ⊆ W × ˆ W for which ther e exists a Bor el m e asur able sto chastic kernel L T ( · | x, ˆ x, w , ˆ w , ˆ ν ) on X × ˆ X su ch that • ∀ ( x, ˆ x ) ∈ R x , k h ( x ) − ˆ h ( ˆ x ) k ≤ ǫ , • ∀ ( x, ˆ x ) ∈ R x , ∀ ˆ w ∈ ˆ W , ∀ ˆ ν ∈ ˆ U , ther e exists ν ∈ U such that ∀ w ∈ W with ( w , ˆ w ) ∈ R w , T ( · | x, w , ν ) ¯ R δ ˆ T ( · | ˆ x , ˆ w, ˆ ν ) with lifting L T ( · | x, ˆ x, w , ˆ w , ˆ ν ) , • π ¯ R δ ˆ π . Second condition of Definition 3.2 implies implicitly that there exis ts a function ν = ν ( x, ˆ x, ˆ w , ˆ ν ) such that the state probability meas ur es are in the lifted relation a fter one tra ns ition for any ( x, ˆ x ) ∈ R x , ˆ w ∈ ˆ W , and ˆ ν ∈ ˆ U . This function is ca lled the interfac e function , which can b e employ ed for re fining a s y nt hesized p olicy ˆ ν for b Σ to a polic y ν for Σ. COMPOSITIONAL ABSTRACT ION-BASED SYNTHE S IS OF MDPS VIA APPRO XIM A TE P ROB ABILISTIC RELA TIONS 5 Remark 3.3. Definition 3.2 extends appr oximate pr ob abilistic r elation in [HSA17] by adding r elation R w to c aptur e the effe ct of internal inputs. Interfac e function ν = ν ˆ ν ( x, ˆ x, ˆ w , ˆ ν ) is also al lowe d t o dep end on t he internal input of the abstr act gMDP b Σ . Remark 3.4. Note that Defi n ition 3.2 gener alizes the r esu lts of [LSMZ17] , that assumes indep endent noises in two similar gMDPs, and of [LSZ18b] , that assumes shar e d n oises, by making no p articular assu mption but r e quiring this dep endency to b e r efle cte d in lifting L T . We emphasize that this gener alization is c onsider e d only for a c oncr ete gMDP and its abstr action. We stil l r etain t he assumption of indep endent u nc ertainties b etwe en gMDPs in a network (cf. Defin ition 4.1 and R emark 4.2). Definition 3.2 can b e applied to gMDPs without internal inputs that may a r ise from comp osing gMDP s v ia their internal inputs. F or such gMDPs, we eliminate R w and interface function b ecomes indep endent of int ernal input, thus the definitio n reduces to tha t of [HSA17], provided in the App endix as Definition 9.1 . Figure 1 illustrates ingredients of Definition 3.2. As seen, rela tion R w and sto chastic kernel L T capture the effect of internal inputs, and the r e lation of tw o noises, resp ectively . Mo reov er, interface function ν ˆ ν ( x, ˆ x, ˆ w , ˆ ν ) is employ ed to re fine a sy nthesized polic y ˆ ν for b Σ to a polic y ν for Σ. Figure 1. Notion of lifting for sp ecifying the s imilarity b etw een gMDP and its abstr a ction. Relations R x and R w are the o nes b etw een states and internal inputs, res pec tively . L T sp ec- ifies the relation o f tw o noises, and interface function ν ˆ ν ( x, ˆ x, ˆ w , ˆ ν ) is used for the refinement po licy . Definition 3 .2 enables us to quantify the e rror in probability b etw een a conc r ete sys tem Σ and its abs traction b Σ. In an y ( ǫ, δ )- approximate probabilistic relation, δ is used to quantify the distance in pro ba bility betw een gMDPs and ǫ for the closeness of output tra jectories as stated in the next theorem. Theorem 3 .5. If b Σ  δ ǫ Σ and ( w ( k ) , ˆ w ( k )) ∈ R w for al l k ∈ { 0 , 1 , . . . , T k } , then for al l p olicies on b Σ ther e exists a p olicy for Σ such that, for al l me asur able events A ⊂ Y T k +1 , P {{ ˆ y ( k ) } 0: T k ∈ A − ǫ } − γ ≤ P {{ y ( k ) } 0: T k ∈ A } ≤ P {{ ˆ y ( k ) } 0: T k ∈ A ǫ } + γ , (3.1) with c onstant 1 − γ := (1 − δ ) T k +1 , and with the ǫ -exp ansion and ǫ -c ontr action of A define d as A ǫ := { y ( · ) ∈ Y T k +1   ∃ ¯ y ( · ) ∈ A with max k ≤ T k k ¯ y ( k ) − y ( k ) k ≤ ǫ } , A − ǫ := { y ( · ) ∈ A   ¯ y ( · ) ∈ A for al l ¯ y ( · ) with max k ≤ T k k ¯ y ( k ) − y ( k ) k ≤ ǫ } . W e ha ve adapted this theor em fro m [HSA17] and a dded its pr o of in the App endix for the sake of co mpleteness. W e employ this theorem to pr ovide the pr obabilistic closeness guarantee b etw een in terconnected gMDPs and that of their comp ositional abstr actions w hich a re discussed in Section 4. In the next section, we define comp osition of g MDPs via their internal inputs and discuss how to relate them to a netw o r k of interconnected a bstraction based on their individual relatio ns. 6 ABOLF AZL LA V AEI 1 , S ADEGH S OUDJANI 2 , AND M AJID ZAMANI 3 , 4 Figure 2. Interconnection of tw o gMDPs Σ 1 and Σ 2 and that of their abstra ctions. 4. In ter connected gMDPs and Their Compositional Abstractions 4.1. In te rconnected gMDPs. Let Σ b e a netw o rk o f N ∈ N ≥ 1 gMDPs Σ i = ( X i , W i , U i , π i , T i , Y i , h i ) , i ∈ { 1 , . . . , N } . (4.1) W e partition internal input and o utput of Σ i as w i = [ w i 1 ; . . . ; w i ( i − 1) ; w i ( i +1) ; . . . ; w iN ] , y i = [ y i 1 ; . . . ; y iN ] , (4.2) and also output space and function as h i ( x i ) = [ h i 1 ( x i ); . . . ; h iN ( x i )] , Y i = N Y j =1 Y ij . (4.3) The outputs y ii are denoted as external ones, whereas the o utputs y ij with i 6 = j as internal ones w hich ar e employ ed for interconnection by requiring w j i = y ij . This can b e explicitly wr itten using appropria te functions g i defined as w i = g i ( x 1 , . . . , x N ) :=  h 1 i ( x 1 ); . . . ; h ( i − 1) i ( x i − 1 ); h ( i +1) i ( x i +1 ); . . . ; h N i ( x N )  . (4.4) If ther e is no co nnection fro m Σ i to Σ j , then the co nnecting o utput function is iden tically zero for all arguments, i.e., h ij ≡ 0. Now, we define the inter c onne cte d gMDP Σ as follows. Definition 4. 1. Consider N ∈ N ≥ 1 gMDPs Σ i = ( X i , W i , U i , π i , T i , Y i , h i ) , i ∈ { 1 , . . . , N } , with t he input - output c onfigur ation as in (4.2) and (4.3) . The inter c onne ction of Σ i , i ∈ { 1 , . . . , N } , is a gMDP Σ = ( X, U, π , T , Y , h ) , denote d by I (Σ 1 , . . . , Σ N ) , such that X := Q N i =1 X i , U := Q N i =1 U i , Y := Q N i =1 Y ii , and h = Q N i =1 h ii , with t he fol lowing c onstr aints: ∀ i, j ∈ { 1 , . . . , N } , i 6 = j : w j i = y ij , Y ij ⊆ W j i . (4.5) Mor e over, one has c onditional st o chastic kernel T := Q N i =1 T i and initial pr ob ability distribution π := Q N i =1 π i . An exa mple of the interconnection of tw o gMDPs Σ 1 and Σ 2 and that of their abstr a ctions is illustra ted in Figure 2. COMPOSITIONAL ABSTRACT ION-BASED SYNTHE S IS OF MDPS VIA APPRO XIM A TE P ROB ABILISTIC RELA TIONS 7 Remark 4.2. Definition 4.1 assumes that unc ertainties affe cting individual gMDPs in a network I (Σ 1 , . . . , Σ N ) ar e indep endent and, thus, c onstruct s T and π by taking pr o duct s of T i and π i , r esp e ctively. This definition c an b e gener alize d for dep endent unc ertainties by using their joint distribut ion in the c onstruction of T and π , in the same manner as we discusse d in R emark 3.4 for expr essing dep en dent u nc ertainties in c oncr ete and abstr act gMDPs. 4.2. Comp osi tional Abstractions for In terconnected gMDPs. W e assume that we ar e given N gMDPs as in Definition 2.1 together with their cor resp onding abstractio ns b Σ i = ( ˆ X i , ˆ W i , ˆ U i , ˆ π i , ˆ T i , Y i , ˆ h i ) such that b Σ i  δ i ǫ i Σ i for some r e lation R x i and consta nt s ǫ i , δ i . Next theorem shows the main comp ositionality result of the pap er. Theorem 4.3 . Consider t he inter c onne cte d gMDP Σ = I (Σ 1 , . . . , Σ N ) induc e d by N ∈ N ≥ 1 gMDPs Σ i . Supp ose b Σ i is ( ǫ i , δ i )-sto chastic al ly simulate d by Σ i with the c orr esp onding re lations R x i and R w i and lifting L i . If g i ( x ) R w i ˆ g i ( ˆ x ) , ∀ ( x, ˆ x ) ∈ R x i , (4.6) with int er c onne ction c onstr aint m aps g i , ˆ g i define d as in (4.4 ) , t hen b Σ = I ( b Σ 1 , . . . , b Σ N ) is ( ǫ, δ ) - sto chastic al ly simulate d by Σ = I (Σ 1 , . . . , Σ N ) with r elation R x define d as    x 1 . . . x N    R x    ˆ x 1 . . . ˆ x N    ⇔      x 1 R x 1 ˆ x 1 , . . . x N R x N ˆ x N , and c onstants ǫ = P N i =1 ǫ i , and δ = 1 − Q N i =1 (1 − δ i ) . Lifting L and interfac e ν ar e obtaine d by taking pr o ducts L = Q N i =1 L i and ν = Q N i =1 ν i , and then substitu ting inter c onne ction c onstr aints (4.5) . The pro of of Theorem 4.3 is provided in the App endix. Remark 4. 4. Note that The or em 4.3 r e quir es g i ( x ) R w i ˆ g i ( ˆ x ) for any ( x, ˆ x ) ∈ R x . This c ondition puts r e- striction on the structu r e of the network and how t he dynamics of gMDPs ar e c ouple d in the network (cf. R emark 3.3 ). It is similar t o the c ondition imp ose d in disturb anc e bisimulation r elation define d in [MSSM1 6] . W e provide the following e xample to illustrate our comp ositio na lity res ults. Example 4. 5. Assum e that we ar e given two line ar dynamic al systems as Σ i :  x i ( k + 1) = A i x i ( k ) + D i w i ( k ) + B i ν i ( k ) + R i ς i ( k ) , y i ( k ) = x i ( k ) , i ∈ { 1 , 2 } , (4.7) wher e the additive noise ς i ( · ) is a se quenc e of indep endent r andom ve ctors with multivariate standar d normal distributions for i ∈ { 1 , 2 } , and R i , i ∈ { 1 , 2 } , ar e invertible. Le t b Σ i b e the abstr action of gMDP (4.7) as b Σ i :  ˆ x i ( k + 1) = ˆ A i ˆ x i ( k ) + ˆ D i ˆ w i ( k ) + ˆ B i ˆ ν i ( k ) + ˆ R i ˆ ς i ( k ) , ˆ y i ( k ) = ˆ x i ( k ) . T r ansition kernels of Σ i and b Σ i c an b e written as T i ( ·| x i , w i , ν i ) = N ( ·| A i x i + D i w i + B i ν i , R i R T i ) , ˆ T i ( ·| ˆ x i , ˆ w i , ˆ ν i ) = N ( ·| ˆ A i ˆ x i + ˆ D i ˆ w i + ˆ B i ˆ ν i , ˆ R i ˆ R T i ) , ∀ i ∈ { 1 , 2 } , wher e N ( · | m , D ) indic ates normal distribution with me an m and c ovarianc e m atrix D . Indep endent unc ertai nties. If ς i ( · ) and ˆ ς i ( · ) in the c oncr ete and abstr act systems ar e indep endent, a c andidate for lifte d me asur e is L T i ( ·| x i , ˆ x i , w i , ˆ w i , ˆ ν i ) = N ( ·| A i x i + D i w i + B i ν i , R i R T i ) × N ( ·| ˆ A i ˆ x i + ˆ D i ˆ w i + ˆ B i ˆ ν i , ˆ R i ˆ R T i ) . 8 ABOLF AZL LA V AEI 1 , S ADEGH S OUDJANI 2 , AND M AJID ZAMANI 3 , 4 Now we c onne ct two subsystems with e ach other b ase d on the inter c onne ction c onstr aint (4.5) which ar e w i = x 3 − i and ˆ w i = ˆ x 3 − i for i ∈ { 1 , 2 } . F or any x = [ x 1 ; x 2 ] ∈ X , ˆ x = [ ˆ x 1 ; ˆ x 2 ] ∈ ˆ X , ν = [ ν 1 ; ν 2 ] ∈ U, ˆ ν = [ ˆ ν 1 ; ˆ ν 2 ] ∈ ˆ U , the c omp ositional t ra nsition kernels for the inter c onne cte d gMDPs ar e T ( · | x, ν ) = N ( · | Ax + B ν , R R T ) , ˆ T ( · | ˆ x , ˆ ν ) = N ( · | ˆ A ˆ x + ˆ B ˆ ν , ˆ R ˆ R T ) , wher e ν := ν ( x, ˆ x , ˆ ν ) and A =  A 1 D 1 D 2 A 2  , B = diag ( B 1 , B 2 ) , R = diag ( R 1 , R 2 ) , ˆ A =  ˆ A 1 ˆ D 1 ˆ D 2 ˆ A 2  , ˆ B = diag ( ˆ B 1 , ˆ B 2 ) , ˆ R = diag ( ˆ R 1 , ˆ R 2 ) . (4.8) Then the c andidate lifte d me asur e for t he inter c onne cte d gMDPs is L T ( ·| x, ˆ x, ˆ ν ) = N ( ·| Ax + B ν , R R T ) N ( ·| ˆ A ˆ x + ˆ B ˆ ν , ˆ R ˆ R T ) . Note that after c onne cting t he subsystems with e ach other using the pr op ose d int er c onne ct ion c onstra int in (4 .5) , the internal input s wil l disap p e ar. Dep endent u nc ertainties. Supp ose Σ i and b Σ i shar e the same noise ς i ( · ) = ˆ ς i ( · ) . In this c ase, the c andidate lifte d me asur e for i ∈ { 1 , 2 } is obtaine d by L T i ( dx ′ i × d ˆ x ′ i | x i , ˆ x i , w i , ˆ w i , ˆ ν i ) = N ( dx ′ i | A i x i + D i w i + B i ν i , R i R T i ) × δ d ( d ˆ x ′ i | ˆ A i ˆ x i + ˆ D i ˆ w i + ˆ B i ˆ ν i + ˆ R i R − 1 i ( x ′ i − A i x i − D i w i − B i ν i )) , wher e δ d ( ·| a ) indic ates Dir ac delta distribution c enter e d at a . Now we c onne ct two su bsyst ems with e ach other. F or any x = [ x 1 ; x 2 ] ∈ X , ˆ x = [ ˆ x 1 ; ˆ x 2 ] ∈ ˆ X , ν = [ ν 1 ; ν 2 ] ∈ U, ˆ ν = [ ˆ ν 1 ; ˆ ν 2 ] ∈ ˆ U , the c andidate lifte d me asur e for the inter c onne cte d gMDPs is L T ( dx ′ × d ˆ x ′ | x, ˆ x, ˆ ν ) = N ( dx ′ | Ax + B ν, RR T ) × δ d ( d ˆ x ′ | A ˆ x + B ˆ ν − ¯ Ax + ˜ Ax ′ − ¯ B ν ) , wher e A, B , R, ˆ A, ˆ B ar e define d as in (4.8) , and ¯ A =  ˆ R 1 R − 1 1 A 1 ˆ R 1 R − 1 1 D 1 ˆ R 2 R − 1 2 D 2 ˆ R 2 R − 1 2 A 2  , ˜ A =  ˆ R 1 R − 1 1 0 0 ˆ R 2 R − 1 2  , ¯ B =  ˆ R 1 R − 1 1 B 1 0 0 ˆ R 2 R − 1 2 B 2  . In the next section, we focus on a particular class of sto chastic no nlinear systems, and construct its infinite and finite a bs tractions in a unified framework. W e provide explicit inequalities for esta blishing Theorem 4.3, which g ives a proba bilis tic relatio n after comp o sition and enables us to get g uarantees of Theorem 3.5 on the closeness of the comp osed system and that of its abstr action. 5. Construction of A bstractions for No nlinear Systems Here, w e fo cus on a sp ecific class of s to chastic no nlinear control systems Σ as Σ :  x ( k + 1) = Ax ( k ) + E ϕ ( F x ( k )) + D w ( k ) + B ν ( k ) + Rς ( k ) , y ( k ) = C x ( k ) , (5.1) where ς ( · ) ∼ N (0 , I n ), and ϕ : R → R satisfies a ≤ ϕ ( c ) − ϕ ( d ) c − d ≤ b, ∀ c, d ∈ R , c 6 = d, (5.2) for some a ∈ R and b ∈ R > 0 ∪ {∞} , a ≤ b . COMPOSITIONAL ABSTRACT ION-BASED SYNTHE S IS OF MDPS VIA APPRO XIM A TE P ROB ABILISTIC RELA TIONS 9 W e use the tuple Σ = ( A, B , C, D , E , F , R , ϕ ) , to refer to the class of nonlinear systems of the form (5.1). Remark 5.1 . If E is a zer o matrix or ϕ in (5.1) is line ar including t he zer o fun ction (i.e. ϕ ≡ 0 ), one c an r emove or push the term E ϕ ( F x ) to Ax , and c onse quent ly the nonline ar tuple r e duc es to the line ar one Σ = ( A, B , C, D , R ) . Then, every t ime we ment ion the tuple Σ = ( A, B , C, D , E , F , R, ϕ ) , it implicitly implies that ϕ is nonline ar and E is nonzer o. Remark 5.2. Without loss of gener ality [AK01] , we c an assum e a = 0 in (5.2) for t he class of nonline ar systems in (5.1) . If a 6 = 0 , one c an define a new function ˜ ϕ ( s ) := ϕ ( s ) − as satisfyi ng (5.2) with ˜ a = 0 and ˜ b = b − a , and r ewrite (5.1) as Σ :  x ( k + 1) = ˜ Ax ( k ) + E ˜ ϕ ( F x ( k )) + D w ( k ) + B ν ( k ) + Rς ( k ) , y ( k ) = C x ( k ) wher e ˜ A = A + aE F . Remark 5.3. We r estrict ourselves her e to syst ems with a single nonline arity as in (5 .1) for the sake of simple pr esentation. However, it would b e str aightforwa r d to get analo gous r esults for s ystems with multiple nonline arities as Σ :  x ( k + 1) = Ax ( k ) + P ¯ M i =1 E i ϕ i ( F i x ( k )) + D w ( k ) + B ν ( k ) + Rς ( k ) , y ( k ) = C x ( k ) , wher e ϕ i : R → R satisfies (5.2) for some a i ∈ R and b i ∈ R > 0 ∪ {∞} , for any i ∈ { 1 , . . . , ¯ M } . Existing co mpo sitional abs traction results for this class of mo dels are based on either mo del or der reduc- tion [LSMZ17], [LSZ1 9a] o r finite MDPs [LSZ18b], [LSZ18 a]. Our prop osed results her e combine these tw o approaches in one unified framework. In other words, o ur abstrac t mo del is obtained by discretizing the s tate space of a reduced-or der version of the co ncrete mo del. 5.1. Construction of Finite Abstractions. Co nsider a nonlinear system Σ = ( A, B , C, D , E , F , R , ϕ ) a nd its reduced-or der version b Σ r = ( ˆ A r , ˆ B r , ˆ C r , ˆ D r , ˆ E r , ˆ F r , ˆ R r , ϕ ). Note that index r in the whole pap er signifies the reduced-or der version of the or iginal mo de l. W e discuss the co ns truction of b Σ r from Σ in Theorem 5.5 of the next subsection. Constructio n of a finite gMDP from b Σ r follows the approa ch of [So u14, SA13]. Denote the state and input spaces of b Σ r resp ectively b y ˆ X r , ˆ W r , ˆ U r . W e construct a finite gMDP b y s electing par titio ns ˆ X r = ∪ i X i , ˆ W r = ∪ i W i , and ˆ U r = ∪ i U i , and cho osing repres ent ative p oints ¯ x i ∈ X i , ¯ w i ∈ W i , and ¯ ν i ∈ U i , as abstract states and inputs. The finite abstraction of Σ is a gMDP b Σ = ( ˆ X , ˆ W , ˆ U , ˆ π , ˆ T , Y , ˆ h ), wher e ˆ X = { ¯ x i , i = 1 , . . . , n x } , ˆ U = { ¯ u i , i = 1 , . . . , n u } , ˆ W = { ¯ w i , i = 1 , . . . , n w } . T ransitio n proba bility matrix ˆ T is constructed accor ding to the dyna mics ˆ x ( k + 1) = ˆ f ( ˆ x ( k ) , ˆ w ( k ) , ˆ ν ( k ) , ς ( k )) with ˆ f ( ˆ x, ˆ ν , ˆ w , ς ) := Π x ( ˆ A r ˆ x + ˆ E r ϕ ( ˆ F r ˆ x ) + ˆ D r ˆ w + ˆ B r ˆ ν + ˆ R r ς ) , (5.3) where Π x : ˆ X r → ˆ X is the map tha t assigns to any ˆ x r ∈ ˆ X r , the repr esentativ e p oint ˆ x ∈ ˆ X of the corresp onding partition set containing ˆ x r . The output map ˆ h ( ˆ x ) = ˆ C ˆ x . The initial state of b Σ is also selected ac c ording to ˆ x 0 := Π x ( ˆ x r (0)) with ˆ x r (0) b eing the initial state of b Σ r . Remark 5.4. Abstr action map Π x satisfies the ine quality k Π x ( ˆ x r ) − ˆ x r k ≤ β for al l ˆ x r ∈ ˆ X r , wher e β is the state discr etization p ar ameter define d as β := sup {k ˆ x r − ˆ x ′ r k , ˆ x r , ˆ x ′ r ∈ X i , i = 1 , 2 , . . . , n x } . 10 ABOLF AZL LA V AEI 1 , S ADEGH S OUDJANI 2 , AND M AJID ZAMANI 3 , 4 5.2. Establishing Probabilistic Relatio ns. In this subs ection, we provide conditions under w hich b Σ is ( ǫ, δ )-sto chastically simulated by Σ, i.e. b Σ  δ ǫ Σ, with r elations R x and R w . Here we candida te rela tions R x = n ( x, ˆ x ) | ( x − P ˆ x ) T M ( x − P ˆ x ) ≤ ǫ 2 o , (5.4a) R w = n ( w, ˆ w ) | ( w − P w ˆ w ) T M w ( w − P w ˆ w ) ≤ ǫ 2 w o , (5.4b) where P ∈ R n × ˆ n and P w ∈ R m × ˆ m are matr ic e s of appropria te dimensio ns (p otentially w ith the lowest ˆ n and ˆ m ), and M , M w are p ositive-definite matrice s . Next theorem gives conditions for having b Σ  δ ǫ Σ with relations (5.4a) and (5.4b). Theorem 5.5 . L et Σ = ( A, B , C, D , E , F , R , ϕ ) and b Σ r = ( ˆ A r , ˆ B r , ˆ C r , ˆ D r , ˆ E r , ˆ F r , ˆ R r , ϕ ) b e t wo nonline ar systems with the same additive n oise. Su pp ose b Σ is a fi n ite gMDP c onst ructe d fr om b Σ r ac c or ding t o subse ct ion 5.1 . Then b Σ is ( ǫ, δ )- sto chastic al ly simu late d by Σ with r elations (5.4a) - (5.4 b) if ther e exist matric es K , Q , S , L 1 , L 2 and ˜ R such t hat M  C T C, (5.5a) ˆ C r = C P , (5.5b) ˆ F r = F P , (5.5c) E = P ˆ E r − B ( L 1 − L 2 ) , (5.5d) AP = P ˆ A r − B Q, (5.5e) D P w = P ˆ D r − B S, (5.5f ) P { ( H + P G ) T M ( H + P G ) ≤ ǫ 2 }  1 − δ, (5.5g) wher e H = (( A + B K ) + ¯ δ ( B L 1 + E ) F )( x − P ˆ x ) + D ( w − P w ˆ w ) + ( B ˜ R − P ˆ B r ) ˆ ν + ( R − P ˆ R r ) ς , G = ˆ A r ˆ x + ˆ E r ϕ ( ˆ F r ˆ x ) + ˆ D r ˆ w + ˆ B r ˆ ν + ˆ R r ς − Π x ( ˆ A r ˆ x + ˆ E r ϕ ( ˆ F r ˆ x ) + ˆ D r ˆ w + ˆ B r ˆ ν + ˆ R r ς ) . The pro of of Theorem 5.5 is provided in the App endix. Remark 5.6 . Note that c ondition (5.5 g) is a chanc e c onstr aint. We satisfy this c ondition by sele cting c onst ant c ς such that P { ς T ς ≤ c 2 ς } ≥ 1 − δ , and r e qu iring ( H + P G ) T M ( H + P G ) ≤ ǫ 2 for any ς with ς T ς ≤ c 2 ς . Sinc e ς ∼ (0 , I n ) , ς T ς has chi-squar e distribution with 2 de gr e es of fr e e dom. Thus, c ς = X − 1 2 (1 − δ ) with X − 1 2 b eing chi-squar e inverse cumulative distribution fun ction with 2 de gr e es of fr e e dom. 6. Case Study In this section, w e demonstra te the e ffectiv eness of the prop osed results on a netw ork of four sto chastic nonlinear systems (totally 12 dimensions), i.e. Σ = I (Σ 1 , Σ 2 , Σ 3 , Σ 4 ). W e wan t to construct finite gMDPs from their reduce d- order versions (together 4 dimensions). The interconnected gMDP Σ is illustrated in Figure 3 such that the output of Σ 1 (resp. Σ 2 ) is co nnected to the internal input of Σ 4 (resp. Σ 3 ), and the output of Σ 3 (resp. Σ 4 ) connects to the internal input of Σ 1 (resp. Σ 2 ). The matrices of the system are given by A i =   0 . 7882 0 . 3956 0 . 8333 0 . 7062 0 . 7454 0 . 9552 0 . 6220 0 . 3116 0 . 4409   , B i =   0 . 7555 0 . 1557 0 . 3487 0 . 1271 0 . 9836 0 . 2030 0 . 4735 0 . 4363 0 . 4493   , C i = 0 . 01 1 T 3 , E i =  0 . 6482 ; 0 . 60 08; 0 . 6209  , F i =  0 . 5146 ; 0 . 87 56; 0 . 2461  T , R i =  0 . 4974 ; 0 . 33 39; 0 . 4527  , (6.1) for i ∈ { 1 , 2 , 3 , 4 } . The internal input a nd o utput matrices are also given by C 14 = C 23 = C 31 = C 42 = 0 . 01 1 T 3 , D 13 = D 24 = D 32 = D 41 =  0 . 074; 0 . 010 ; 0 . 08 6  . COMPOSITIONAL ABSTRACT ION-BASED SYNTHE S IS OF MDPS VIA APPRO XIM A TE P ROB ABILISTIC RELA TIONS 11 Σ 3 Σ 4 Σ 1 Σ 2 y 33 y 44 ν 1 ν 2 ν 3 ν 4 y 31 y 42 y 14 y 23 Figure 3. The interconnected gMDP Σ = I (Σ 1 , Σ 2 , Σ 3 , Σ 4 ). W e consider ϕ i ( x ) = si n ( x ), ∀ i ∈ { 1 , . . . , 4 } . Then functions ϕ i satisfy condition (5.2) with b = 1. In the following, we first construct the reduced-or der version of the g iven dyna mic by sa tisfying conditions (5.5a)- (5.5f). W e then establish rela tions be tw een s ubs ystems by fulfilling c o ndition (5.5g). Afterwards, we sa tisfy the comp ositionality condition (4.6) to g et a relation on the comp ose d s ystem, and finally , we utilize Theore m 3.5 to provide the pr obabilistic closenes s guar antee betw een the in terconnected mo del and its constructed finite MDP . Conditions (5.5a)-(5.5f) are satisfied with, ∀ i ∈ { 1 , 2 , 3 , 4 } , Q i =  − 1 . 656 8; − 1 . 2280 ; 1 . 9276  , S i =  0 . 0775 ; 0 . 07 26; − 0 . 175 9  , P i =  0 . 5931 ; 0 . 39 81; 0 . 5398  , L 1 i =  − 0 . 654 6; − 0 . 4795 ; − 0 . 226 4  , L 2 i =  − 0 . 171 3; − 0 . 0777 ; − 0 . 104 4  , P wi = 1 , M i = I 3 . Accordingly , matrices of r educed-order systems can be o btained as , ∀ i ∈ { 1 , 2 , 3 , 4 } , ˆ A r i = 0 . 512 7 , ˆ E r i = 0 . 3 , ˆ F r i = 0 . 786 6 , ˆ C r i = 0 . 0371 , ˆ D r i = 0 . 140 3 , ˆ R r i = 0 . 838 6 . Moreov er, w e compute ˜ R i = ( B T i M i B i ) − 1 B T i M i P i ˆ B r i , i ∈ { 1 , 2 , 3 , 4 } , to make c hance constraint (5.5g) less conserv ative. By tak ing ˆ B r i = 2, we hav e ˜ R i = [1 . 141 8 ; 0 . 5182; 0 . 6 9 65]. The interface functions for i ∈ { 1 , 2 , 3 , 4 } ar e acquired by (9.3) as ν i =   − 0 . 666 5 − 0 . 3652 − 0 . 96 80 − 0 . 437 2 − 0 . 5536 − 0 . 57 81 − 0 . 401 2 − 0 . 1004 − 0 . 26 12   ( x i − P i ˆ x i ) + Q i ˆ x i + ˜ R i ˆ ν i + S i ˆ w i + L 1 i ϕ i ( F i x i ) − L 2 i ϕ i ( F i P i ˆ x i ) . W e pro ceed with showing that condition (5.5g) holds as well, using Remark 5 .6. This condition can b e satisfied via the S-pro cedur e [BV0 4], which ena ble s us to reformulate (5.5 g) as existence of λ ≥ 0 such that matrix inequality λ i  ˜ F 1 i ˜ g 1 i ˜ g T 1 i ˜ h 1 i  −  ˜ F 2 i ˜ g 2 i ˜ g T 2 i ˜ h 2 i   0 , (6.2) holds. Here, ˜ F 1 i and ˜ F 2 i are sy mmetric matrices , ˜ g 1 i and ˜ g 2 i are vectors, ˜ h 1 i and ˜ h 2 i are real num b er s. W e first bo und the external input of abstract s y stems a s ˆ ν 2 i ≤ c ˆ ν i and sele c t c ς i = X − 1 2 (1 − δ i ), fo r all i ∈ { 1 , 2 , 3 , 4 } . Then matr ices, v ectors and real num ber s of ineq uality (6.2), ∀ i ∈ { 1 , 2 , 3 , 4 } , can be constructed as in (9.1) and (9) provided in the Appendix . By taking ǫ i = 1 . 2 5, ǫ w i = 0 . 05, c ˆ ν i = 0 . 25, δ i = 0 . 0 01, β i = 0 . 1 , λ i = 0 . 34 7, for all i ∈ { 1 , 2 , 3 , 4 } , one can readily verify that the matrix inequality (6 .2 ) holds. Then b Σ i is ( ǫ i , δ i )-sto chastically simulated b y Σ i with relations R xi = n ( x i , ˆ x i ) | ( x i − P i ˆ x i ) T M i ( x i − P i ˆ x i ) ≤ ǫ 2 i o , R wi = n ( w i , ˆ w i ) | ( w i − ˆ w i ) 2 ≤ ǫ 2 wi o , for i ∈ { 1 , 2 , 3 , 4 } . W e pro ceed with showing that the comp ositionality c o ndition in (4.6) holds, a s well. T o do so, by employing S-pro c edure, one should satisfy the matrix inequality in (6.2) with the following matr ices: 12 ABOLF AZL LA V AEI 1 , S ADEGH S OUDJANI 2 , AND M AJID ZAMANI 3 , 4 ˜ F 1 i =  M i − M i P i ∗ P T i M i P i  , ˜ F 2 i =  C T r i M wi C r i − C T r i M wi P wi ˆ C r i ∗ ˆ C T r i P T wi M wi P wi ˆ C r i  , ˜ g 1 i = ˜ g 2 i = 0 4 , ˜ h 1 i = − ǫ 2 i , ˜ h 2 i = − ǫ 2 wi , for i ∈ { 1 , 2 , 3 , 4 } . This condition is sa tisfiable with λ i = 0 . 0 01 ∀ i ∈ { 1 , 2 , 3 , 4 } , thus b Σ is ( ǫ, δ )-sto chastically simulated by Σ with ǫ = 6, a nd δ = 0 . 0 03. According to (3.1), we guarantee that the distance b etw een outputs of Σ a nd of b Σ will no t exceed ǫ = 6 during the time horizo n T k = 10 with pr obability at least 96% ( γ = 0 . 0 4). 6.1. Comparison. T o demonstrate the effectiveness of the prop osed approa ch, let us now compa re the gua r- antees provided by our approa ch and by [LSZ19a, LSZ1 8 b]. Note that our result is based on the δ -lifted relation while [LSZ1 9a, LSZ18 b] employ dissipativity-t yp e r easoning to provide a co mpo sitional methodo lo gy for constr uc ting bo th infinite abstractio ns (reduced-order mo dels) and finite MDPs in tw o consecutive steps . Since we ar e not able to satisfy the prop os ed matrix inequalities in [LSZ18 b, Ineqality (2 2 )], a nd [LSZ19a, Inequality (5.5 )] for the given system in (6.1), we change the system dynamics to hav e a fair co mparison. In other words, in order to show the conser v atism nature of the exis ting techniques in [L SZ 18b, LSZ1 9a], we provide another example and compa re our techniques with the existing ones in great detail. The matrices of the new system are given by A i = I 5 , B i = I 5 , C i = 0 . 05 1 T 5 , R i = 1 5 , for i ∈ { 1 , 2 , 3 , 4 } , where matrices E i , F i are ide ntically zero. The internal input and output matrices are also given by: C 14 = C 23 = C 31 = C 42 = 0 . 05 1 T 5 , D 13 = D 24 = D 32 = D 41 = 0 . 1 1 5 . Conditions (5.5a),(5.5b),(5.5e),(5.5 f) a re sa tisfied b y: M i = I 5 , P xi = 1 5 , P wi = 1 , Q i = 1 5 , S i = 0 . 1 1 5 , for i ∈ { 1 , 2 , 3 , 4 } . Accor ding ly , the ma trices of reduced-o rder systems ar e given as: ˆ A r i = 2 , ˆ C r i = 0 . 25 , ˆ D r i = 0 . 2 , ˆ R r i = 0 . 97 , ∀ i ∈ { 1 , 2 , 3 , 4 } . Moreov er, by taking ˆ B r i = 1, we compute ˜ R i , i ∈ { 1 , 2 , 3 , 4 } , as ˜ R i = 1 5 . The interface function for i ∈ { 1 , 2 , 3 , 4 } is c o mputed as: ν i = − 0 . 95 I 5 ( x i − 1 5 ˆ x i ) + 1 5 ˆ x i + 1 5 ˆ ν i + 0 . 1 1 5 ˆ ω i . W e pro ceed with showing that conditio n (5.5g) holds, as well. By ta k ing ǫ i = 5 , ǫ w i = 0 . 75 , c ˆ ν i = 0 . 25 , δ i = 0 . 001 , β i = 0 . 1 , λ i = 0 . 825 , ∀ i ∈ { 1 , 2 , 3 , 4 } , and by employing S-pro cedure, one can rea dily verify that condition (5.5g) holds. Then b Σ i is ( ǫ i , δ i )- sto chastically simulated by Σ i , for i ∈ { 1 , 2 , 3 , 4 } . Additiona lly , by a pply ing S- pr o cedure, o ne ca n readily verify that b Σ is ( ǫ , δ )-sto chastically simulated by Σ with ǫ = 20 , and δ = 0 . 005 . Acco rding to (3 .1), we guarantee that the distance betw een outputs of Σ and of b Σ will not e x ceed ǫ = 2 0 during the time horizon T k = 5 with probability at least 9 7% ( γ = 0 . 03). Now we apply the propo sed results in [LSZ18 b, LSZ1 9a] for the same matrices of the new system and also employing the same ǫ and discretiza tio n parameter β . Since the prop os e d approaches in [LSZ1 8b, LSZ19a] ar e presented in tw o cons ecutive steps, we employ the next prop osition which provides the ov erall er ror b ound in t wo-step a bstraction scheme. COMPOSITIONAL ABSTRACT ION-BASED SYNTHE S IS OF MDPS VIA APPRO XIM A TE P ROB ABILISTIC RELA TIONS 13 Prop ositi o n 6. 1. Supp ose Σ 1 , Σ 2 , and Σ 3 ar e thr e e st o chastic systems without internal signals. F or any external input t ra je ctories ν 1 , ν 2 , and ν 3 and for any a 1 , a 2 , and a 3 as t he initial states of the thr e e systems, if P  sup 0 ≤ k ≤ T k k y 1 a 1 ν 1 ( k ) − y 2 a 2 ν 2 ( k ) k ≥ ǫ 1 | [ a 1 ; a 2 ]  ≤ γ 1 , P  sup 0 ≤ k ≤ T k k y 2 a 2 ν 2 ( k ) − y 3 a 3 ν 3 ( k ) k ≥ ǫ 2 | [ a 2 ; a 3 ]  ≤ γ 2 , for some ǫ 1 , ǫ 2 > 0 and γ 1 , γ 2 ∈ ]0 1[ , then the pr ob abilistic mismatch b et we en output tr aje ctories of Σ 1 and Σ 3 is qu antifie d as P  sup 0 ≤ k ≤ T k k y 1 a 1 ν 1 ( k ) − y 3 a 3 ν 3 ( k ) k ≥ ǫ 1 + ǫ 2 | [ a 1 ; a 2 ; a 3 ]  ≤ γ 1 + γ 2 . The pro of is provided in the App endix. By applying the prop o s ed results in [LSZ19a] to construct the infinite abstractio n b Σ r , one can gua rantee that the distance betw een outputs of Σ and of b Σ r will exc e ed ǫ 1 = 15 during the time horizon T k = 5 with probability at most 87 . 94 %, i.e., P ( k y aν ( k ) − ˆ y r ˆ a r ˆ ν r ( k ) k ≥ 15 , ∀ k ∈ [0 , 5 ]) ≤ 87 . 94 . After a pplying the prop osed results in [LSZ18 b] to construct the finite abstraction b Σ from b Σ r , one can gua rantee that the distance b etw een outputs of b Σ r and of b Σ will exceed ǫ 2 = 5 dur ing the time horizo n T k = 5 with probability at most 0 . 011 7%, i.e., P ( k ˆ y r ˆ a r ˆ ν r ( k ) − ˆ y ˆ a ˆ ν ( k ) k ≥ 5 , ∀ k ∈ [0 , 5]) ≤ 0 . 011 7 . By employing Prop o sition 6.1, one can guar antee that the distance b etw een outputs of Σ and o f b Σ w ill ex ceed ǫ = 20 during the time horizon T k = 5 with probability at most 0 . 8911%, i.e. P ( k y aν ( k ) − ˆ y ˆ a ˆ ν ( k ) k ≥ 20 , ∀ k ∈ [0 , 5]) ≤ 0 . 891 1 . This means that the distance betw een outputs of Σ and of b Σ will not exce e d ǫ = 20 during the time hor izon T k = 5 with probability a t least 0 . 10 89%. As seen, our provided results dramatica lly o utper form the ones prop osed in [LSZ1 8 b, LSZ19a]. More prec isely , since our pr op osed a pproach here is pr esented in a unified framework than t wo-step abstraction sc heme whic h is the case in [LSZ18b, LSZ19a], we only need to c heck our prop osed conditions one time, and consequently , our prop osed approach here is muc h le s s conserv a tive. 7. Discussio n In this pa p e r, we provided a unified c o mpo sitional s cheme for c onstructing bo th finite and infinite abstrac - tions of g MDPs with in ternal inputs. W e defined ( ǫ, δ )-approximate probabilistic relatio ns that ar e suitable for constructing comp ositiona l abstra ctions o f gMDPs. W e focuse d on a sp ecific class of no nlinear dynamical systems, a nd co nstructed b oth infinite (reduced-or der mo dels) and finite abstra c tions in a unified framework, using quadr atic relations on the space and linear in terface functions. W e then provided conditions for com- po sing such r elations. Finally , we demonstrated the effectiveness of the prop o sed r esults by co nsidering a net work of four nonlinear systems (totally 12 dimensions) and co nstructing finite gMDPs from their reduced- order versions (together 4 dimensions) with g uaranteed b ounds on their probabilistic output tr a jectories . W e benchmarked our results against the co mpo sitional abstraction techniques of [LSZ18b, LSZ19a], and show e d that our prop osed approa ch is muc h le ss conserv a tive than the ones prop o sed in [LSZ18b, LSZ19a]. 8. Acknowledgment This work was supp or ted in par t by the H202 0 ERC Sta r ting Gra nt AutoCPS (grant ag reement No. 8046 39). 14 ABOLF AZL LA V AEI 1 , S ADEGH S OUDJANI 2 , AND M AJID ZAMANI 3 , 4 References [Aba13] A. Abate. Approximation metrics based on probabilistic bisi mulations f or general state-space m ar k ov pro cesses: a survey . Ele c tr onic Notes in The or etica l Computer Science , 297:3–25, 2013. [AK01] M. Arcak and P . K ok oto vic. Observer-based con trol of systems with slop e-restricted nonlinearities. IEEE T r ansac- tions on Automatic Contr ol , 46(7):1146 –1150, 2001. [AKNP14] Al. Abate, M. Kwiatko wsk a, G. Norman, and D. Park er. Probabilistic mo del c heck ing of lab elled marko v processes via finite approximate bisimulations. In Horizons of the Mind. A T ribute to Pr akash Panangad en , pages 40–58. Springer, 2014. [APLS08] A. Abate, M . Prandini, J. Lygeros, and S. Sastry . Probabilistic reachabilit y and safet y f or controlled discrete-time stochastic hyb rid systems. Automatic a , 44(11):272 4–2734, 2008. [BKL08] C.l Baier, J.- P . Kato en, and K. G. Larsen. Principles of mo del che cking . M IT press, 2008. [BV04] S. Boy d and L. V anden b erghe. Convex optimization . Cambridge universit y press, 2004. [DAK12] A. D’Inno cenzo, A. Abate, and J.P . Katoen. Robust PCTL mo del chec king. In Pr o c e edings of the 15th ACM international c onfer enc e on H ybrid Syste ms: Computation and Contr ol , pages 275–286, 2012. [DGJP04] J. Desharnais, V. Gupta, R. Jagadeesan, and P . Pana ngaden. Metrics for lab elled marko v pro cesses. The or et i c al c omputer scienc e , 318(3):323–35 4, 2004. [DL T08] J. Desharnais, F. Laviolette, and M. T racol. Approximate analysis of pr obabilistic pro cesses: Logic, si mulation and games. In Pr o c e e dings of the 5th i nternational c onfer e nc e on quantitative ev aluation of syst em , pages 264–273, 2008. [HHHK13] E. M. Hahn, A. Hartmanns, H . Hermanns, and J.- P . Katoen. A comp ositional mo delling and analysis framework for sto c hastic h ybrid systems. F ormal Metho ds in Syste m D e sign , 43(2):191–23 2, 2013. [HS18] Sofie Haesaert and Sadegh Soudjani. Robust dynamic programmi ng f or temporal logic con trol of sto chast ic systems. CoRR , abs/1811.11445, 2018. [HSA17] S. H aesaert, S. Soudjani, and A. Abate. V erification of general Mar k ov decision pro cesses by approx imate similarity relations and p olicy r efinemen t. SIAM Journal on Contr ol and Optimization , 55(4):2333–2367 , 2017. [HSA18] Sofie Haesaert, Sadegh Soudjani, and Al essandro Abate. T emp oral logic control of general marko v decision processes b y approx imate p olicy r efinemen t. IF AC-Pap ersOnL ine , 51(16):73 – 78, 2018. 6th IF A C Conference on Analysis and Design of Hybrid Systems ADHS 2018. [JP09] A. A. Juli us and G. J. Pa ppas. Approximations of stochastic h ybrid systems. IEEE T r ansactions on Automatic Contr ol , 54(6):1193–12 03, 2009. [KSL13] M. Kamgarpour, S. Summers, and J. Lygeros. Control design for sp ecifications on stochastic h ybrid systems. In Pr o c e e dings of the 16th ACM International Confer enc e on Hybrid Syste ms: Computation and Contro l , pages 303– 312, 2013. [LS91] K. G. Larsen and A. Skou. Bisimulation through probabilis tic testing. Information and c omputation , 94(1):1–28, 1991. [LSMZ17] A. Lav aei, S. Soudjani, R. Ma jumdar, and M. Zamani. Comp ositional abstractions of interco nnected di s crete-time stochastic cont rol systems. In Pr o ce e dings of the 56th IEEE Confer enc e on Decision and Contr ol , pages 3551–3556, 2017. [LSZ18a] A. La v aei, S. Soudjani, and M. Zamani. Compositional synthesis of finite abstractions for con tinuous-space s tochastic con trol systems: A small- gain approach. In Pr o c e ed ings of the 6th IF AC Confer enc e on A nalysis and Design of Hybrid Systems , volume 51, pages 265–270, 2018. [LSZ18b] A. La v aei, S. Soudjani, and M . Zamani. F r om di ssipativity theory to comp osi tional construction of finite Mar ko v decision pro cesses. In Pr o ce ed ings of the 21st ACM Inte rnational Confer enc e on Hybrid Sy st ems: Computation and Contr ol , pages 21–30, 2018. [LSZ19a] A. La v aei, S. Soudjani, and M. Zamani. Composi tional construction of infinite abstractions for netw orks of stochastic con trol systems. Automa tic a , 107:125–137, 2019. [LSZ19b] A. Lav aei, S. Soudjani, and M . Zamani. Comp ositional synthesis of large-scale stochastic systems: A relaxed dissi- pativit y approac h. , F ebruar y 2019. [MSSM16] K . Malli k, A.-K. Schm uc k, S. Soudjani, and R. M a jumdar. Comp ositional abstraction-based con troller synt hesis for con tinuo us-time systems. arXiv:1612.085 15 , December 2016. [SA13] S. Soudjani and A. Abate. Adaptiv e and sequen tial gridding pro cedures for the abstraction and v erification of stochastic pro cesses. SIAM Journal on Applie d Dynamic al Sy stems , 12(2):921–95 6, 2013. [SAM15] S. Soudjani, A. Abate, and R. Ma jumdar. Dynamic Bay esian net works as formal abstractions of structured stochast ic processes. In Pr o c e e dings of the 26th International Confer e nc e on Concurr ency The ory , pages 1–14, 2015. [SL95] R. Segala and N. Lynch. Probabilistic si mulat ions for probabilistic pr o cesses. Nor dic Journal of Computing , 2(2):250– 273, 1995. [Sou14] Sadegh Soudjani. F ormal Abstr actions for Autom ate d V erifica tion and Synt hesis of Sto chastic Systems . PhD thesis, T echnisc he Universiteit Delft, The Netherlands, 2014. [Sv06] S.N. Strubb e and A.J v an der Schaft. Comp ositional mo del ling of sto chastic hybrid systems , pages 47–77. Number 500-266 in Control Engineering. CRC Pr ess, 2006. COMPOSITIONAL ABSTRACT ION-BASED SYNTHE S IS OF MDPS VIA APPRO XIM A TE P ROB ABILISTIC RELA TIONS 15 [T A11] I. Tk achev and A. A bate. On infinite-horizon probabilistic prop erties and s tochastic bisimulation f unctions. In Pr o c e e dings of the 50th IEEE Confer enc e on De cision and Contr ol and Eur op ea n Contr ol Confer enc e (CDC-ECC) , pages 526–531, 2011. [ZMEM + 14] M. Zamani, P . M oha jerin Esfahani, R. Ma j umdar, A. A bate, and J. Lygeros. Symbolic control of stochastic systems via appro ximately bisim ilar finite abstractions. IEEE T r ansactions on Auto matic Contro l , 59(12):3135–3150, 2014. 9. A ppendix Definition 9.1. ( [HSA17] ) Consider t wo gMDPs without internal inputs Σ = ( X , U, π , T , Y , h ) and b Σ = ( ˆ X , ˆ U , ˆ π , ˆ T , Y , ˆ h ) , that have the same output sp ac es. b Σ is ( ǫ , δ )-sto chastic al ly simulate d by Σ , i.e. b Σ  δ ǫ Σ , if ther e exists a r elation R x ⊆ X × ˆ X for which ther e exists a Bor el me asu r able sto chastic kernel L T ( · | x, ˆ x, ˆ ν ) on X × ˆ X such t hat • ∀ ( x, ˆ x ) ∈ R x , k h ( x ) − ˆ h ( ˆ x ) k ≤ ǫ , • ∀ ( x, ˆ x ) ∈ R x , ∀ ˆ ν ∈ ˆ U , ∃ ν ∈ U such t hat T ( · | x, ν ( x, ˆ x , ˆ ν )) ¯ R δ ˆ T ( · | ˆ x, ˆ ν ) with L T ( · | x, ˆ x , ˆ ν ) , • π ¯ R δ ˆ π . Matrices app eared in (6.2) : ˜ F 1 i =         M i 0 3 × 3 0 3 0 3 0 3 0 3 0 3 × 3 0 3 × 3 0 3 0 3 0 3 0 3 ∗ ∗ M wi 0 0 0 ∗ ∗ ∗ 1 0 0 ∗ ∗ ∗ ∗ 1 0 ∗ ∗ ∗ ∗ ∗ 1         , ˜ F 2 i =         ˜ F 11 i ˜ F 12 i ˜ F 13 i ˜ F 14 i ˜ F 15 i ˜ F 16 i ∗ ˜ F 22 i ˜ F 23 i ˜ F 24 i ˜ F 25 i ˜ F 26 i ∗ ∗ ˜ F 33 i ˜ F 34 i ˜ F 35 i ˜ F 36 i ∗ ∗ ∗ ˜ F 44 i ˜ F 45 i ˜ F 46 i ∗ ∗ ∗ ∗ ˜ F 55 i ˜ F 56 i ∗ ∗ ∗ ∗ ∗ ˜ F 66 i         , (9.1) where ˜ F 11 i = ( A i + B i K i ) T M i ( A i + B i K i ) , ˜ F 12 i = ( A i + B i K i ) T M i ( B i L 1 i + E i ) F i , ˜ F 13 i = ( A i + B i K i ) T M i D i , ˜ F 14 i = ( A i + B i K i ) T M i ( B i ˜ R i − P i ˆ B r i ) , ˜ F 15 i = ( A i + B i K i ) T M i P i , ˜ F 16 i = ( A i + B i K i ) T M i ( R i − P i ˆ R r i ) , ˜ F 22 i = F T i ( B i L 1 i + E i ) T M ( B i L 1 i + E i ) F i , ˜ F 23 i = F T i ( B i L 1 i + E i ) T M i D i , ˜ F 24 i = F T i ( B i L 1 i + E i ) T M i ( B i ˜ R i − P i ˆ B r i ) , ˜ F 25 i = F T i ( B i L 1 i + E i ) T M i P i , ˜ F 26 i = F T i ( B i L 1 i + E i ) T M i ( R i − P i ˆ R r i ) , ˜ F 33 i = D T i M i D i , ˜ F 34 i = D T i M i ( B i ˜ R i − P i ˆ B r i ) , ˜ F 35 i = D T i M i P i , ˜ F 36 i = D T i M i ( R i − P i ˆ R r i ) , ˜ F 44 i = ( B i ˜ R i − P i ˆ B r i ) T M i ( B i ˜ R i − P i ˆ B r i ) , ˜ F 45 i = ( B i ˜ R i − P i ˆ B r i ) T M i P i , ˜ F 46 i = ( B i ˜ R i − P i ˆ B r i ) T M i ( R i − P i ˆ R r i ) , ˜ F 55 i = P T i M i P i , ˜ F 56 i = P T i M i ( R i − P i ˆ R r i ) , ˜ F 66 i = ( R i − P i ˆ R r i ) T M i ( R i − P i ˆ R r i ) . V ectors and real num b ers app eared in (6.2) : ˜ g 1 i = ˜ g 2 i = 0 10 , ˜ h 1 i = − ( ǫ 2 i + ǫ 2 wi + c ˆ ν i + c ς i + β i ) , ˜ h 2 i = − ǫ 2 i . (9.2) Pr o of. (Theorem 3.5) The definition of lifting implies that the initial states of the t wo systems are in the relation with probability at least 1 − δ . Mor eov er, if the tw o states are in the relation at time k , they remain in the relation at time k + 1 with probability a t least 1 − δ . Then, we ca n wr ite P { ( x ( k ) , ˆ x ( k )) ∈ R x for all k ∈ [0 , T k ] } ≥ (1 − δ ) T k +1 . This can b e prov ed by induction and conditioning the probability on the intermediate states. 16 ABOLF AZL LA V AEI 1 , S ADEGH S OUDJANI 2 , AND M AJID ZAMANI 3 , 4 Note that if { ˆ h ( ˆ x ( k )) } 0: T k ∈ A − ǫ and ( x ( k ) , ˆ x ( k )) ∈ R x for a ll k ∈ [0 , T k ], then { y ( k ) } 0: T k ∈ A . As a consequence P {{ ˆ h ( ˆ x ) } 0: T k ∈ A − ǫ } ∧ ( x ( k ) , ˆ x ( k )) ∈ R x for all k ∈ [0 , T k ] } ≤ P {{ h ( x ) } 0: T k ∈ A } . Now by employing the union b ounding ar gument, we hav e P {{ ˆ h ( ˆ x ) } 0: T k ∈ A − ǫ } − (1 − δ ) T k +1 ≤ P {{ ˆ h ( ˆ x ) } 0: T k ∈ A − ǫ ∧ ( x ( k ) , ˆ x ( k )) ∈ R x , for all k ∈ [0 , T k ] } . Then 1 − P {{ ˆ h ( ˆ x ) } 0: T k ∈ A − ǫ ∧ ( x ( k ) , ˆ x ( k )) ∈ R x for all k ∈ [0 , T k ] } ≤ (1 − P { { ˆ h ( ˆ x ) } 0: T k ∈ A − ǫ } ) + (1 − P { ( x ( k ) , ˆ x ( k )) ∈ R x for all k ∈ [0 , T k ] } ) ≤ (1 − P { { ˆ h ( ˆ x ) } 0: T k ∈ A − ǫ } ) + (1 − (1 − δ ) T k +1 ) . One can deduce that P {{ ˆ h ( ˆ x ) } 0: T k ∈ A − ǫ } − (1 − (1 − δ ) T k +1 ) ≤ P {{ h ( x ) } 0: T k ∈ A } . Similarly , if { h ( x ( k )) } 0: T k ∈ A and ( x ( k ) , ˆ x ( k )) ∈ R x , then { ˆ h ( ˆ x ( k )) } 0: T k ∈ A ǫ . Thus via simila r a rguments it holds that P {{ h ( x ) } 0: T k ∈ A } ≤ P {{ ˆ h ( ˆ x ) } 0: T k ∈ A ǫ } + (1 − (1 − δ ) T k +1 ) .  Pr o of. (Theorem 4.3) W e fir st show that the first co ndition in Definition 9.1 holds. F or a ny x = [ x 1 ; . . . ; x N ] ∈ X and ˆ x = [ ˆ x 1 ; . . . ; ˆ x N ] ∈ ˆ X with x R x ˆ x , one gets: k h ( x ) − ˆ h ( ˆ x ) k = k [ h 11 ( x 1 ); . . . ; h N N ( x N )] − [ ˆ h 11 ( ˆ x 1 ); . . . ; ˆ h N N ( ˆ x N )] k ≤ N X i =1 k h ii ( x i ) − ˆ h ii ( ˆ x i ) k ≤ N X i =1 k h i ( x i ) − ˆ h i ( ˆ x i ) k ≤ N X i =1 ǫ i . As seen, the first c o ndition in Definition 9.1 holds w ith ǫ = P N i =1 ǫ i . The second condition is also s a tisfied as follows. F o r any ( x, ˆ x ) ∈ R x , and ˆ ν ∈ ˆ U , we have: L n x ′ R x ˆ x ′ | x, ˆ x , ˆ ν o = L n x ′ i R x i ˆ x ′ i , i ∈ { 1 , 2 , . . . , N } | x, ˆ x, ˆ ν o = N Y i =1 L i n x ′ i R x i ˆ x ′ i , | g i ( x ) , ˆ g i ( x ) , ˆ ν i o ≥ N Y i =1 (1 − δ i ) . The second condition in Definition 9.1 also holds with δ = 1 − Q N i =1 (1 − δ i ) which completes the pro o f.  Pr o of. (Theorem 5.5) First, we show that the firs t conditio n in Definition 3 .2 holds for all ( x, ˆ x ) ∈ R x . According to (5.5a) and (5.5b), we hav e k C x − ˆ C r ˆ x k 2 = ( x − P ˆ x ) T C T C ( x − P ˆ x ) ≤ ( x − P ˆ x ) T M ( x − P ˆ x ) ≤ ǫ 2 , for a ny ( x, ˆ x ) ∈ R x . N ow we pro ceed with showing the second condition. This condition re quires that ∀ ( x, ˆ x ) ∈ R x , ∀ ( w , ˆ w ) ∈ R w , ∀ ˆ ν ∈ ˆ U , the next states ( x ′ , ˆ x ′ ) should also b e in relation R x with proba bilit y at least 1 − δ : P { ( x ′ − P ˆ x ′ ) T M ( x ′ − P ˆ x ′ ) ≤ ǫ 2 } ≥ 1 − δ. Given any x , ˆ x , and ˆ ν , w e choose ν via the following interfac e function: ν = ν ˆ ν ( x, ˆ x, ˆ w , ˆ ν ) := K ( x − P ˆ x ) + Q ˆ x + ˜ R ˆ ν + S ˆ w + L 1 ϕ ( F x ) − L 2 ϕ ( F P ˆ x ) . (9.3) COMPOSITIONAL ABSTRACT ION-BASED SYNTHE S IS OF MDPS VIA APPRO XIM A TE P ROB ABILISTIC RELA TIONS 17 By s ubstituting dynamics of Σ and b Σ, employing (5.5c)-(5 .5f), and the definition o f the interface function (9.3), w e simplify x ′ − P ˆ x ′ = Ax + E ϕ ( F x ) + Dw + B ν ˆ ν ( x, ˆ x, ˆ w , ˆ ν ) + Rς − P ( ˆ A r ˆ x + ˆ E r ϕ ( ˆ F r x ) + ˆ D r ˆ w + ˆ B r ˆ ν + ˆ R r ς ) + P G, to ( A + B K )( x − P ˆ x ) + D ( w − P w ˆ w ) + ( B ˜ R − P ˆ B r ) ˆ ν + ( B L 1 + E )( ϕ ( F x ) − ϕ ( F P ˆ x r )) + ( R − P ˆ R r ) ς + P G, (9.4) with G = ˆ A r ˆ x + ˆ E r ϕ ( ˆ F r ˆ x ) + ˆ D r ˆ w + ˆ B r ˆ ν + ˆ R r ς − Π x ( ˆ A r ˆ x + ˆ E r ϕ ( ˆ F r ˆ x ) + ˆ D r ˆ w + ˆ B r ˆ ν + ˆ R r ς ). F ro m the slop e restriction (5.2), one obtains ϕ ( F x ) − ϕ ( F P ˆ x ) = ¯ δ ( F x − F P ˆ x ) = ¯ δ F ( x − P ˆ x ) , (9.5) where ¯ δ is a function of x and ˆ x , and tak es v alues in the interv al [0 , b ]. Using (9.5), the expre s sion in (9 .4 ) reduces to (( A + B K ) + ¯ δ ( B L 1 + E ) F )( x − P ˆ x ) + D ( w − P w ˆ w ) + ( B ˜ R − P ˆ B r ) ˆ ν + ( R − P ˆ R r ) ς + P G. This gives condition (5.5g) for having the pr obabilistic relation.  Pr o of. (Prop osition 6. 1) By defining A = {k y 1 a 1 ν 1 ( k ) − y 2 a 2 ν 2 ( k ) k < ǫ 1 | [ a 1 ; a 2 ; a 3 ] } , B = {k y 2 a 2 ν 2 ( k ) − y 3 a 3 ν 3 ( k ) k < ǫ 2 | [ a 1 ; a 2 ; a 3 ] } , C = {k y 1 a 1 ν 1 ( k ) − y 3 a 3 ν 3 ( k ) k < ǫ 1 + ǫ 1 | [ a 1 ; a 2 ; a 3 ] } , we have P { ¯ A} ≤ γ 1 and P { ¯ B} ≤ γ 2 , wher e ¯ A and ¯ B are the co mplement of A and B , resp ectively . Since P {A ∩ B } ≤ P {C } , we hav e P { ¯ C } ≤ P { ¯ A ∪ ¯ B } ≤ P { ¯ A} + P { ¯ B} ≤ γ 1 + γ 2 . Then P  sup 0 ≤ k ≤ T k k y 1 a 1 ν 1 ( k ) − y 3 a 3 ν 3 ( k ) k ≥ ǫ 1 + ǫ 2 | [ a 1 ; a 2 ; a 3 ]  ≤ γ 1 + γ 2 .  1 Dep ar tment of Electrical and Compu ter Eng ineering, Technical Univ ersity of M unich, Germany. E-mail addr ess : lavaei@tum.d e 2 School of Computing, Newcastle Un iversity, UK. E-mail addr ess : sadegh.soudj ani@ncl.ac.uk 3 Dep ar tment of Computer Science, Univ ersity of Colorado Boulder, US A. 4 Dep ar tment of Computer Science, Ludwig M aximilian University of Munich, Germ any. E-mail addr ess : majid.zamani @colorado.edu