Parameter Synthesis for Markov Models: Covering the Parameter Space

F ormal Metho ds in System Design man uscript No. (will b e inserted b y the editor) P arameter Syn thesis for Mark o v Mo dels Co v ering the P arameter Space Sebastian Junges · Erik a ´ Abrah´ am · Christian Hensel · Nils Jansen · Jo ost-Pieter Kato en · Tim Quatmann · Matthias V olk the date of receipt and acceptance should be inserted later Abstract Mark ov chain analysis is a key technique in formal veriﬁcation. A practi- cal obstacle is that all probabilities in Mark ov mo dels need to b e known. Ho wev er, system quan tities such as failure rates or pac ket loss ratios, etc. are often not—or only partially—known. This motiv ates considering parametric mo dels with transi- tions labeled with functions o ver parameters. Whereas traditional Mark ov c hain analysis relies on a single, ﬁxed set of probabilities, analysing parametric Mark ov mo dels focuses on synthesising parameter v alues that establish a giv en safet y or p erformance speciﬁcation φ . Examples are: what component failure rates ensure the probabilit y of a system breakdo wn to b e b elo w 0.00000001?, or whic h failure rates maximise the p erformance, for instance the throughput, of the system? This pap er presents v arious analysis algorithms for parametric discrete-time Marko v c hains and Marko v decision pro cesses. W e fo cus on three problems: (a) do all parameter v alues within a given region satisfy φ ?, (b) which regions satisfy φ and whic h ones do not?, and (c) an appro ximate version of (b) fo cusing on cov ering a large fraction of all p ossible parameter v alues. W e give a detailed account of the v arious algorithms, presen t a softw are to ol realising these tec hniques, and rep ort on an extensive exp erimental ev aluation on b enchmarks that span a wide range of applications. Keyw ords F ormal Methods · V eriﬁcation · Mo del Checking · Probabilistic Systems · P arameter Synthesis · Marko v Chains The work has b een partially supp orted by the DFG R TG 2236 UnRA V eL, the European Union’s Horizon 2020 research and inno v ation programme under the Marie Sk lodowsk a-Curie grant agreemen t No. 101008233 (Mission), and the ERC Starting Grant 101077178 (DEUCE). S. Junges Radboud University , Nijmegen, The Netherlands N. Jansen Ruhr-Universit y , Boch um, Germany and Radboud Universit y , Nijmegen, The Netherlands E. ´ Abrah´ am, C. Hensel, J.-P . Kato en, T. Quatmann Department of Computer Science, R WTH Aachen Univ ersity , Aachen, Germany M. V olk Eindhov en Universit y of T echnology , Eindhov en, The Netherlands 2 Sebastian Junges et al. 1 Introduction Unc ertainty. Probabilistic mo del c hecking subsumes a m ultitude of formal veriﬁca- tion techniques for systems that exhibit uncertainties [ 16 , 54 , 98 ]. Such systems are typically mo deled b y Marko v chains or Mark ov decision pro cesses [ 123 ]. Appli- cations range from reliabilit y , dep endability and performance analysis to systems biology , take for instance reliability measures such as the mean time b et ween failures in fault trees [ 28 , 125 ] and the probability of a system breakdown within a time limit. The results of probabilistic mo del chec king algorithms are rigorous, their quality dep ends solely on the system mo dels. Y et, there is one ma jor practical obstacle: All probabilities (or rates) in the Marko v mo del are precisely kno wn a priori . In man y cases, this assumption is to o severe. System quantities suc h as comp onent fault rates, molecule reaction rates, pack et loss ratios, etc. are often not, or at b est partially , kno wn. Let us give a few examples. The quality of service of a (wireless) comm unication channel ma y b e mo delled b y e.g., the p opular Gilbert- Elliott mo del, a t wo-state Marko v c hain in which pack et loss has an unkno wn probabilit y depending on the c hannel’s state [ 113 ]. Other examples include the bac k- oﬀ probabilit y in CSMA/CA proto cols determining a no de’s delay b efore attempting a transmission [ 1 ], the bias of used coins in self-stabilising proto cols [ 88 , 105 ], and the randomised choice of selecting the type of time-slots (sleeping, transmit, or idle) in the birthda y proto col, a key mec hanism used for neighbour discov ery in wireless sensor net works [ 110 ] to low er p ow er consumption. In particular, in early stages of reliable system design, the concrete failure rate of comp onents [ 55 ] is left unspeciﬁed. Optimally , analyses in this stage may ev en guide the c hoice of a concrete comp onen t from a particular man ufacturer. The probabilities in all these systems are deliberately left unsp eciﬁed. They can later be determined in order to optimise some p erformance or dep endability measure. Dually , some systems should be robust for all (reasonable) failure rates. F or example, a netw ork proto col should ensure a reasonable qualit y of service for eac h reasonable channel quality . Par ametric prob abilistic mo dels. What do these examples hav e in common? The random v ariables for pack et loss, failure rate etc. are not fully deﬁned, but are p ar ametric . Whether a parametric system satisﬁes a given prop ert y or not—“is the probabilit y that the system goes down within k steps below 10 − 8 ”—dep ends on these parameters. Relev ant questions are then: for whic h concrete parameter v alues is such a prop ert y satisﬁed—the (p ar ameter) synthesis pr oblem —and, in case of decision-making models, which parameter v alues yield optimal designs? That is, for whic h ﬁxed probabilities do such proto cols work in an optimal w ay , i.e., lead to maximal reliabilit y , maximise the probability for nodes to b e discov ered, or minimise the time un til stabilisation, and so on. These questions are intrinsically hard as parameters can take inﬁnitely man y diﬀerent v alues that, in addition, can dep end on eac h other. This pap er faces these challenges and pr esents various algorithmic te chniques to tr e at diﬀer ent variations of the (optimal) p arameter synthesis pr oblem . T o deal with uncertain ties in randomness, p ar ametric probabilistic mo dels are adequate. These mo dels are just lik e Marko v mo dels except that the transition probabilities are sp eciﬁed by arithmetic expressions ov er real-v alued parameters. T ransition Parameter Synthesis for Marko v Mo dels 3 s 0 s 1 s 2 s 3 s 4 s 5 s 6 2 / 5 3 / 5 7 / 10 3 / 10 2 / 5 3 / 5 3 / 5 2 / 5 7 / 10 3 / 10 2 / 5 3 / 5 3 / 5 2 / 5 (a) Unfair coins s 0 s 1 s 2 s 3 s 4 s 5 s 6 p 1 − p q 1 − q p 1 − p 1 − p p q 1 − q p 1 − p 1 − p p (b) P arametric probabilities Fig. 1 A (a) biased and (b) parametric v ariant of Knuth-Y ao’s algorithm. In gray states an unfair coin is ﬂipp ed with probability 2 / 5 for ‘heads’; for the unfair coin in the white states this probability equals 7 / 10 . On the right, the t wo biased coins ha ve parametric probabilities. probabilities are th us functions ov er a set of parameters. A simple instance is to use in terv als o ver system parameters imp osing constan t lo wer and upp er b ounds on ev ery parameter [ 74 , 100 ]. The general setting as considered here is more lib eral as it e.g., includes the p ossibility to express complex parameter dep endencies. W e address the analysis of p ar ametric Marko v mo dels where probability distributions are functions o ver system parameters, sp eciﬁcally , parametric discrete-time Mark ov chains (pMCs) and parametric discrete-time Marko v de cision pr oc esses (pMDPs). Example 1 The Kn uth-Y ao randomised algorithm [ 99 ] uses repeated coin ﬂips to mo del a six-sided die. It uses a fair coin to obtain eac h p ossible outcome (‘one’, ‘t wo’, ..., ‘six’) with probabilit y 1 / 6 . Figure 1(a) depicts a Marko v chain (MC) of a v ariant in whic h t wo unfair coins are ﬂipped in an alternating fashion. Flipping the unfair coins yields he ads with probability 2 / 5 (gra y states) or 7 / 10 (white states), resp ectiv ely . Accordingly , the probability of tails is 3 / 5 and 3 / 10 , resp ectiv ely . The ev ent of thro wing a ‘tw o’ corresp onds to reaching the state in the MC. Assume no w a sp e ciﬁc ation that requires the probabilit y to obtain ‘t wo’ to be larger than 3 / 20 . Knuth-Y ao ’s original algorithm accepts this sp eciﬁcation as using a fair coin results in 1 / 6 as probabilit y to end up in . The biased mo del, ho w ever, do es not satisfy the sp eciﬁcation; in fact, a ‘t wo’ is reac hed with probabilit y 1 / 10 . Pr ob abilistic mo del che cking. The analysis algorithms presented in this pap er are strongly related to (and presented as) tec hniques from probabilistic mo del chec king. Mo del che cking [ 13 , 46 ] is a p opular approac h to verify the correctness of a sy stem b y systematically ev aluating all p ossible system runs. It either certiﬁes the absence of undesirable (dangerous) behaviour or deliv ers a system run witnessing a violating system behaviour. T raditional model chec king typically tak es tw o inputs: a ﬁnite transition system mo delling the system at hand and a temporal logic formula sp ecifying a system requirement. Model chec king then amounts to c hecking whether the transition system satisﬁes the logical speciﬁcation, which in its simplest form describ es that a particular state can (not) b e reac hed. Mo del chec king is no wada ys a successful analysis technique adopted by mainstream hardware and softw are industry [49, 101]. 4 Sebastian Junges et al. T o cop e with real-world systems exhibiting random b eha viour, mo del chec king has b een extended to deal with probabilistic, typically Marko v, mo dels. Pr ob abilistic mo del che cking [ 13 , 16 , 98 ] tak es as input a Marko v mo del of the system at hand together with a quantitativ e sp eciﬁcation sp eciﬁed in some probabilistic extension of L TL or CTL. Example sp eciﬁcations are e.g., “is the probability to reach some bad (or degraded) state b elow a safet y threshold λ ?” or “is the exp ected time un til the system recov ers from a fault b ounded by some threshold κ ”. Eﬃcient probabilistic model-chec king tec hniques do exist for models such as discrete-time Mark ov chains (MCs), Marko v decision pro cesses (MDPs), and their con tinuous- time counterparts [ 98 ]. Probabilistic mo del chec king extends and complemen ts long-standing analysis tec hniques for Marko v mo dels. It has been adopted in the ﬁeld of p erformance analysis to analyse stochastic P etri nets [ 4 , 38 ], in dep endability analysis for analysing arc hitectural system descriptions [ 29 ], in reliability engineering for fault tree analysis [ 27 , 140 ], as w ell as in security [ 116 ], distributed computing [ 105 ], and systems biology [ 104 ]. Unremitting algorithmic impro v ements employing the use of symbolic techniques to deal with large state spaces hav e led to p ow erful and p opular softw are to ols realising probabilistic mo del chec king techniques such as PRISM [ 102 ] and Storm [ 65 ]. 1.1 Problem statements W e now give a more detailed description of the parameter synthesis problems considered in this paper. W e start oﬀ by establishing the connection b etw een parametric Mark ov mo dels and concrete ones, i.e., ones in which the probabilities are ﬁxed suc h as MCs and MDPs. Eac h parameter in a pMC or pMDP (where p stands for parametric) has a given parameter range. The p ar ameter sp ac e of the parametric mo del is the Cartesian pro duct of these parameter ranges. Instan tiating the parameters with a concrete v alue in the parameter space to the parametric mo del results in an instantiate d mo del. The parameter space deﬁnes all p ossible parameter instantiations, or equiv alently , the instantiated models. A parameter instan tiation that yields a Marko v mo del, e.g., results in probabilit y distributions, is called wel l-deﬁne d . In general, a parametric Marko v mo del deﬁnes an uncountably inﬁnite family of Markov models , where each family member is obtained b y a w ell-deﬁned instan tiation. A r e gion R is a fragmen t of the parameter space; it is w ell-deﬁned if all instantiations in R are well-deﬁned. Example 2 (pMC) Figure 1(b) on the previous page depicts a parametric v ersion of the biased Kn uth-Y ao die from Example 1 on the previous page. It has parameters V = { p, q } , where p is the probabilit y of outcome he ads in gray states and q the same for white states. The parameter space is { ( p, q ) | 0 < p, q < 1 } . The probability for tails is 1 − p and 1 − q , respectively . The sample instan tiation u with u ( p ) = 2 / 5 and u ( q ) = 7 / 10 is well-deﬁned and results in the MC in Figure 1(a) on the previous page. The region R = { u : V → R | 1 / 10 ≤ u ( p ) ≤ 9 / 10 and 3 / 4 ≤ u ( q ) ≤ 5 / 6 } is w ell-deﬁned. Contrarily , region R ′ = { u | 1 / 5 ≤ u ( p ) ≤ 6 / 5 and 2 / 5 ≤ u ( q ) ≤ 7 / 10 } Parameter Synthesis for Marko v Mo dels 5 is not well-deﬁned, as it con tains the instantiation u ′ with u ′ ( p ) = 6 / 5 whic h does not yield an MC. F or pMCs whose transition probabilities are high-degree polynomials, it is not alw ays obvious whether a region is w ell-deﬁned. W e are now in a p osition to describ e the three problems considered in this pap er. The veriﬁc ation pr oblem is deﬁne d as fol lows: The v eriﬁcation problem. Given a parametric Mark ov model D , a well-deﬁned region R , and a sp eciﬁcation φ , the veriﬁc ation pr oblem is to c heck whether al l instan tiations of D within R satisfy φ . Consider the follo wing p ossible outcomes: – If R only contains instantiations of D satisfying φ , then the veriﬁcation problem ev aluates to true and the Marko v mo del D on region R ac c epts sp eciﬁcation φ . Whenev er D and φ are clear from the con text, we call R acc epting . – If R con tains an instan tiation of D refuting φ , then the problem ev aluates to false . If R con tains only instantiations of D refuting φ , then D on R r eje cts φ . Whenev er D and φ are clear from the con text, we call R reje cting . – If R con tains instantiations satisfying φ as w ell as instantiations satisfying ¬ φ , then D on R is inconclusiv e w. r. t. φ . In this case, we call R inc onsistent . In case the veriﬁcation problem yields false for φ , one can only infer that the region R is not accepting, but not conclude whether R is inconsisten t or rejecting. T o determine whether R is rejecting, we need to consider the veriﬁcation problem for the negated sp eciﬁcation ¬ φ . Inconsistent regions for φ are also inconsistent for ¬ φ . Example 3 (V eriﬁc ation pr oblem) Consider the pMC D , the well-deﬁned region R from Example 2 on the previous page, and the speciﬁcation φ ′ := ¬ φ that constrains the probability to reac h to b e at most 3 / 20 . The veriﬁcation problem is to determine whether all instantiations of D in R satisfy φ ′ . As there is no instan tiation within R for which the probability to reach is ab ov e 3 / 20 , the v eriﬁcation problem ev aluates to true . Thus, R ac c epts φ ′ . T ypical structurally simple regions are describ ed by hyperrectangles or given by linear constrain ts, rather than non-linear constrain ts; w e refer to suc h regions as simple . A simple region comprising a large range of parameter v alues may lik ely be inconsisten t, as it con tains b oth instan tiations satisfying φ , and some satisfying ¬ φ . Th us, we generalise the problem to synthesise a partition of the parameter space. The exact synthesis pr oblem is describ e d as fol lows: The syn thesis problem. Given a parametric Marko v mo del D and a speciﬁca- tion φ , the (p ar ameter) synthesis problem is to partition the parameter space of D in to an accepting region R a and a rejecting region R r for φ . The aim is to obtain such a partition in an automated manner. A complete sub- division of the parameter space into accepting and rejecting regions pro vides deep insigh t in to the eﬀect of parameter v alues on the system’s b ehaviour. The exact division t ypically is describ ed b y non-linear functions o ver the parameters, referred to as solution functions . 6 Sebastian Junges et al. Example 4 Consider the pMC D , the region R , and the sp eciﬁcation φ as in Exam- ple 3 on the previous page. The solution function: f φ ( p, q ) = p · (1 − q ) · (1 − p ) 1 − p · q describ es the probability to ev entually reach . Given that φ imp oses a low er b ound of 3 / 20 , we obtain R a = { u | f ( u ( p ) , u ( q )) ≥ 3 / 20 } and R r = R \ R a . The example illustrates that exact sym b olic represen tations of the accepting and rejecting regions may be complex and hard to compute algorithmically . The primary reason is that the b oundaries are describ ed by non-linear functions. A viable alternativ e therefore is to consider an appro ximative version of the synthesis problem. The appr oximate synthesis pr oblem: As argued before, the regions obtained via exact syn thesis are typically not simple. The aim of the approximate syn thesis problem is to use simpler and more tractable representations of regions. As such shap es ultimately approximate the exact solution function, simple regions b ecome inﬁnites- imally small when getting close to the b order b etw een accepting and rejecting areas. F or computational tractability , w e are thus in terested in appr oximating a partition of the parameter space in accepting and rejecting regions, where we allo w also for a (t ypically small) part to b e co vered by possibly inconsistent regions. Practically this means that c % of the entire parameter space is co vered b y simple regions that are either accepting or rejecting, for some adequate v alue of c . Altogether this results in the follo wing problem description: The approximate synthesis problem. Given a parametric Marko v mo del, a sp eciﬁcation φ , and a p ercen tage c , the appr oximate (p arameter) synthesis pr oblem is to partition the parameter space of D in to a simple accepting region R a and a simple rejecting region R r for φ suc h that R a ∪ R r co ver at least c % of the en tire parameter space. Example 5 Consider the pMC D , the region R , and the sp eciﬁcation φ as in Exam- ple 3 on the previous page. The parameter space in Figure 2 on the following page is partitioned into simple regions (rectangles). The green (dotted) area—the union of a n umber of smaller rectangular accepting regions—indicates the parameter v alues for which φ is satisﬁed, whereas the red (hatched) area indicates the set of rejecting regions for φ . The white area indicates the unknown regions. The indicated partition co vers 95% of the parameter space. The sub-division into accepting and rejecting (simple) regions approximates the solution function f φ ( p, q ) giv en b efore. 1.2 Solution approaches W e now outline our approac hes to solve the v eriﬁcation problem and the tw o syn thesis problems. F or the sake of conv enience, we start with the syn thesis problem. Parameter Synthesis for Marko v Mo dels 7 0.0 0.2 0.4 0.6 0.8 1.0 x 0.0 0.2 0.4 0.6 0.8 1.0 y Fig. 2 P arameter space partitioning into accepting (green), rejecting (red), and unknown (white) regions. Synthesis. The most straightforw ard description of the sets R a and R r is of the form: R a = { u | D [ u ] satisﬁes φ } and R r = { u | D [ u ] satisﬁes ¬ φ } . The satisfaction relation (denoted | =) can b e concisely describ ed by a set of linear equations ov er the transition probabilities [ 13 ]. As in the parametric setting the transition probabilities are no longer ﬁxed, but rather deﬁned ov er a set of parameters, the equations b ecome non-linear. Example 6 (Non-line ar e quations for r e achability) T ake the MC from Figure 1(a) on page 3. T o compute the probability of ev entually reaching, e.g., state , one in tro duces a v ariable p s for eac h transient state s enco ding that probabilit y for s . F or state s 0 and v ariable p s 0 , the corresp onding linear equation reads: p s 0 = 2 / 5 · p s 1 + 3 / 5 · p s 2 , where p s 1 and p s 2 are the v ariables for s 1 and s 2 , resp ectiv ely . The corresp onding equation for the pMC from Figure 1(b) on page 3 reads: p s 0 = p · p s 1 + (1 − p ) · p s 2 . The m ultiplication of parameters in the model and equation v ariables leads to a non-line ar equation system. Th us, we can describ e the sets R a and R r collo quially as: R a , R r = { u | u satisﬁes a set of non-linear constrain ts } . W e provide further details on these constraint systems in Section 6. 8 Sebastian Junges et al. syn thesise description of: accepting region R a , and rejecting region R r parametric MDP M , parameter space R , speciﬁcation φ c heck R ∧ R a unsatisﬁable, c heck R ∧ R r unsatisﬁable yes for R a → reject, yes for R r → accept, otherwise → unknown Fig. 3 V eriﬁcation via exact syn thesis A practical drawbac k of the resulting equation system is the substantial num b er of auxiliary v ariables p s , one for each state in the pMC. A viable p ossibility for pMCs is to simplify the equations by (v ariants of ) state elimination [ 63 ]. This pro cedure successiv ely remov es states from the pMC un til only a start and ﬁnal state (represen ting the reachabilit y ob jective) remain that are connected by a transition whose label is (a mild v ariant of ) the solution function f φ that exactly describ es the probabilit y to reac h a target state: R a = { u | f φ ( u ) > 0 } and R r = { u | f ¬ φ ( u ) > 0 } . W e recapitulate state elimination and present several alternativ es in Section 5. V eriﬁc ation. The basic approach to the veriﬁcation problem is depicted in Figure 3. W e use a description of the accepting region as computed via the syn thesis procedure ab o ve. Then, we com bine the description of the accepting region with the region R to be veriﬁed, as follows: A region R accepts a speciﬁcation, if R ∩ R a = R , or equiv alently , if R ∩ R r = ∅ . The existence of a rejecting instance in R is thus of relev ance; if such a p oint do es not exist, the region is accepting. Using R a and R r as obtained ab ov e, the query “is R ∩ R r = ∅ ?” can b e solved via satisﬁability mo dulo the ories (SMT) o ver non-linear arithmetic, chec king the conjunction ov er the corresp onding constraints for unsatisﬁability . With the help of SMT solvers o ver this theory like Z3 [ 93 ], MathSAT [ 31 ], or SMT-RAT [ 52 ], this can b e solved in a fully automated manner. This pro cedure is complete, and is computationally in volv ed. Details of the pro cedure are discussed in Section 6. Par ameter lifting [ 124 ] is an alternativ e, appr oximative solution to the v eriﬁca- tion problem. Intuitiv ely , this approach ov er-approximates R r for a given R , by ignoring parameter dep endencies. Region R is accepted if the intersection with the o ver-appro ximation of R r is empt y . This pro cedure is sound but ma y yield false negativ es as a rejecting p oint ma y lie in the ov er-approximation but not in R r . Tigh tening the ov er-approximation mak es the approach complete. A ma jor b eneﬁt of parameter lifting (details in Section 7 and Section 8) is that the intersection with the o ver-appro x imation of R r can be inv estigated by standard probabilistic Parameter Synthesis for Marko v Mo dels 9 mo del-c hecking pro cedures. This applicability of mature to ols results—as will be sho wn in Section 11—in a practically eﬃcient pro cedure. Appr oximate synthesis. W e solv e the appro ximate synthesis problem with an iterative syn thesis lo op. Here, the central issue is to obtain represen tations of R a and R r b y simple regions. Our approach for this p ar ameter sp ac e p artitioning therefore iterativ ely obtains partial partitions of the parameter space. The main idea is to compute a sequence  R i a  i of simple accepting regions that successively extend each other. Similarly , an increasing sequence  R i r  i of simple rejecting regions is computed. The typical approach is to let R i +1 a b e the union of R i a , the approximations in the previous iteration, together with some accepting region with a simple representation. Rejecting regions are handled analogously . At the i -th iteration, R i a ∪ R i r is the co vered fragmen t of the parameter space. The iterativ e approac h halts when this fragment forms at least c % of the entire parameter space. T ermination is guaran teed. In the limit, the accepting and rejecting regions conv erge to the exact solution, lim i →∞ R i a = R a and lim i →∞ R i r = R r , under some mild constraints on the ordering of the regions R i . Figure 4 outlines a procedure to address the appro ximate syn thesis problem. As part of our synthesis method, we algorithmically guess a (candidate) region R and guess whether it is accepting or rejecting. W e then exploit one of our v eriﬁcation metho ds to verify whether R is indeed accepting (or rejecting). If it is not accepting (rejecting), w e exploit this information together with any additional information obtained during veriﬁcation to reﬁne the candidate region. This pro cess is rep eated un til an accepting or rejecting region results. W e discuss the metho d and essential impro vemen ts in Section 9. Example 7 Consider the pMC D and the sp eciﬁcation φ as in Example 2 on page 4. The parameter space in Figure 2 on page 7 is partitioned into regions. The green (dotted) area—the union of a num b er of smaller rectangular accepting regions— indicates the parameter v alues for whic h φ is satisﬁed, whereas the red (hatc hed) area indicates the set of rejecting regions for φ . Chec king whether a region is accepting, rejecting, or inconsisten t is done by veriﬁcation. The small white area consists of regions that are unknown (i.e., not yet considered) or inconsisten t. 1.3 Overview of the pap er Section 2 in tro duces the required formalisms and concepts. Section 3 deﬁnes the notion of a region and formalises the three problems: the v eriﬁcation problem and the t wo syn thesis problems. It ends with a bird’s eye view of the veriﬁcation approac hes that are later discussed in detail. Section 4 details sp eciﬁc region structures and pro cedures to chec k elemen tary region prop erties suc h as well-deﬁnedness and graph-preserv edness, tw o prerequisites for the veriﬁcation procedures. Section 5 sho ws how to do exact syn thesis by computing the solution function. Sections 6–8 presen t algorithms for the veriﬁcation problem. Section 9 details the approach to reduce the syn thesis problem to a series of v eriﬁcation problems. Sections 10 and 11 con tain information ab out the implementation of the approaches, as well as an extensive experimental ev aluation. Section 12 contains a discussion of the approac hes and related work. Section 13 concludes with an outlo ok. 10 Sebastian Junges et al. reﬁne undecided region: guess candidate v eriﬁcation : either M , R ′ | = φ (accept) or M , R ′ | = ¬ φ (reject) parametric MDP M , parameter space R , speciﬁcation φ accepting/rejecting regions region R ′ and h yp othesis φ or ¬ φ not ac c epte d / r eje cte d ac c epte d / r eje cte d Fig. 4 Approximate synthesis process using veriﬁcation as black b ox. 1.4 Contributions of this pap er The pap er is lo osely based on the conference pap ers [ 64 ] and [ 124 ] and extends these works in the following wa ys. It gives a uniform treatment of the solution tec hniques to the synthesis problem, and treats all techniques uniformly for all diﬀeren t ob jectives—bounded and un b ounded reac habilit y as well as exp ected rew ard sp eciﬁcations. The material on SMT-based region veriﬁcation has b een extended in the follo wing wa y: The pap er gives the complete characterisations of the SMT encoding with or without solution function. F urthermore, it is the ﬁrst to extend this enco ding to MDPs under angelic and demonic non-determinism and includes an explicit and in-depth discussion on exact region c hecking via SMT c heck ers. It presents a uniform treatmen t of the linear equation system for Mark ov c hains and its relation to state elimination and Gaussian elimination. It presents a no vel and simpliﬁed description of state elimination for expected rewards, and a v ersion of state elimination that is targeted to wards MTBDDs. The pap er con tains a correctness proof of approximate v eriﬁcation for a wider range of pMDPs and con tains pro ofs for exp ected rewards. It also supp orts exp ected-time prop erties for parametric contin uous-time MDPs (via the em b edded pMDP). Nov el heuristics hav e b een dev elop ed to improv e the iterative synthesis loop. All presented techniques, mo dels, and speciﬁcations are realised in the state-of-the-art to ol PROPhESY 1 . 1 PROPhESY is av ailable on https://github.com/moves- rwth/prophesy . Parameter Synthesis for Marko v Mo dels 11 2 Preliminaries 2.1 Basic notations W e denote the set of real num b ers by R , the rational num b ers by Q , and the natural n umbers including 0 b y N . Let [0 , 1] ⊆ R denote the close d interval of all real n umbers b et ween 0 and 1, including the b ounds; (0 , 1) ⊆ R denotes the op en interval of all real num b ers b et ween 0 and 1 excluding 0 and 1. Let X, Y denote arbitrary sets. If X ∩ Y = ∅ , we write X ⊎ Y for the disjoint union of the sets X and Y . W e denote the p ower set of X b y 2 X = { X ′ | X ′ ⊆ X } . Let X b e a ﬁnite or coun tably inﬁnite set. A pr ob ability distribution ov er X is a function µ : X → [0 , 1] ⊆ R with P x ∈ X µ ( x ) = µ ( X ) = 1. 2.2 Polynomials, rational functions Let V denote a ﬁnite set of parameters ov er R and dom ( p ) ⊆ R denote the domain of parameter p ∈ V . Deﬁnition 1 (P olynomial, rational function) F or a ﬁnite set V = { p 1 , . . . , p n } of n parameters, a monomial m is m = p e 1 1 · . . . · p e n n with e i ∈ N . Let Mon [ V ] denote the set of monomials o ver V . A p olynomial g (o ver V ) with t terms is a weigh ted sum of monomials: g = t X j =1 a j · m j with a j ∈ Q \ { 0 } , m j ∈ Mon [ V ] . Let Q [ V ] be the set of p olynomials ov er V . A r ational function f = g 1 g 2 o ver V is a fraction of p olynomials g 1 , g 2 ∈ Q [ V ] with g 2 ≡ 0 (where ≡ states equiv alence). Let Q ( V ) b e the set of rational functions ov er V . A monomial is line ar , if P | V | i =1 e i ≤ 1, and multi-line ar , if e i ≤ 1 for all 1 ≤ i ≤ | V | . A p olynomial g is (multi-)linear, if all monomials o ccurring in g are (multi-)linear. Instantiations replace parameters by constan t v alues in p olynomials or rational functions. Deﬁnition 2 (P arameter instan tiations) A (p ar ameter) instantiation u of p ar ame- ters V is a function u : V → R . W e abbreviate the parameter instantiation u with u ( p i ) = a i ∈ R b y the n - dimensional v ector ( a 1 , . . . , a n ) ∈ R n for ordered parameters p 1 , . . . , p n . Applying the instan tiation u on V to polynomial g ∈ Q [ V ] yields g [ u ] whic h is obtained by replacing each p ∈ V in g b y u ( p ), with subsequent application of + and · . F or rational function f = g 1 g 2 , let f [ u ] = g 1 [ u ] g 2 [ u ] ∈ R if g 2 [ u ] ≡ 0, and otherwise f [ u ] = ⊥ . 12 Sebastian Junges et al. 2.3 Probabilistic mo dels Let us no w in tro duce the probabilistic models used in this pap er. W e ﬁrst deﬁne parametric Mark o v mo dels and presen t conditions such that their instantiations result in Marko v mo dels with constant probabilities. Then, we discuss ho w to resolv e non-determinism in decision pro cesses. 2.3.1 Par ametric Markov mo dels The transitions in parametric Marko v models are equipp ed with rational functions o ver the set of parameters. Although this is the general setting, for some of our algorithmic tec hniques we will restrict ourselv es to linear p olynomials 2 . W e consider parametric MCs and MDPs as sub-classes of a parametric version of classical t wo- pla yer stochastic games [ 130 ]. The state space of suc h games is partitioned into t wo parts, S  and S 2 . A t each state, a play er chooses an action upon which the successor state is determined according to the (parametric) probabilities. Choices in S  and S 2 are made b y pla yer  and 2 , respectively . pMDPs and pMCs are parametric sto c hastic one- and zero-pla yer games resp ectively . Deﬁnition 3 (P arametric mo dels) A p ar ametric sto chastic game (pSG) is a tuple G = ( S , V , s I , A ct , P ) with a ﬁnite set S of states with S = S  ⊎ S 2 , a ﬁnite set V of parameters o ver R , an initial state s I ∈ S , a ﬁnite set A ct of actions, and a transition function P : S × A ct × S → Q ( V ) ∪ R ∪ {⊥} with | A ct ( s ) | ≥ 1 for all s ∈ S , where A ct ( s ) = { α ∈ Act | ∃ s ′ ∈ S. P ( s, α, s ′ ) ≡ 0 } is the set of enable d actions at state s . – A pSG is a p ar ametric Markov de cision pr o c ess (pMDP) if S  = ∅ or S 2 = ∅ . – A pMDP is a p ar ametric Markov chain (pMC) if | A ct ( s ) | = 1 for all s ∈ S . A parametric state-action r ewar d function rew : S × A ct → Q ( V ) ∪ R ∪ {⊥} asso ciates rew ards with state-action pairs 3 . It is assumed that deadlock states are absen t, i.e., A ct ( s )  = ∅ for all s ∈ S . Entries in R ∪ {⊥} in the co-domains of the functions P and rew ensure that the model is closed under instantiations, see Deﬁnition 5 on page 14 b elow. Throughout the rest of this pap er, we silently assume that any giv en pSGs only uses constants from Q and rational functions Q ( V ), but no elements from R \ Q or ⊥ . A model is called p ar ameter-fr ee if all its transition probabilities are constan t. A pSG intuitiv ely works as follows. In state s ∈ S  , pla yer  non-deterministically selects an action α ∈ A ct ( s ). With (parametric) probability P ( s, α, s ′ ) the pla y then ev olves to state s ′ . On leaving state s via action α , the reward rew ( s, α ) is earned. If s ∈ S 2 , the choice is made by play er 2 , and as for play er  , the next state is determined in a probabilistic w ay . As b y assumption no deadlo ck states o ccur, this game go es on forever. A pMDP is a game with one play er, whereas a pMC has no pla yers; a pMC th us evolv es in a fully probabilistic wa y . Let D denote a pMC, M a pMDP , and G a pSG. Example 8 Figure 5 on the following page(a)–(c) depict a pSG, a pMDP , and a pMC respectively o ver parameters V = { p, q } . The states of the play ers  and 2 2 Most models use only simple polynomials such as p and 1 − p , and benchmarks av ailable e.g., at the PRISM benchmark suite [103] or at the PARAM [77] web page are of this form. 3 Recall that ⊥ represents, e.g., 1 / 0 . Parameter Synthesis for Marko v Mo dels 13 s 0 s 1 s 2 s 3 s 4 β 1 − p · q p · q α 1 − p p q 1 − q α 3 / 5 2 / 5 β 1 − q q 1 1 (a) Sample pSG G s 0 s 1 s 2 s 3 s 4 β 1 − p · q p · q α 1 − p p q 1 − q q 1 − q 1 1 (b) Sample pMDP M s 0 s 1 s 2 s 3 s 4 p 1 − p q 1 − q q 1 − q 1 1 (c) Sample pMC D s 0 s 1 s 2 s 3 s 4 4 / 5 1 / 5 3 / 5 2 / 5 3 / 5 2 / 5 1 1 (d) MC D [ u ] with u ( p ) = 4 / 5 and u ( q ) = 3 / 5 Fig. 5 The considered t ypes of parametric probabilistic mo dels (a)–(c) and an instan tiated model (d). are dra wn as circles and rectangles, resp ectively . The initial state is indicated by an incoming arro w without source. W e omit actions in state s if | A ct ( s ) | = 1. In state s 0 of Figure 5(a), play er  can select either action α or β . On selecting α , the game mov es to state s 1 with probability p , and to s 2 with probability 1 − p . In state s 2 , pla yer 2 can select α or β ; in s 1 there is a single c hoice only . A tr ansition ( s, α, s ′ ) exists if P ( s, α, s ′ ) ≡ 0. As pMCs ha v e a single enabled action at each state, we omit this action and just write P ( s, s ′ ) for P ( s, α, s ′ ) if A ct ( s ) = { α } . A state s ′ is a suc cessor of s , denoted s ′ ∈ succ ( s ), if P ( s, α, s ′ ) ≡ 0 for some α ; in this case, s ∈ pred( s ′ ) is a predecessor of s ′ . R emark 1 P arametric sto chastic games are the most general mo del used in this pap er. They subsume pMDPs and pMCs and parameter-free SGs, which are used throughout this pap er. W e concisely introduce the formal foundations on this general class and indicate how these apply to sub classes. Most algorithmic approac hes in this pap er are not directly applicable to pSGs, but tailored to either pMDPs or pMCs. This is indicated when in tro ducing these tec hniques. 14 Sebastian Junges et al. Deﬁnition 4 (Sto c hastic game) A pSG G = ( S , V , s I , A ct , P ) is a sto chastic game (SG) if P : S × A ct × S → [0 , 1] and P s ′ ∈ S P ( s, α, s ′ ) = 1 for all s ∈ S and α ∈ A ct ( s ). A state-action reward function rew : S × A ct → R ≥ 0 asso ciates (non-negative, ﬁnite) rewards to outgoing actions. Analogously , Marko v c hains (MCs) and Mark ov decision processes (MDPs) are deﬁned as sp ecial cases of pMCs and pMDPs, resp ectiv ely . W e use D to denote a MC, M for an MDP and G for an SG. 2.3.2 Paths and r e achability An inﬁnite p ath of a pSG G is an inﬁnite sequence π = s 0 α 0 s 1 α 1 . . . of states s i ∈ S and actions α i ∈ A ct ( s i ) with P ( s i , α i , s i +1 ) ≡ 0 for i ≥ 0. A ﬁnite path of a pSG G is a non-empty ﬁnite preﬁx s 0 α 0 . . . s n of an inﬁnite path s 0 α 0 . . . s n α n . . . of G for some n ∈ N . Let Paths G denote the set of all ﬁnite or inﬁnite paths of G while Paths G ﬁn ⊆ Paths G denotes the set of all ﬁnite paths. F or paths in (p)MCs, w e omit the actions. The set Paths G ( s ) contains all paths that start in state s ∈ S . F or a ﬁnite path π ∈ Paths G ﬁn , last ( π ) = s n denotes the last state of π . The length | π | of a path π is | π | = n for π ∈ Paths G ﬁn and | π | = ∞ for inﬁnite paths. The ac cumulate d r ewar d along the ﬁnite path s 0 α 0 . . . α n − 1 s n is given b y the sum of the rewards rew( s i , α i ) for 0 ≤ i < n . W e denote the set of states that can reac h a set of states T as follo ws: ♢ T = { s ∈ S | ∃ π ∈ Paths G ﬁn ( s ) . last ( π ) ∈ T } . A set of states T ⊆ S is r e achable from s ∈ S , written s ∈ ♢ T , iﬀ there is a path from s to some s ′ ∈ T . A state s is absorbing iﬀ P ( s, α, s ) = 1 for all α ∈ A ct ( s ). Example 9 The pMC in Figure 5(c) on the previous page has a path π = s 0 s 1 s 3 s 3 with | π | = 3. Thus s 0 ∈ ♢ { s 3 } . There is no path from s 4 to s 3 , so s 4 ∈ ♢ { s 3 } . States s 3 and s 4 are the only absorbing states. 2.3.3 Mo del instantiation Instan tiated parametric models are obtained b y instantiating the rational functions in all transitions as in Deﬁnition 2 on page 11. Deﬁnition 5 (Instan tiated pSG) F or a pSG G = ( S , V , s I , A ct , P ) and instanti- ation u of V , the instantiated pSG at u is giv en by G [ u ] = ( S, s I , A ct , P [ u ]) with P [ u ]( s, α, s ′ ) = P ( s, α, s ′ )[ u ] for all s, s ′ ∈ S and α ∈ A ct . The instan tiation of the parametric reward function rew at u is rew [ u ] with rew [ u ]( s, α ) = rew ( s, α )[ u ] for all s ∈ S, α ∈ Act . Instan tiating pMDP M and pMC D at u is denoted by M [ u ] and D [ u ], resp ectively . R emark 2 The instantiation of a pSG at u is a pSG, but not necessarily an SG. This is due to the fact that an instantiation do es not ensure that P ( s, α, · ) is a probabilit y distribution. In fact, instantiation yields a transition function of the form P : S × A ct × S → R ∪ {⊥} . Similarly , there is no guaran tee that the rew ards rew [ u ] are non-negative. Therefore, we imp ose restrictions on the parameter instan tiations. Deﬁnition 6 (W ell-deﬁned instantiation) An instantiation u is wel l-deﬁne d for a pSG G if the pSG G [ u ] is an SG. Parameter Synthesis for Marko v Mo dels 15 The reward function rew is well-deﬁned on u if it do es only asso ciate non-negativ e reals to state-action pairs. Example 10 Consider again the pMC in Figure 5(c) on page 13. The instantiation u with u ( p ) = 4 / 5 and u ( q ) = 3 / 5 is well-deﬁned and induces the MC D [ u ] depicted in Figure 5(d) on page 13. F rom now on, we silen tly assume that every pSG w e consider has at least one w ell-deﬁned instantiation. This condition can b e assured through chec king the satisﬁabilit y of the conditions in Def. 4 on the previous page, whic h we discuss in Section 4.2. Our metho ds necessitate instantiations that are not only well-deﬁned, but also preserve the top ology of the pSG. In particular, we are interested in the setting where reachabilit y b etw een tw o states coincides for the pSG and the set of instan tiations u we consider. W e detail this discussion in Section 4.2. Deﬁnition 7 (Graph preserving) A w ell-deﬁned instantiation u for pSG G = ( S , V , s I , A ct , P ) is gr aph pr eserving if for all s, s ′ ∈ S and α ∈ A ct , P ( s, α, s ′ ) ≡ 0 = ⇒ P ( s, α, s ′ )[ u ] ∈ R \ { 0 } . Example 11 The w ell-deﬁned instantiation u with u ( p ) = 1 and u ( q ) = 3 / 5 for the pMC in Figure 5(c) on page 13 is not gr aph pr eserving . 2.3.4 R esolving non-determinism Str ate gies 4 resolv e the non-deterministic c hoices in stochastic games with at least one pla yer. F or the ob jectives considered here, it suﬃces to consider so-called deterministic strategies [ 137 ]; more general strategies can b e found in [ 13 , Ch. 10]. W e deﬁne strategies for pSGs and assume well-deﬁned instantiations as in Deﬁnition 6 on the previous page. Deﬁnition 8 (Strategy) A (deterministic) str ate gy σ i for play er i ∈ {  , 2 } in a pSG G with state space S = S  ⊎ S 2 is a function σ i : { π ∈ Paths G ﬁn | last ( π ) ∈ S i } → Act suc h that σ i ( π ) ∈ A ct ( last ( π )). Let Str G denote the set of strategies σ = ( σ  , σ 2 ) for pSG G and Str G i the set of strategies of pla yer i . A pMDP has only a play er- i strategy for the play er with S i  = ∅ ; in this case the index i is omitted. A play er- i strategy σ i is memoryless if last ( π ) = last ( π ′ ) implies σ i ( π ) = σ i ( π ′ ) for all ﬁnite paths π , π ′ . A memoryless strategy can thus b e written in the form σ i : S i → A ct . A pSG-strategy σ = ( σ  , σ 2 ) is memoryless if b oth σ  and σ 2 are memoryless. R emark 3 F rom now on, w e only consider memoryless strategies and refer to them as strategies. A strategy σ for a pSG resolves all non-determinism and results in an induc e d pMC . 4 Also referred to as policies, adversaries, or schedulers. 16 Sebastian Junges et al. Deﬁnition 9 (Induced pMC) The pMC G σ induc e d by str ate gy σ = ( σ  , σ 2 ) on pSG G = ( S , V , s I , A ct , P ) equals ( S, V , s I , P σ ) with: P σ ( s, s ′ ) = ( P ( s, σ  ( s ) , s ′ ) if s ∈ S  P ( s, σ 2 ( s ) , s ′ ) if s ∈ S 2 . Example 12 Let σ b e a strategy for the pSG G in Figure 5 on page 13(a) with σ  ( s 0 ) = α and σ 2 ( s 2 ) = β . The induced pMC G σ equals pMC D in Figure 5 on page 13(c). Analogously , imp osing strategy σ ′ with σ ′ ( s 0 ) = α on the pMDP in Figure 5 on page 13(b) yields M σ ′ = D . The notions of strategies for pSGs and pMDPs and of induced pMCs naturally carry o ver to non-parametric mo dels; e.g., the MC G σ is induced b y strategy σ ∈ Str G on SG G . 2.4 Sp eciﬁcations and solution functions 2.4.1 Sp e ciﬁc ations Sp eciﬁcations constrain the measures of interest for (parametric) probabilistic mo dels. Before considering parameters, let us ﬁrst consider MCs. Let D = ( S , s I , P ) b e an MC and T ⊆ S a set of tar get states that (without loss of generalit y) are assumed to b e absorbing. Let ♢ T denote the path prop erty to reac h T 5 . F urthermore, the pr ob ability me asure Pr s o ver sets of paths can b e deﬁned using a cylinder construction with Pr s ( s 0 α 0 . . . s n ) = Π n − 1 i =0 P ( s i , α i , s i +1 ), see [ 13 , Ch. 10]. W e consider three kinds of sp eciﬁcations: 1. Unb ounde d prob abilistic r e achability A sp eciﬁcation P ≤ λ ( ♢ T ) asserts that the probabilit y to reach T from the initial state s I shall b e at most λ , where λ ∈ Q ∩ [0 , 1]. More generally , sp eciﬁcation φ r is satisﬁed b y MC D , written: D | = P ∼ λ ( ♢ T ) iﬀ Pr D s I ( ♢ T ) ∼ λ, where Pr D s I ( ♢ T ) is the probabilit y mass of all inﬁnite paths that start in s I and visit an y state from T . 2. Bounde d pr ob abilistic r e achability In addition to reachabilit y , these sp eciﬁcations imp ose a b ound on the maximal n umber of steps until reac hing a target state. Sp eciﬁcation φ b = P ∼ λ ( ♢ ≤ n T ) asserts that in addition to P ∼ λ ( ♢ T ), states in T should b e reached within n ∈ N steps. The satisfaction of P ∼ λ ( ♢ ≤ n T ) is deﬁned similar as ab o v e. 3. Exp e cted r ewar d until a tar get The sp eciﬁcation E ≤ κ ( ♢ T ) asserts that the ex- p ected reward until reaching a state in T shall b e at most κ ∈ R . Let ER D s I ( ♢ T ) denote the expected accumulated rew ard until reac hing a state in T ⊆ S from state s I . W e obtain this rew ard by multiplying the probability of every path 5 Thereby ov erloading the earlier notation to denote the set of states for which there exists a path on which this prop erty holds. Parameter Synthesis for Marko v Mo dels 17 reac hing T with the accumulated reward of that path, up un til reaching T . Details are giv en in [13, Chapter 10]. 6 . Then w e deﬁne D | = E ∼ κ ( ♢ T ) iﬀ ER D s I ( ♢ T ) ∼ κ, W e do not treat the accumulated rew ard to reac h a target within n steps, as this is not a very useful measure. In case there is a p ossibility to not reac h the target within n steps, this yields ∞ . W e omit the superscript D if it is clear from the con text. W e write ¬ φ to inv ert the relation: D | = ¬ P ≤ λ ( ♢ T ) is thus equiv alent to D | = P >λ ( ♢ T ). An SG G satisﬁes sp eciﬁcation φ under strategy σ if the induced MC G σ | = φ . Un b ounded reac hability and exp ected rewards are prominent examples of indeﬁnite-horizon properties – they measure b ehaviour up-to some sp eciﬁed even t (the horizon) which ma y b e reac hed after arbitrarily many steps. R emark 4 Bounded reac hability in MDPs can b e reduced to un b ounded reachabilit y b y a technique commonly referred to as unr ol ling [ 5 ]. F or p erformance reasons, it is sometimes b etter to a v oid this unrolling, and present dedicated approaches. 2.4.2 Solution functions Computing (un b ounded) reachabilit y probabilities and exp ected rewards for MCs reduces to solving linear equation systems [ 13 ] ov er the ﬁeld of reals (or rationals). F or parametric MCs, we obtain a linear equation system ov er the ﬁeld of the rational functions o ver V instead. The solution to this equation system is a rational function. (See Examples 4 and 6 on pages 6 and 7). More details on the the solution function and the equation system follo w in Section 5 and Section 6, respectively . Deﬁnition 10 (Solution functions) F or a pMC D = ( S , V , s I , P ), T ⊆ S and n ∈ N , a solution function for a sp eciﬁcation φ is a rational function f r D ,T ∈ Q ( V ) for φ = P ∼ λ ( ♢ T ) f b D ,T ,n ∈ Q ( V ) for φ = P ∼ λ ( ♢ ≤ n T ) , and f e D ,T ∈ Q ( V ) for φ = E ∼ κ ( ♢ T ) , suc h that for every well-deﬁned graph-preserving instan tiation u : f r D ,T [ u ] = Pr D [ u ] s I ( ♢ T ) , f b D ,T ,n [ u ] = Pr D [ u ] s I ( ♢ ≤ n T ), and f e D ,T [ u ] = ER D [ u ] s I ( ♢ T ) . Example 13 Consider the reachabilit y probability to reac h s 2 for the pMC in Figure 6(a) on the next page. Any instan tiation u with u ( p ) , u ( q ) ∈ (0 , 1) is w ell- deﬁned and graph-preserving. As the only tw o ﬁnite paths to reac h s 2 are s 0 s 2 and s 0 s 1 s 2 , w e hav e f r D , { s 2 } = 1 − p + p · q . 6 As standard, if Pr D s I ( ♢ T ) < 1 then we set ER D s I ( ♢ T ) : = ∞ . The rationale is that an inﬁnite amount of reward is collected on visiting a state (with positive reward) inﬁnitely often from which all target states are unreachable. 18 Sebastian Junges et al. s 0 s 1 s 2 s 3 p q 1 − p 1 − q 1 1 (a) pMC s 0 s 1 s 2 s 3 α β 1 p q 1 − p 1 − q 1 1 (b) pMDP Fig. 6 Two sample parametric mo dels. F or pSGs (and pMDPs), the solution function dep ends on the resolution of non- determinism b y strategies, i. e., they are deﬁned on the induced pMCs. F ormally , a solution function for a pSG G , a reachabilit y sp eciﬁcation φ r = P ≤ λ ( ♢ T ), and a strategy σ ∈ Str G is a function f r G ,σ,T ∈ Q ( V ) such that for each w ell-deﬁned graph-preserving instan tiations u it holds: f r G ,σ,T [ u ] = Pr G σ [ u ] s I ( ♢ T ) . These notions are deﬁned analogously for b ounded reac hability (denoted f b G ,σ,T ,n ) and exp ected rew ard (denoted f e G ,σ,T ) sp eciﬁcations. Example 14 F or the pMDP in Figure 6(b), the solution functions for reaching s 2 are 1 − p + p · q , for the strategy σ α = { s 0 7→ α } , and 1 for the strategy σ β = { s 0 7→ β } . R emark 5 W e deﬁne solution functions only for graph-preserving v aluations. F or the more general w ell-deﬁned solutions, a similar deﬁnition can b e giv en [ 94 ] where (solution) functions are no longer rational functions but instead a collection of solution functions obtained on the graph-preserving subsets. In particular, unless a pMC is acyclic, such a function is only semi-con tinuous [ 97 ]. A key reason for the discon tinuit y is the change of states that are in ♢ T , e.g., consider instantiations with q = 1 in Figure 5(c) on page 13. W e pro vide the decomp osition in to graph-preserving subsets in Section 4.3. 2.5 Constraints and formulas W e consider (p olynomial) c onstr aints of the form g ∼ g ′ with g , g ′ ∈ Q [ V ] and ∼∈ { <, ≤ , = , ≥ , > } . W e denote the set of all constraints ov er V with C [ V ]. A constrain t g ∼ g ′ can b e equiv alently formulated as g − g ′ ∼ 0. A formula ψ o ver a set of p olynomial constrain ts is recursively deﬁned: Eac h p olynomial constrain t is a form ula, and the Bo olean com bination of formulae is also a formula. Example 15 Let p, q b e v ariables. 1 − p · q > 0 and p 2 < 0 are constrain ts, ¬  p 2 < 0  and ( 1 − p · q > 0 ) ∨  p 2 < 0  are form ulae. The seman tics of constrain ts are standard: i.e., an instan tiation u satisﬁes g ∼ g ′ if g [ u ] ∼ g ′ [ u ]. An instan tiation satisﬁes ψ ∧ ψ ′ if u satisﬁes b oth ψ and ψ ′ . The seman tics for other Bo olean connectives are deﬁned analogously . Moreo ver, w e will write g  = g ′ to denote the form ula g < g ′ ∨ g > g ′ . Chec king whether there exists an instantiation that satisﬁes a formula is equiv alent to chec king membership of the existential the ory of the r e als [ 21 ]. Suc h Parameter Synthesis for Marko v Mo dels 19 a c hec k can be automated using SMT-solvers capable of handling quantiﬁer-free non-linear arithmetic o ver the reals [93], such as [52, 112]. Statemen ts of the form f ∼ f ′ with f , f ′ ∈ Q ( V ) are not necessarily p olynomial constrain ts: how ever, we are not in terested in instantiations u with f [ u ] = ⊥ , and th us later (in Section 4.2.2) w e can transform suc h constraints into form ulae ov er p olynomial constrain ts. 3 F ormal Problem Statements This section formalises the three problem statements men tioned in the introduction: the veriﬁcation problem and tw o synthesis problems. W e start oﬀ by making precise what regions are and ho w to represent them. W e then deﬁne what it means for a region to satisfy a given speciﬁcation. This puts all in place to making the three problem statements precise. Finally , it surveys the v eriﬁcation approaches that are detailed later in the pap er. 3.1 Regions Instan tiated parametric mo dels are amenable to standard probabilistic mo del c hecking. How ever, sampling an instantiation is very restrictive—v erifying an instan tiated mo del gives results for a single point in the (uncountably large) parameter space. A more interesting problem is to determine whic h parts of the parameter space giv e rise to a model that complies with the sp eciﬁcation. Such sets of parameter v alues are, inspired b y their geometric interpretation, called r e gions . Regions are solution sets of conjunctions of constrain ts ov er the set V of parameters. Deﬁnition 11 (Region) A r e gion R o ver V is a set of instantiations of V (or dually a subset of R | V | ) for whic h there exists a set C ( R ) ⊆ C [ V ] of polynomial constrain ts suc h that for their conjunction Φ ( R ) = V c ∈ C ( R ) c w e hav e R = { u | Φ ( R )[ u ] } . W e call C ( R ) the r epr esentation of R . An y region which is a subset of a region R is called a subre gion of R . Example 16 Let the region R o ver V = { p, q } b e describ ed by C ( R ) = { p 2 + q 2 − 1 ≤ 0 , p + q − 1 ≤ 0 } . Th us, R = { u | ( p 2 + q 2 − 1)[ u ] ≤ 0 ∧ ( p + q − 1)[ u ] ≤ 0 } . The region R con tains the instan tiation u = ( 2 / 5 , 3 / 5 ) as ( 2 / 5 ) 2 + ( 3 / 5 ) 2 − 1 ≤ 0 and 2 / 5 + 3 / 5 − 1 ≤ 0. The instan tiation u ′ = ( 1 / 2 , 3 / 5 ) ∈ R as 1 / 2 + 3 / 5 − 1 > 0. Regions do not ha ve to describe a contiguous area of the parameter space; e.g., consider the region R ′ describ ed by {− p 2 + 1 < 0 } is R ′ = ( −∞ , − 1] ∪ [1 , + ∞ ). 20 Sebastian Junges et al. Regions are semi-algebraic sets [ 21 ] whic h yield the theoretical formalisation of notions suc h as distance, con vexit y , etc. It also ensures that regions are well - b eha ved: Informally , a region in the space R n is given by a ﬁnite num b er of connected semialgebraic sets ( c el ls 7 ), and (the boundaries of ) each cell can be describ ed by a ﬁnite set of p olynomials. The size ∥ R ∥ of a region R is given by the Leb esgue measure. All regions are Lebesgue measurable. A region is called w ell-deﬁned if all its instan tiations are well deﬁned. Deﬁnition 12 (W ell-deﬁned region) Region R is wel l deﬁne d for pSG G if for all u ∈ R , u is a w ell-deﬁned v aluation for G . 3.2 Angelic and demonic satisfaction relations As a next step to wards our formal problem statements, we hav e to deﬁne what it means for a region to satisfy a sp eciﬁcation. W e ﬁrst introduce tw o satisfac- tion relations—angelic and demonic—for parametric Marko v mo dels for a single instan tiation. W e then lift these tw o notions to regions. Deﬁnition 13 (Angelic and demonic satisfaction relations) F or pSG G , w ell- deﬁned instantiation u , and sp eciﬁcation φ , the satisfaction r elations | = a and | = d are deﬁned b y: G , u | = a φ iﬀ ∃ σ ∈ Str G . G [ u ] σ | = φ (angelic) G , u | = d φ iﬀ ∀ σ ∈ Str G . G [ u ] σ | = φ (demonic) . The angelic relation | = a refers to the existenc e of a strategy to fulﬁl the sp eciﬁcation φ , whereas the demonic counterpart | = d requires al l strategies to fulﬁl φ . Observ e that G , u | = a φ if and only if G , u | = d ¬ φ . Th us, demonic and angelic can b e considered to be dual. By | = ♡ w e denote the dual of | = ♣ , that is, if ♣ = a then ♡ = d and vice versa. F or pMCs, the relations | = a and | = d coincide and the subscripts a and d are omitted. Example 17 Consider the pMDP M in Figure 6(b) on page 18, instantiation u = ( 1 / 2 , 1 / 2 ) and φ = P > 4 / 5 ( ♢ { s 2 } ). W e hav e M , u | = a φ , as for strategy σ β = { s 0 7→ β } the state s 2 is reac hed with probability one; th us, M [ u ] σ β | = φ . Ho wev er, M , u | = d φ , as for strategy σ α = { s 0 7→ α } , w e ha ve (1 − p + p · q )[ u ] = 3 / 4  > 4 / 5 ; th us, M [ u ] σ α | = φ . By dualit y , M , u | = a ¬ φ . W e now lift these tw o satisfaction relations to regions. The aim is to consider sp eciﬁcations φ that hold for al l instantiations represented by a region R of a parametric mo del G . This is captured by the follo wing satisfaction relation. Deﬁnition 14 (Satisfaction relation for regions) F or pSG G , w ell-deﬁned region R , and sp eciﬁcation φ , the relation | = ♣ , ♣ ∈ { a, d } , is deﬁned as: G , R | = ♣ φ iﬀ G , u | = ♣ φ for all u ∈ R . 7 Connected here in tuitively refers to the fact that you can draw a path from tw o p oints in a cell that never leaves the cell. Parameter Synthesis for Marko v Mo dels 21 Before w e contin ue, we note the diﬀerence b etw een G , R | = ♣ φ and G , R | = ♣ ¬ φ : G , R | = ♣ ¬ φ implies G , u | = ♣ ¬ φ for al l u ∈ R, whereas in constrast, G , R | = ♣ φ implies G , u | = ♣ φ for some u ∈ R. Deﬁnition 15 (Accepting/rejecting/inconsisten t region) A well-deﬁned region R is ac c epting (for G , φ , ♣ ) if G , R | = ♣ φ . Region R is r eje cting (for G , φ , ♣ ) if G , R | = ♡ ¬ φ . Region R is inc onsistent if it is neither accepting nor rejecting. By the duality of | = a and | = d , a region is th us rejecting iﬀ ∀ u ∈ R . G , u | = ♣ φ . Note that this diﬀers from G , R | = ♣ φ . Example 18 Reconsider the pMDP in Figure 6(b) on page 18, with R = [ 2 / 5 , 1 / 2 ] × [ 2 / 5 , 1 / 2 ] and φ = P > 4 / 5 ( ♢ { s 2 } ). The corresponding solution functions are giv en in Example 14 on page 18. It follo ws that: – M , R | = a φ , as for strategy σ β = { s 0 7→ β } , w e hav e M σ β , u | = φ for all u ∈ R . – M , R | = d φ , as for strategy σ α = { s 0 7→ α } , M σ α , u | = φ for u = ( 1 / 2 , 1 / 2 ). – M , R | = a ¬ φ using strategy σ α . Regions can b e inconsistent w. r. t. a relation, and consistent w. r. t. its dual relation. The region (0 , 1) × (0 , 1) is inconsistent for M and | = d , as for b oth φ and ¬ φ , there is a strategy that is not accepting. F or | = a , there is a single strategy whic h accepts φ ; other strategies do not aﬀect the relation. As an example of an accepting region under the demonic relation, consider R ′ = [ 4 / 5 , 9 / 10 ] × [ 2 / 5 , 9 / 10 ]. W e ha ve M , R ′ | = d φ , as for b oth strategies, the induced probabilit y is alwa ys exceeding 4 / 5 . 3.3 F ormal problem statements W e are now in a position to formalise the tw o syn thesis problems and the veriﬁcation problem from the introduction, page 5. W e present the formal problem statements in the order of treatmen t in the rest of the pap er. The formal synthesis problem. Given pSG G , speciﬁcation φ , and well-deﬁned region R , the synthesis pr oblem is to partition R into R a and R r suc h that: G , R a | = ♣ φ and G , R r | = ♡ ¬ φ. This problem is the topic of Section 5 . R emark 6 The solution function for pMCs precisely describ es how (graph-preserving) instan tiations map to the relev ant measure. Therefore, comparing the solution function with the threshold divides the parameter space into an accepting region R a and a rejecting region R r and deﬁnes the exact result for the formal synthesis problem. Recall also Example 4 on page 6. 22 Sebastian Junges et al. The formal veriﬁcation problem. Given pSG G , sp eciﬁcation φ , and well- deﬁned region R , the veriﬁc ation pr oblem is to chec k whether: G , R | = ♣ φ ( R is accepting) or G , R | = ♡ ¬ φ ( R is rejecting) or G , R | = ♣ φ ∧ G , R | = ♡ ¬ φ ( R is inconsistent) where | = ♡ denotes the dual satisfaction relation of | = ♣ . This problem is the topic of Section 6 – 8 . The v eriﬁcation pro cedure allo ws us to utilise an appro ximate synthesis problem in whic h veriﬁcation pro cedures are used as a back end. The formal appro ximate synthesis problem. Given pSG G , sp eciﬁcation φ , p ercen tage c , and well-deﬁned region R , the appr oximate synthesis pr oblem is to partition R into regions R a , R o , and R r suc h that: G , R a | = ♣ φ and G , R r | = ♡ ¬ φ, where R a ⊎ R r co ver at least c % of the region R . This problem is the topic of Section 9 . Note that no requiremen ts are imp osed on the (unkno wn, op en) region R o . R emark 7 By deﬁnition, the angelic satisfaction relation for region R and pSG G is equiv alent to: G , R | = a φ if and only if ∀ u ∈ R. ∃ σ ∈ Str G . G σ , u | = φ. An alternativ e notion in parameter synthesis is the existence of a r obust strategy: ∃ σ ∈ Str G . ∀ u ∈ R . G σ , u | = φ. Note the sw apping of quantiﬁers compared to | = a . That is, G , R | = a φ considers p oten tially diﬀerent strategies for diﬀeren t parameter instan tiations u ∈ R . The notion of robust strategies leads to a series of quite orthogonal challenges. F or instance, the notion is not compositional, i.e., if robust strategies exist in R 1 and R 2 , then w e c annot conclude the existence of a robust strategy in R 1 ∪ R 2 . Moreo ver, memoryless strategies are not suﬃcien t, see [ 9 ]. Robust strategies are outside the scop e of this paper and are only shortly mentioned in Section 8. 3.4 A bird’s eye view on the veriﬁcation procedures In the later sections, we will present several tec hniques that decide the v eriﬁcation problem for pMCs and pMDPs. (Recall that sto c hastic games w ere only used to deﬁne the general setting.) The v eriﬁcation problem is used to analyse the regions of interest. The assump- tion that this region contains only well-deﬁned instan tiations is therefore natural. It can be c heck ed algorithmically as describ ed in Section 4.2 b elow. Many veriﬁcation pro cedures require that the region is graph preserving. A decomp osition result of w ell-deﬁned into graph-preserving regions is given in Section 4.3. Parameter Synthesis for Marko v Mo dels 23 Section 6 presents tw o veriﬁcation pro cedures. The ﬁrst one directly solves the non-linear equation system, see Example 6 on page 7, as an SMT query . The second pro cedure reformulates the SMT query using the solution function. While this reform ulation drastically reduces the num b er of v ariables in the query , it requires an eﬃcien t computation of the solution function, as describ ed in Section 5. Section 7 cov ers an appr oximate and more eﬃcient veriﬁcation pro cedure, called p ar ameter lifting , whic h is tailored to multi-linear functions and closed rectangular regions. Under these mild restrictions, the v eriﬁcation problem for pMCs (pMDPs) can be approximated using a sequence of standar d veriﬁc ation analyses on non- p ar ametric MDPs (SGs) of similar size, resp ectively . The key steps here are to relax the parameter dep endencies, and consider low er- and upp er-b ounds of parameters as w orst and b est cases. 4 Regions Section 3.1 already introduced regions. This section details sp eciﬁc region structures suc h as linear, rectangular and graph-preserving regions. It then presents pro cedures to chec k whether a region is graph preserving. Finally , w e describe how well-deﬁned but not graph-preserving regions can b e turned in to several regions that are graph preserving. 4.1 Regions with sp eciﬁc structure As deﬁned b efore, a region R is a (typically uncoun tably inﬁnite) set of parameter v aluations describ ed by a set C ( R ) of p olynomial constraints. Tw o classes of regions are particularly relev ant: linear and rectangular regions. Deﬁnition 16 (Linear region) A region with represen tation C ( R ) is line ar if for all g ∼ 0 ∈ C ( R ), the polynomial g is linear. Linear regions describ e con vex p olytopes. W e refer to the vertices (or angular p oin ts) of the p olytop e as the r e gion vertic es . Deﬁnition 17 (Rectangular region) A region R with representation C ( R ) = | V | [ i =1 { − p i + a i  1 i 0 , p i + b i  2 i 0 } with a i ≤ b i ∈ Q and  j i ∈ { <, ≤} for 0 < i ≤ | V | and j ∈ { 1 , 2 } is called re ctangular . A rectangular region is close d if all inequalities  j i in the constrain ts in C ( R ) are non-strict. Rectangular regions are hyper-rectangles and a sub class of linear regions. A close d rectangular region R can b e represented as R = × p ∈ V [ a p , b p ] with parameter in terv als [ a p , b p ] described by the b ounds a p and b p for all p ∈ V . F or a region R , w e refer to the b ounds of parameter p b y B R ( p ) = { a p , b p } and to the interval of parameter p b y I R ( p ) = [ a p , b p ]. W e may omit the subscript R , if it is clear from the con text. F or a rectangular region R , the size ∥ R ∥ equals Q p ∈ V ( b p − a p ). 24 Sebastian Junges et al. Regions represen t sets of instan tiations G [ u ] of a pSG G . The notion of graph- preserv ation from Deﬁnition 7 on page 15 lifts to regions in a straigh tforw ard manner: Deﬁnition 18 (Graph-preserving region) Region R is gr aph pr eserving for pSG G if for all u ∈ R , u is a graph-preserving v aluation for G . By this deﬁnition, all instan tiations from graph-preserving regions ha ve the same top ology as the parametric mo del, cf. Remark 8 b elow. In addition, all suc h instan tiations are well-deﬁned. Example 19 Let D b e the pMC in Figure 5(c) on page 13, R = [ 1 / 10 , 4 / 5 ] × [ 2 / 5 , 7 / 10 ] b e a (closed rectangular) region, and instan tiation u = ( 4 / 5 , 3 / 5 ) ∈ R . Figure 5(d) on page 13 depicts the instantiation D [ u ], an MC with the same top ology as D . As the topology is preserved for all p ossible instantiations D [ u ′ ] with u ′ ∈ R , the region R is graph preserving. The region R ′ = [0 , 1] × [0 , 1] is not graph preserving as, e.g., the instan tiation (0 , 0) ∈ R ′ results in an MC that has no transition from state s 1 to s 2 . R emark 8 Graph-preserving regions ha ve the nice prop erty that if ∃ u ∈ R , G , u | = ♣ P =1 ( ♢ T ) implies G , R | = ♣ P =1 ( ♢ T ) . This property can b e chec ked b y standard graph analysis [ 13 , Ch. 10]. It is thus straigh tforward to c heck G , R | = ♣ P =1 ( ♢ T ), an imp ortant precondition for comput- ing exp ected rewards. In the rest of this pap er when considering expected rew ards, it is assumed that within a region the probabilit y to reac h a target is one. The following tw o properties of regions are frequently (and often implicitly) used in this pap er. Lemma 1 (Characterisation for inconsisten t regions) F or any inc onsistent r e gion R it holds that R = R a ∪ R r for some ac c epting R a  = ∅ and r eje cting R r  = ∅ . Lemma 2 (Comp ositionalit y) Re gion R = R 1 ∪ R 2 is ac c epting (r eje cting) if and only if b oth R 1 and R 2 ar e ac c epting (r eje cting). The statemen ts follo w from the univ ersal quantiﬁcation o ver all instantiations in the deﬁnition of | = ♣ . 4.2 Checking whether a region is graph preserving The veriﬁcation problem for region R requires R to b e well-deﬁned. W e ﬁrst address the problem on how to chec k this condition. In fact, we present a pro cedure to c heck graph preserv ation which is sligh tly more general and useful later, see also Remark 8. T o show that region R is not graph preserving, a p oin t in R suﬃces that violates the conditions in Deﬁnition 7 on page 15. Using the representation of region R , the implication Φ ( R ) = ⇒ R graph preserving needs to be v alid since any violating assignment corresp onds to a non-graph- preserving instantiation inside R . T ec hnically , we consider satisﬁabilit y of the conjunction of: Parameter Synthesis for Marko v Mo dels 25 – the inequalities C ( R ) representing the candidate region, and – a disjunction of (in)equalities describing violating graph-preserving. This conjunction is satisﬁable if and only if the region is not graph preserving. 4.2.1 Gr aph pr eservation for p olynomial tr ansition functions Let us consider the ab ov e for pSGs with p olynomial transition functions. The setting for pSGs with rational functions is discussed at the end of this section. The follo wing constrain ts (1) – (4) , which we denote GP , capture the notion of graph preserv ation: ^ s,s ′ ∈ S,α ∈ A ct ( s ) P ( s,α,s ′ ) ≡ 0 0 ≤ P ( s, α, s ′ ) ≤ 1 (1) ∧ ^ s ∈ S,α ∈ A ct ( s ) X s ′ ∈ S P ( s, α, s ′ ) = 1 (2) ∧ ^ s ∈ S,α ∈ A ct ( s ) rew( s, α ) ≥ 0 (3) ∧ ^ s,s ′ ∈ S,α ∈ A ct ( s ) P ( s,α,s ′ ) ≡ 0 0 < P ( s, α, s ′ ) . (4) The constrain ts ensure that (1) all non-zero entries are ev aluated to a probabilit y , (2) transition probabilities are probability distributions, (3) rew ards are non-negative, and (4) non-zero entries remain non-zero. The constraints (1) – (3) suﬃce to ensure w ell-deﬁnedness. The constrains (1)–(4) can b e simpliﬁed to: ^ s,s ′ ∈ S,α ∈ A ct ( s ) P ( s,α,s ′ ) ≡ 0 P ( s, α, s ′ ) > 0 ∧ ^ s ∈ S,α ∈ A ct ( s ) X s ′ ∈ S P ( s, α, s ′ ) = 1 ∧ ^ s ∈ S,α ∈ A ct ( s ) rew( s, α ) ≥ 0 . Example 20 Recall the pMC from Figure 5(c) on page 13. GP = p > 0 ∧ 1 − p > 0 ∧ p +1 − p = 1 ∧ q > 0 ∧ 1 − q > 0 ∧ q +1 − q = 1 . This equation simpliﬁes to 0 < p < 1 ∧ 0 < q < 1. T o c heck whether the region R describ ed b y Φ ( R ) = 1 / 10 ≤ p ≤ 4 / 5 ∧ 2 / 5 ≤ q ≤ 7 / 10 is graph preserving, we chec k whether the conjunction Φ ( R ) ∧ ¬ GP is satisﬁable, with ¬ GP = p ≤ 0 ∨ p ≥ 1 ∨ q ≤ 0 ∨ q ≥ 1 . As the conjunction is not satisﬁable, the region R is graph preserving. Con trary , R ′ = [0 , 1] × [0 , 1] is not graph preserving as u = (0 , 0) satisﬁes the conjunction Φ ( R ′ ) ∧ ¬ GP . 26 Sebastian Junges et al. Satisﬁabilit y of GP , or equiv alently , deciding whether a region is graph preserving, is as hard as the existen tial theory of the reals [ 21 ], if no assumptions are made about the transition probabilit y and rew ard functions. This c hec king can b e automated using SMT-solv ers capable of handling quantiﬁer-free non-linear arithmetic o ver the reals [ 93 ]. The complexity drops to p olynomial time once both the region R and all transition probabilit y (and reward) functions are linear as linear programming has a p olynomial complexit y and the formula is then a disjunction ov er linear programs (with trivial optimisation functions). 4.2.2 Gr aph pr eservation for r ational tr ansition functions In case the transition probability and rew ard function of a pSG are not p olynomials, the left-hand side of the statements in (1) – (4) are not p olynomials, and the state- men ts w ould not be constraints. W e therefore perform the follo wing transformations on (1)–(4): – T ransforming equalities: g 1 g 2 = c becomes g 1 − c · g 2 = 0 ∧ g 2  = 0 with c ∈ Q . – T ransforming inequalities  ∈ { >, ≥} : g 1 g 2  c b ecomes g 2  = 0 ∧  ( g 2 > 0 ∧ g 1  c · g 2 ) ∨ ( g 2 < 0 ∧ g 1   c · g 2 )  with c ∈ Q , and   equals < for  > and ≤ for ≥ . – T ransforming <, ≤ is analogous. – T ransforming g  = g ′ (i.e., g < g ′ ∨ g > g ′ ) in volv es transforming b oth disjuncts. The result is a formula with polynomial constraints that correctly describ es graph preserv ation (or well-deﬁnedness). Example 21 Consider a state with outgoing transition probabilities q and p 1+ p . The graph preserv ation statements are (after some simpliﬁcation): q > 0 and p 1 + p > 0 and q + p 1 + p = 1 . T ransforming the second item as explained ab o ve yields: 1 + p  = 0 ∧  (1 + p > 0 ∧ p > 0) ∨ (1 + p < 0 ∧ p < 0)  while transforming the third item yields: (1 + p  = 0) ∧ q · (1+ p ) − 1 = 0 . Finally , we obtain the following form ula (after some further simpliﬁcations): q > 0 ∧ ( p > 0 ∨ p < − 1 ) ∧ q · (1 + p ) − 1 = 0 . Parameter Synthesis for Marko v Mo dels 27 p q (a) Subregions s 0 s 1 s 2 s 3 s 4 1 − p q 1 − q q 1 − q 1 1 (b) sub-pMC for p = 0 Fig. 7 Ensuring graph-preserv ation on subregions. 4.3 Reduction to graph-preserving regions In this section, w e sho w how we can partition a well-deﬁned region in to a set of graph-preserving regions. This is useful, e.g., as w e only deﬁne solution functions for graph-preserving regions. The decomp osition in this section allows to deﬁne solution functions on each of these partitions, see also Remark 5 on page 18. Before w e illustrate the decomp osition, we deﬁne sub-pSGs: Given tw o pSGs G = ( S , V , s I , A ct , P ) and G ′ = ( S ′ , V ′ , s I ′ , A ct ′ , P ′ ), G ′ is a sub-pSG of G if S ′ ⊆ S , V ′ ⊆ V , s ′ I = s I ∈ S ′ , A ct ′ ⊆ A ct , and P ′ ( s, α, s ′ ) ∈ {P ( s, α, s ′ ) , 0 } for all s, s ′ ∈ S ′ and α ∈ A ct ′ . Note that for a given state s ∈ S and action α ∈ A ct ( s ), the sub-pSG migh t not contain s or α migh t not b e enabled in s , but it is also p ossible that the sub-pSG omits some but not all successors of α in s . Example 22 Reconsider the pMC D from Figure 5(c) on page 13, and let R = [0 , 1] × [0 , 1], which is well-deﬁned but not graph preserving. Region R can b e partitioned in to 9 regions, see Figure 7(a) where each dot, line segmen t, and the inner region are subregions of R . All subregions are graph preserving on some sub-pMC of D . Consider, e.g., the line-region R ′ = { u ∈ R | p [ u ] = 0 } . The subregion R ′ is not graph preserving on pMC D , as the transition s 0 p − → s 1 v anishes when p = 0. How ever, R ′ is graph preserving on the sub-pMC D ′ in Figure 7(b), which is obtained from D by remo ving the transitions on the line-region p =0. Let us formalise the construction from this example. F or a given well-deﬁned region R , and pSG G , let Z R describ e the set of constrain ts: {P ( s, α, s ′ )=0 | s, s ′ ∈ S ∧ α ∈ A ct ( s ) ∧ P ( s, α, s ′ ) ≡ 0 ∧ ∃ u ∈ R. P ( s, α, s ′ )[ u ] = 0 } . F or X ⊆ Z R , the subregion R X ⊆ R is deﬁned as: Φ ( R X ) = Φ ( R ) ∧ ^ c ∈ X c ∧ ^ c ∈Z R \ X ¬ c. 28 Sebastian Junges et al. s s ′ t . . . p q 1 − p 1 − q 1 (a) pMC-fragmen t s s ′ t . . . . . . q 1 − p + p · q 1 − q 1 p · (1 − q ) (b) Reac hability probabilities s t t ′ p x y (c) Before loop elimination s t t ′ x / 1 − p y / 1 − p (d) After loop elimination Fig. 8 Essential ideas for state elimination It follo ws that X uniquely c haracterises which transition probabilities in G are set to zero. In fact, eac h instance in R X is graph preserving for the unique sub-pSG G ′ of G obtained from G b y removing all zero-transitions in R X . The pSG G ′ is w ell-deﬁned as R on G is w ell-deﬁned. By construction, it holds that G [ u ] = G ′ [ u ] for all instan tiations u ∈ R ′ . 5 Exact Synthesis by Computation of the Solution F unction This section discusses how to compute the solution function . The solution function for pMCs describ es the exact accepting and rejecting regions, as discussed in Section 3.3 8 . This section thus provides an algorithmic approach to the exact syn thesis problem. In Section 6, w e will also see that the solution function may be b eneﬁcial for the performance of SMT-based (region) veriﬁcation. The original approach to compute the solution function of pMCs is via state elimination [ 63 , 78 ], and is analogous to the computation of regular expressions from nondeterministic ﬁnite automata (NF As) [ 90 ]. It is suitable for a range of indeﬁnite-horizon properties. The core idea b ehind state elimination and the related approac hes presented here is based on tw o op erations: – A dding short-cuts: Consider the pMC-fragmen t in Figure 8(a). The reachabilit y probabilities from an y state to t are as in Figure 8(b), where we replaced the transition from s to s ′ b y shortcuts from s to t and all other successors of s ′ , b ypassing s ′ . By successiv e application of shortcuts, an y path from the initial state to the target state ev entually has length 1. – Elimination of self-lo ops: A prerequisite for in tro ducing a short-cut is that the b ypassed state is loop-free. Recall that the probability of staying forev er in a 8 for pMDPs, one ma y compute a solution function for ev ery strategy , but this has little practical relev ance Parameter Synthesis for Marko v Mo dels 29 non-absorbing state is zero, and justiﬁes elimination of self-lo ops by rescaling all other outgoing transitions, as depicted in the transition from Figure 8(c) to Figure 8(d). The remainder of this section is organised as follows: Section ?? recaps the original state elimination approac h in Section 5.1, albeit slightly rephrased. The algorithm is given for (indeﬁnite) reac hability probabilities, expected rewards, and b ounded reac hability probabilities. In the last part, we present alternativ e, equiv a- len t form ulations whic h sometimes allo w for superior p erformance. In particular, Section 5.2 clariﬁes the relation to solving a linear equation system ov er a ﬁeld of rational functions, and Section 5.3 discusses a v ariation of state elimination applicable to pMCs describ ed b y m ulti-terminal binary decision diagrams. 5.1 Algorithm based on state elimination Let T ⊆ S b e a set of target states and assume w. l. o. g. that all states in T are absorbing and that s I ∈ T . 5.1.1 R e achability pr ob abilities W e describ e the algorithm to compute reac hability probabilities based on state elimination in Algorithm 1 on the next page. In the follo wing, P is the tran- sition matrix. The function eliminate selﬂo op ( P , s ) rescales all outgoing prob- abilities of a non-absorbing state s b y eliminating its self-lo op. The function eliminate transition ( P , s 1 , s 2 ) adds a shortcut from s 1 to the successors of s 2 . Both op erations preserve reachabilit y to T . The function eliminate state ( P , s ) “b ypasses” a state s b y adding shortcuts from all its predecessors. More precisely , w e eliminate the incoming transitions of s , and after all incoming transitions are remo ved, the state s is unreachable. It is thereby eﬀectively remov ed from the mo del. After remo ving all non-absorbing, non-initial states S ? , the remaining model con tains only self-lo ops at the absorbing states and transitions emerging from the initial state. Eliminating the self-lo op on the initial state (b y rescaling) yields a pMC. In this pMC, after a single step, an absorbing state is reached. These absorbing states are either a target or a sink. The solution function is then the sum o ver all (one-step) transition probabilities to target states. Example 23 Consider again the pMC from Example 8 on page 12, also depicted in Figure 9(a) on page 31. Assume state s 2 is to b e eliminated. Applying the function eliminate state ( P , s 2 ), w e ﬁrst eliminate the transition s 1 → s 2 , whic h yields Figure 9(b), and subsequently eliminate the transition s 0 → s 2 (Figure 9(c)). State s 2 is now unreac hable, so w e can eliminate s 2 , reducing computational eﬀort when eliminating state s 1 . F or state s 1 , we ﬁrst eliminate the self-lo op (Figure 9(e)) and then eliminate the transition s 0 → s 1 . The ﬁnal result, after additionally removing the now unreachable s 1 , is depicted in Figure 9(f ). The result, i.e., the probability to even tually reac h s 3 from s 0 in the original mo del, can no w b e read from the single transition b et w een these tw o states. 30 Sebastian Junges et al. Algorithm 1 State elimination for pMCs reac hability (pMC D = ( S , V , s I , P ), T ⊆ S ) S ? : = { s ∈ S | s  = s I ∧ s ∈ ♢ T \ T } while S ?  = ∅ do select s ∈ S ? eliminate selﬂoop ( P , s ) eliminate state ( P , s ) S ? := S ? \ { s } eliminate selﬂoop ( P , s I ) // All S ? eliminated. Only direct transitions to target. return P t ∈ T P ( s I , t ) eliminate selﬂoop ( P , s ∈ S ) assert P ( s, s )  = 1 for eac h s 2 ∈ succ( s ) , s  = s 2 do P ( s, s 2 ) : = P ( s,s 2 ) 1 −P ( s,s ) P ( s, s ) : = 0 eliminate transition ( P , s 1 ∈ S, s ∈ S ) assert s 1 ∈ pred( s ), P ( s, s ) = 0 for eac h s 2 ∈ succ( s ) do P ( s 1 , s 2 ) : = P ( s 1 , s 2 ) + P ( s 1 , s ) · P ( s, s 2 ) P ( s 1 , s ) : = 0 eliminate state ( P , s ∈ S ) assert P ( s, s ) = 0 for eac h s 1 ∈ pred( s ) do eliminate transition ( P , s 1 , s ) As for computing of regular expressions from NF As, the order in which the states are eliminated is essential. Computing an optimal order with resp ect to minimalit y of the result, how ever, is already NP-hard for acyclic NF As, see [ 84 ]. F or state elimination on pMCs, the analysis is more intricate, as the cost of every op eration crucially dep ends on the size and the structure of the rational functions. W e brieﬂy discuss the implemented heuristics in Section 10.2.1. R emark 9 The elimination of self-lo ops yields a rational function. In order to keep these functions as small as p ossible, it is natural to eliminate common factors of the numerator and the denominator. Such a reduction, how ever, inv olves the computation of greatest common divisors (gcds). This op eration is expensive for m ultiv ariate p olynomials. In [ 91 ], data structures to a void their computation are in tro duced, in [ 17 ] a metho d is presented that mostly a voids introducing common factors. 5.1.2 Exp e cte d r ewar ds The state elimination approac h can also b e adapted to compute exp e cte d r ewar ds [ 78 ]. When eliminating a state s , in addition to adjusting the probabilities of the tran- sitions from all predecessors s 1 of s to all successors s 2 of s , it is also necessary to “summarise” the reward that would ha v e b een gained from s 1 to s 2 via s . The Parameter Synthesis for Marko v Mo dels 31 s 0 s 1 s 2 s 3 s 4 p 1 − p q 1 − q q 1 − q 1 1 (a) pMC s 0 s 1 s 2 s 3 s 4 p 1 − p q 2 1 − q q − q 2 q 1 − q 1 1 (b) Eliminating s 1 → s 2 s 0 s 1 s 2 s 3 s 4 p + q − pq (1 − p )(1 − q ) q 2 1 − q q − q 2 q 1 − q 1 1 (c) Eliminating s 0 → s 2 s 0 s 1 s 3 s 4 p + q − pq (1 − p )(1 − q ) q 2 1 − q q − q 2 1 1 (d) Remo ve unreachable state s 2 s 0 s 1 s 3 s 4 p + q − pq (1 − p )(1 − q ) 1 1+ q q 1+ q 1 1 (e) Eliminate loop on s 1 s 0 s 3 s 4 ( p + q − pq ) · 1 1+ q 1 −  ( p + q − pq ) · 1 1+ q  1 1 (f ) Eliminate s 1 Fig. 9 State elimination exempliﬁed presen tation in [ 78 ] describ es these op erations on so-called transition rewards. Observ e that for the analysis of exp ected rewards in MCs, we can alw a ys refor- m ulate transition rewards in terms of state rew ards. W e prepro cess pMCs to only ha ve rewards at the states: this adjustment simpliﬁes the necessary op erations considerably . The treatment of the exp ected rew ard computation is easiest from an adapted (and more performant) implemen tation of state elimination, as outlined in Algo- rithm 2 on the next page. Here, w e eliminate the probabilities to reac h a target state in exactly one step, and collect these probabilities in a vector x whic h we refer to as one-step-pr ob abilities . Then, w e pro ceed similar as b efore. How ever, the elimination of a transition from s 1 to s no w has tw o eﬀects: it up dates the probabilities within the non-target states as b efore, and (p oten tially) up dates the probability x ( s 1 ) to reac h the target within one step from s 1 (with the probabilit y that the target 32 Sebastian Junges et al. Algorithm 2 State elimination with one-step probabilities reac hability (pMC D = ( S , V , s I , P ), T ⊆ S ) S ? : = { s ∈ S | s ∈ ♢ T \ T } // x : S ? → [0 , 1] x ( s ) := P t ∈ T P ( s, t ) for eac h s ∈ S ? P ( s, t ) : = 0 for all s ∈ S, t ∈ T while S ?  = ∅ do eliminate state ( P , x, s ) for some s ∈ S ? S ? := S ? \ { s } // All S ? eliminated. One-step probability is reachability probability . return x ( s I ) eliminate transition ( P , x, s 1 ∈ S, s ∈ S ) // Algorithm mo diﬁes P assert s 1  = s , P ( s, s )  = 1 x ( s 1 ) : = x ( s 1 ) + P ( s 1 ,s ) · x ( s ) 1 −P ( s,s ) for eac h s 2 ∈ succ( s ) , s  = s 2 do P ( s 1 , s 2 ) : = P ( s 1 , s 2 ) + P ( s 1 ,s ) ·P ( s,s 2 ) 1 −P ( s,s ) P ( s 1 , s ) : = 0 eliminate state ( P , x, s ∈ S ) // Algorithm mo diﬁes P assert P ( s, s ) = 0 for eac h s 1 ∈ pred( s ) do eliminate transition ( P , x, s 1 , s ) w as reached via s in t wo steps). Up on termination of the outer loop, the vector x con tains the probabilities from all states to reach the target, that is, x ( s i ) = x s i . Finally , when considering rewards, the one-step-pr ob abilities contain initially the rew ards for the states. Eliminating a transition then mov es the (expected) rew ard to the predecessors b y the same sequence of arithmetic op erations. 5.1.3 Bounde d r e achability As discussed in Remark 4 on page 17, b ounded reachabilit y can typically be considered b y an unfolding of the Marko v mo del and considering an un b ounded reac hability prop erty on that (acyclic) unfolding. In com bination with state elimi- nation, that yields the creation of many states that are eliminated afterwards, and do es not take in to account an y problem-sp eciﬁc prop erties. Rather, and analogous to the parameter-free case [ 13 ], it is b etter to do the adequate matrix-vector mul- tiplication (# n umber of steps often). The matrix originates from the transition matrix, the vector (after i m ultiplications) enco des the probability to reac h a state within i steps. 5.2 Algorithm based on solving the linear equation system The following set of equations is a straigh tforward adaption of the Bellman linear equation system for MCs found in, e.g., [ 13 , 123 ] to pMCs. F or each state s , a v ariable x s is used to express the probability Pr s ( ♢ T ) to reach a state in T from the state s . Recall that w e ov erloaded ♢ T to also denote the set of states from which Parameter Synthesis for Marko v Mo dels 33 T is reachable (with positive probabilit y). Analogously , we use ¬ ♢ T to denote the set of states from whic h T is not reachable, i. e., ¬ ♢ T = S \ ♢ T . W e ha ve: x s = 0 ∀ s ∈ ¬ ♢ T (5) x s = 1 ∀ s ∈ T (6) x s = X s ′ ∈ S P ( s, s ′ ) · x s ′ ∀ s ∈ ♢ T \ T . (7) This system of equations has a unique solution for every well-deﬁned parameter instan tiation. In particular, the set of states satisfying ¬ ♢ T is the same for all well- deﬁned graph-preserving parameter instantiations, as instan tiations that maintain the graph of the pMC do not aﬀect the reac habilit y of states in T . F or pMCs, the co eﬃcients are no longer from the ﬁeld of the real n um b ers, but rather from the ﬁeld of rational functions. Example 24 Consider the equations for the pMC from Figure 9(a) on page 31. x 0 = p · x 1 + (1 − p ) · x 2 x 1 = q · x 2 + (1 − q ) · x 3 x 2 = q · x 1 + (1 − q ) · x 4 x 3 = 1 x 4 = 0 . Bringing the system in normal form yields: x 0 − p · x 1 − (1 − p ) · x 2 = 0 x 1 − q · x 2 − (1 − q ) · x 3 = 0 − q · x 1 + x 2 − (1 − q ) · x 4 = 0 x 3 = 1 x 4 = 0 . Adding q times the second equation to the third equation (concerning state s 2 ) brings the left-hand side matrix in upper triangular form: x 0 − p · x 1 − (1 − p ) · x 2 = 0 x 1 − q · x 2 − (1 − q ) · x 3 = 0 (1 − q 2 ) · x 2 − q (1 − q ) · x 3 − (1 − q ) · x 4 = 0 x 3 = 1 x 4 = 0 . The equation system yields the same result as the elimination of the transition from s 2 to s 1 (notice the symmetry b et w een s 1 and s 2 ). The example illustrates that there is no elemen tary adv antage in doing state elimination o ver resorting to solving the linear equation sytem by (some v arian t of ) Gaussian elimination. If we are only interested in the probability from the initial state, we do not need to solve the full equation system. The state-elimination algorithm, in whic h w e can remov e unreac hable states, optimises for this observ ation, in contrast to (standard) linear equation solving. As in state elimination, the elimination order of the ro ws has a signiﬁcant inﬂuence. 34 Sebastian Junges et al. 5.3 Algorithm based on set-based transition elimination T o succinctly represent large state spaces, Marko v chains are often represen ted by m ulti-terminal binary decision diagrams (or v ariants thereof ) [ 14 ]. Such a symbolic r epr esentation handles sets of states instead of single states (and thus also sets of transitions), and thereby exploits symmetries and similarities in the underlying graph of a model. T o supp ort eﬃcient elimination, w e describ e ho w to eliminate sets of transitions at once. The metho d is similar to the Flo yd-W arshall algorithm for all-pair shortest paths [ 51 ]. The transition matrix contains one-step probabilities for ev ery pair of source and target states. Starting with a self-loop-free pMC (obtained b y eliminating all self-lo ops from the original pMC), we iterate tw o op erations un til conv ergence. By doing a matrix-matrix multiplication, w e eﬀectively eliminate all transitions emanating from all non-absorbing states simultane ously . As this step ma y reintroduce self-lo ops, we eliminate them in a second step. As b efore, ev entually only direct transitions to absorbing states remain, whic h eﬀectively yield the unbounded reachabilit y probabilities. The corresp onding pseudo-co de is given in Algorithm 3 on the follo wing page. The approac h of this algorithm can conv eniently b e explained in the equation system represen tation. Let us therefore conduct one step of the algorithm as an example, where we use the observ ation that the matrix-matrix multiplication corresp onds to replacing the v ariables x s b y their deﬁning equations in all other equations. Example 25 Reconsider the equations from Example 24 on the previous page: x 0 = p · x 1 + (1 − p ) · x 2 x 1 = q · x 2 + (1 − q ) · x 3 x 2 = q · x 1 + (1 − q ) · x 4 x 3 = 1 x 4 = 0 . Using the equations for x 0 , x 1 , x 2 to replace their o ccurrences in all other equations yields: x 0 = p · ( q · x 2 + (1 − q ) · x 3 ) + (1 − p )( q · x 1 + (1 − q ) · x 4 ) x 1 = q · ( q · x 1 + (1 − q ) · x 4 ) + (1 − q ) · x 3 x 2 = q · ( q · x 2 + (1 − q ) · x 3 ) + (1 − q ) · x 4 x 3 = 1 x 4 = 0 Parameter Synthesis for Marko v Mo dels 35 s 0 s 1 s 2 s 3 s 4 (1 − p ) · q p · q 1 1+ q q 1+ q q 1+ q 1 1+ q 1 1 p · (1 − q ) (1 − p ) · (1 − q ) (a) After ﬁrst iteration s 0 s 1 s 2 s 3 s 4 1 1+ q q 1+ q q 1+ q 1 1+ q 1 1 ( p + q − pq ) · 1 1+ q 1 − ( p + q − pq ) · 1 1+ q (b) After second iteration Fig. 10 One step of set-based transition elimination exempliﬁed Algorithm 3 Set-based transition elimination for pMCs reac hability (pMC D = ( S , V , s I , P ), T ⊆ S ) S ? : = { s ∈ S | s  = s I ∧ s ∈ ♢ T \ T } for eac h s ∈ S ? do // can b e done in parallel for all s eliminate selﬂoop ( P , s ) while ∃ s, s ′ ∈ S ? . P ( s, s ′ )  = 0 do for eac h s ∈ S ? , s ′ ∈ S do // can b e done in parallel for all s, s ′ P ′ ( s, s ′ ) := P s ′′ P ( s, s ′′ ) · P ( s ′′ , s ′ ) for eac h s ∈ S ? do // can b e done in parallel for all s eliminate selﬂo op ( P ′ , s ) P := P ′ // All S ? eliminated. Only direct paths to target. return P t ∈ T P ( s I , t ) whic h simpliﬁes to x 0 = (1 − p ) · q · x 1 + p · q · x 2 + p · (1 − q ) · x 3 + (1 − p )(1 − q ) · x 4 x 1 = 1 1 + q · x 3 + q 1 + q · x 4 x 2 = q 1 + q · x 3 + 1 1 + q · x 4 x 3 = 1 x 4 = 0 . W e depict the pMC which corresp onds to this equation system in Figure 10(a). Again, notice the similarity to state elimination. F or completeness, the result after another iteration is giv en in Figure 10(b). The correctness follows from the follo wing argumen t: After every iteration, the equations describ e a pMC ov er the same state space as b efore. As all absorbing states ha ve deﬁning equations x i ∈ { 0 , 1 } , the equation system is known to hav e a unique solution [ 13 ]. Moreo ver, as the equation system in iteration i implies the equation system in iteration i + 1, they preserve the same (unique) solution. 36 Sebastian Junges et al. s 0 s 1 s 2 s 3 p q 1 − p 1 − q 1 1 (a) pMC s 0 s 1 s 2 s 3 α β 1 p q 1 − p 1 − q 1 1 (b) pMDP Fig. 11 T o y-examples (rep eated from Figure 6 on page 18) 6 SMT-based region veriﬁcation In this section, we discuss a complete procedure to verify regions by enco ding them as queries for an SMT solver, or more precisely , in the existential theory of the reals (the QF NRA theory in the SMT literature). W e ﬁrst in tro duce the constraints for verifying regions on pMCs in Section 6.1. The constrain ts are either based on the equation system enco ding from Section 5.2 or use the solution function, whic h yields an equation system with less v ariables at the cost of precomputing the solution function. In Section 6.2, we then introduce the encodings for region v eriﬁcation on pMDPs under angelic and demonic strategies. Throughout the section, we fo cus on unbounded reachabilit y , that is, w e assume φ = P ≤ λ ( ♢ T ). As exp ected rewards can b e describ ed by a similar equation system, lifting the concepts is straigh tforward. W e assume a graph-preserving region R : Assuming that R is graph preserving eases the enco dings signiﬁcantly , but is not strictly necessary: In [94, Ch. 4], w e provide enco dings for well-deﬁned regions R . 6.1 Satisﬁability chec king for pMC region c hecking Recall from Section 5.2 the equation system for pMCs, exempliﬁed b y the following running example. Example 26 Reconsider the pMC D from Figure 6(a) on page 18, repeated in Figure 11(a) for conv enience. The concrete equation system of (5) – (7) on page 33 for reac hing T = { s 2 } , using x i to denote x s i , is giv en by: x 0 = p · x 1 + (1 − p ) · x 2 x 1 = q · x 2 + (1 − q ) · x 3 x 2 = 1 x 3 = 0 . The conjunction of the equation system for the pMC, (5) – (7) on page 33, is an implicitly existen tial quantiﬁed formula to whic h we refer b y Φ ( D )—consider the remark b elo w. By construction, this form ula is satisﬁable. R emark 10 If transitions in the pMC are not p olynomial but rational functions, the equations are not p olynomial constrain ts, hence their conjunction is not a form ula (Section 2.5). Instead, eac h x = P P ( s, s ′ ) has to be transformed b y the rules in Section 4.2.2: then, their conjunction is a formula. This transformation Parameter Synthesis for Marko v Mo dels 37 can alwa ys b e applied, in particular, in the equalities we are nev er interested in the ev aluation of instantiations u ∈ R with P ( s, s ′ )[ u ] = ⊥ : Recall that w e are in terested in analysing this equation system on a wel l-deﬁne d parameter region R : Therefore, for an y u ∈ R , P ( s, s ′ )[ u ]  = ⊥ for eac h s, s ′ ∈ S . Th us, when Φ ( D ) is used in conjunction with Φ ( R ), we do not need to consider this sp ecial case. W e consider the conjunction of the equation system, a property and a region. Concretely , let us ﬁrst consider the conjunction of: – the equation system Φ ( D ), – a comparison of the initial state s I with the threshold λ , and – a formula Φ ( R ) describing the parameter region R . Satisﬁabilit y of this conjunction means that—for some parameter instantiation within the region R —the reac hability probability from the initial state s I satisﬁes the b ound. Unlik e Φ ( D ), this conjunction ma y b e unsatisﬁable. Example 27 W e contin ue with Example 26 on the previous page. Let φ = P ≤ 0 . 4 ( ♢ { s 2 } ) and R = { ( p, q ) ∈ [0 . 4 , 0 . 6] × [0 . 2 , 0 . 5] } . W e hav e Φ ( R ) = 0 . 4 ≤ p ∧ p ≤ 0 . 6 ∧ 0 . 2 ≤ q ∧ q ≤ 0 . 5. W e obtain the follo wing conjunction: Φ ( D ) ∧ x 0 ≤ 0 . 4 ∧ Φ ( R ) (8) where Φ ( D ) is the conjunction of the equation system, i.e.: Φ ( D ) =  x 0 = p · x 1 + (1 − p ) · x 2 ∧ x 1 = q · x 2 + (1 − q ) · x 3 ∧ x 2 = 1 ∧ x 3 = 0  . F ormula (8) is unsatisﬁable, th us, no instance of p and q within the region R induces a reac hability probability of at most 2 / 5 . T ow ards region v eriﬁcation, consider that the satisfaction relations | = a 9 as deﬁned in Deﬁnition 13 on page 20, w e ha ve to certify that all parameter v alues within a region yield a reachabilit y probability that satisﬁes the threshold. Thus, w e ha ve to quan tify ov er all instantiations u , (roughly) leading to a formula of the form ∀ u . . . | = φ . By negating this statemen t, w e obtain the pro of obligation ¬∃ u . . . | = ¬ φ : no parameter v alue within the region R satisﬁes the negated comparison with the initial state. This in tuition leads to the following conjunction of: – the equation system Φ ( D ), – a comparison of the initial state with the threshold, by inv erting the giv en threshold-relation, and – a formula Φ ( R ) describing the parameter region. This conjunction is formalised in the follo wing deﬁnition. Deﬁnition 19 (Equation system form ula) Let D b e a pMC, φ = P ∼ λ ( ♢ T ), and R a region. The e quation system formula is given by: Φ ( D ) ∧ x s I ∼ λ ∧ Φ ( R ) . Theorem 1 The e quation system formula is unsatisﬁable iﬀ D , R | = φ . 9 Recall that | = d coincides with | = a for pMCs. 38 Sebastian Junges et al. Otherwise, a satisfying solution is a c ounter example . Example 28 W e contin ue Example 27 on the previous page. W e inv ert the relation x 0 ≤ 0 . 4 and obtain: Φ ( D ) ∧ x 0 > 0 . 4 ∧ Φ ( R ) . By SMT-c hecking, we determine that the form ula is satisﬁable, e.g., with p = 0 . 5 and q = 0 . 3. Thus, D , R | = φ . If we consider instead the region R ′ = { ( p, q ) ∈ [0 . 8 , 0 . 9] × [0 . 1 , 0 . 2] } with Φ ( R ′ ) = 0 . 8 ≤ p ∧ p ≤ 0 . 9 ∧ 0 . 1 ≤ q ∧ q ≤ 0 . 2, w e obtain: Φ ( D ) ∧ x 0 > 0 . 4 ∧ Φ ( R ′ ) whic h is unsatisﬁable. Hence, no point in R ′ induces a probability larger than 2 / 5 and, equiv alently , all p oints in R ′ induce a probability of at most 2 / 5 . Thus, D , R ′ | = φ . W e observe that the n umber of v ariables in this enco ding is | S | + | V | . In particular, we are often interested in systems with at least thousands of states. The n umber of v ariables is therefore often to o large for SMT-solvers dealing with non-linear real arithmetic. Ho w ever, many of the v ariables are auxiliary v ariables that encode the probability to reac h target states from each individual state. W e can get rid of these v ariables by replacing the full equation system by the solution function (Deﬁnition 10 on page 17). Deﬁnition 20 (Solution function formula) Let D b e a pMC, φ = P ∼ λ ( ♢ T ), and R a region. The solution function formula 10 is giv en by: f r D ,T ∼ λ ∧ Φ ( R ) . Corollary 1 The solution function formula is unsatisﬁable iﬀ D , R | = φ . Example 29 W e consider the same scenario as in Example 27 on the previous page. The solution function is giv en in Example 13 on page 17. The solution function form ula is: 1 − p + p · q > 0 . 4 ∧ Φ ( R ) . By construction, the equation system formula and the solution function form ula for pMC D and reachabilit y prop ert y φ are equisatisﬁable. 6.2 Existentially quantiﬁed form ula for parametric MDPs W e can also utilise an SMT solver to tac kle the veriﬁcation problem on pMDPs. F or parametric MDPs, w e distinguish b etw een the angelic and the demonic case, cf. Deﬁnition 14 on page 20. W e use the fact that optimal strategies for unbounded reac hability ob jectives are memoryless and deterministic [123]. 10 Remark 10 on page 36 applies also here. Parameter Synthesis for Marko v Mo dels 39 6.2.1 Demonic str ate gies The satisfaction relation | = d is deﬁned b y tw o universal quan tiﬁers, ∀ u ∀ σ . . . | = φ . W e therefore try to refute satisﬁability of ∃ u ∃ σ . . . | = ¬ φ . Put in a game-theoretical sense, the same play er can choose b oth the parameter instantiation u and the strategy σ to resolve the non-determinism. W e generalise the set of linear equations from the pMC to an encoding for pMDPs, where w e deﬁne a disjunction o ver all p ossible nondeterministic c hoices: x s = 0 ∀ s ∈ ¬ ♢ T (9) x s = 1 ∀ s ∈ T (10) _ α ∈ A ct ( s )  x s = X s ′ ∈ S P ( s, α, s ′ ) · x s ′  ∀ s ∈ ♢ T \ T . (11) W e denote the conjunction of (9) – (11) as Φ d ( M ) for pMDP M 11 . Instead of a single equation for the probability to reach the target from state s , we get one equation for each action. The solv er can now freely choose whic h (memoryless deterministic) strategy it uses to refute the property . Deﬁnition 21 (Demonic equation system form ula) Let M b e a pMDP , φ = P ≤ λ ( ♢ T ), and R a region. The demonic e quation system formula is given by: Φ d ( M ) ∧ x s I > λ ∧ Φ ( R ) . Theorem 2 The demonic e quation system formula is unsatisﬁable iﬀ M , R | = d φ . Example 30 Let M b e the pMDP from Figure 11(b) on page 36. Let R, φ b e as in Example 27 on page 37. The demonic equation system form ula is Φ d ( M ) ∧ x 0 > 0 . 4 ∧ Φ ( R ) with Φ ( R ) as b efore, and Φ d ( M ) =   x 0 = p · x 1 + (1 − p ) · x 2 ∨ x 0 = x 2  ∧ x 1 = q · x 2 + (1 − q ) · x 3 ∧ x 2 = 1 ∧ x 3 = 0  . Similarly , when using the (p otentially exp onential) set of solution functions, w e let the solv er choose: Deﬁnition 22 (Demonic solution function formula) Let M b e a pMDP , φ = P ∼ λ ( ♢ T ), and R a region. The demonic solution function formula is given b y: _ σ ∈ Str M f r M σ ,T ∼ λ ∧ Φ ( R ) . Corollary 2 The demonic solution function formula is unsatisﬁable iﬀ M , R | = d φ . 11 Recall again Remark 10 on page 36. 40 Sebastian Junges et al. As the set of solution functions can b e exp onential, the demonic solution function form ula can grow exp onentially . Example 31 The demonic solution function form ula for M , φ, R as in Example 30 on the previous page, is giv en by:  1 > 0 . 4 ∨ 1 − p + p · q > 0 . 4  ∧ Φ ( R ) . 6.2.2 Angelic str ate gies The satisfaction relation | = a has tw o diﬀeren t quantiﬁers, ∀ u ∃ σ . . . | = φ . Again, w e equiv alently try to refute the satisﬁability of ∃ u ∀ σ . . . | = ¬ φ . The quantiﬁer alternation can be circumv ented b y lifting the linear programming (LP) form ulation for MDPs [ 123 ], where for each nondeterministic c hoice an upp er b ound on the probabilit y v ariables x s is obtained: x s = 0 ∀ s ∈ ¬ ♢ T (12) x s = 1 ∀ s ∈ T (13) ^ α ∈ A ct ( s )  x s ≤ X s ′ ∈ S P ( s, α, s ′ ) · x s ′  ∀ s ∈ ♢ T \ T . (14) In tuitively , the conjunction in constraint (14) eliminates the freedom of c ho osing an y strategy from the solv er and forces it to use the strategy that minimises the reac hability probability . This means that the constrain t system is only satisﬁable if al l strategies violate the probability b ound. W e denote the conjunction of (12) – (14) as Φ a ( M ). Notice that, as for parameter-free MDPs, the optimisation ob jectiv e of the LP formulation can be substituted b y a constraint on probabilit y in the initial state. Deﬁnition 23 (Angelic equation system formula) Let M b e a pMDP , φ = P ≤ λ ( ♢ T ), and R a region. The angelic e quation system formula is given by: Φ a ( M ) ∧ x s I > λ ∧ Φ ( R ) . Theorem 3 The angelic e quation system formula is unsatisﬁable iﬀ M , R | = a φ . Example 32 Let M , φ, R as in Example 30 on the previous page. The angelic equation system form ula is given by Φ a ( M ) ∧ x 0 > 0 . 4 ∧ Φ ( R ) with Φ a ( M ) =   x 0 ≤ p · x 1 + (1 − p ) · x 2 ∧ x 0 ≤ x 2  ∧ x 1 ≤ q · x 2 + (1 − q ) · x 3 ∧ x 2 = 1 ∧ x 3 = 0  . When using the set of solution functions, all strategies hav e to be considered. Again, for most pMDPs, this set is prohibitiv ely large. Parameter Synthesis for Marko v Mo dels 41 s 0 s 1 s 2 s 3 s 4 p 1 − p q 1 − q q 1 − q 1 1 (a) D s 0 s 1 s 2 s 3 s 4 1 / 10 9 / 10 4 / 5 1 / 5 2 / 5 3 / 5 7 / 10 3 / 10 2 / 5 3 / 5 7 / 10 3 / 10 1 1 (b) sub R ( D ) s 0 s 1 s 2 s 3 s 4 p s 0 1 − p s 0 q s 1 1 − q s 1 q s 2 1 − q s 2 1 1 (c) rel ( D ) Fig. 12 A pMC D and its substitution sub R ( D ) and its relaxation rel ( D ). Deﬁnition 24 (Angelic solution function formula) Let M b e a pMDP , φ = P ≤ λ ( ♢ T ), and R a region. The angelic solution function formula is given b y: ^ σ ∈ Str M f r M σ ,T > λ ∧ Φ ( R ) . Corollary 3 The angelic solution function formula is unsatisﬁable iﬀ M , R | = a φ . Example 33 The angelic solution function formula for M , φ, R as in Example 30 on page 39 is giv en by:  1 > 0 . 4 ∧ 1 − p + p · q > 0 . 4  ∧ Φ ( R ) . 7 Mo del-chec king-based Region V eriﬁcation of P arametric MCs This section discusses an abstraction (and reﬁnement) pro cedure for region v eri- ﬁcation of pMCs. In tuitively , in order to b ound the probability in a region from ab o ve, we b ound the v alue induced by an y instan tation from ab ov e. W e aim to do this by ﬁnding an instantiation that maximises the reachabilit y probability in the region. This problem is particularly hard, as there are dep endencies betw een the diﬀeren t parameters: Example 34 Consider the pMC D in Figure 12(a)—rep eating Figure 5(c) on page 13— and region R = [ 1 / 10 , 4 / 5 ] × [ 2 / 5 , 7 / 10 ]. W e again aim to reac h s 3 . W e mak e tw o observ ations: s 4 is the only state from whic h we cannot reac h s 3 , furthermore, s 4 is only reachable via s 2 . Hence, it is b est to av oid s 2 . F rom state s 0 , it is thus b eneﬁcial if the transition probability to s 2 is as small as p ossible. Equiv alently , it is b eneﬁcial if p is as large as p ossible, as this minimises the probability of reaching s 2 and as p do es not o ccur elsewhere. Now w e consider state s 1 : As we w ant to reac h s 3 , the v alue of q should be preferably low. How ev er, q o ccurs also at transitions lea ving s 2 . F rom s 2 , q should be assigned a high v alue as we wan t to a void s 4 . In particular, the optimal v alue for q dep ends on the probabilit y that we ever visit s 2 , whic h is directly inﬂuenced by the v alue of p . 42 Sebastian Junges et al. In a n utshell, the abstraction w e prop ose in this section ignores the dependencies b et ween the same o ccurence of a parameter. Con venien tly , the abstraction trans- forms a pMC into an (parameter-free!) MDP whose minimal (maximal) reachabilit y probabilit y under-appro ximates (ov er-approximates) the reachabilit y probability of the pMC. This result is formalised in Theorem 5, below. Example 35 Consider the pMC in Figure 12(a) on the previous page and a region R = [ 1 / 10 , 4 / 5 ] × [ 2 / 5 , 7 / 10 ]. The metho d creates the MDP in Figure 12(b), where diﬀeren t types of arro ws reﬂect diﬀeren t actions. The MDP is created by adding in eac h state tw o actions: One reﬂecting the low er b ound of the parameter range, one reﬂecting the upp er b ound. Mo del chec king on this MDP yields a maximal probabilit y of 47 / 60 . F rom this result, we infer that max u ∈ R Pr D [ u ] ( ♢ T ) ≤ 47 / 60 . The essence of this construction is to consider parameter v alues as a lo cal, discrete choice that we can capture with nondeterminism. T o supp ort the discreti- sation, we must ensure that the optimal v alues are taken at the b ounds of the region. While this is not true in general due to the nonlinearity of the solution function, creating a suitable ov er-approximation, called the relaxation, enforces this prop ert y , as we show in Theorem 4, also below. In the remainder of this section, w e ﬁrst clarify helpful assumptions on the t yp e of pMCs w e supp ort in Section 7.1. W e then construct so-called relaxed pMCs in Section 7.2. In Section 7.3, w e translate relaxed pMCs to parameter-free MDPs to allo w oﬀ-the-shelf MDP analysis for region veriﬁcation of pMCs. 7.1 Preliminaries W e formalise the p ersp ective that underpins our approac h to region veriﬁcation and in tro duce some assumptions. 7.1.1 A Persp e ctive for R e gion V eriﬁc ation The probability Pr D ( ♢ T ) can b e expressed as a rational function f = g 1 / g 2 with p olynomials g 1 , g 2 due to Deﬁnition 10 on page 17. Recall that we assume region R to b e graph preserving. Therefore, g 2 [ u ]  = 0 for all u ∈ R and f is contin uous on any closed region R . Hence, there is an instantiation u ∈ R that induces the maximal (or minimal) reac hability probability: sup u ∈ R Pr D [ u ] ( ♢ T ) = max u ∈ R Pr D [ u ] ( ♢ T ) and inf u ∈ R Pr D [ u ] ( ♢ T ) = min u ∈ R Pr D [ u ] ( ♢ T ) . T o infer that R is accepting (i.e. all instantiations u ∈ R induce probabilities at most λ ), it suﬃces to show that the maximal reachabilit y probabilit y ov er all instan tiations is at most λ : D , R | = P ≤ λ ( ♢ T ) ⇐ ⇒  max u ∈ R Pr D [ u ] ( ♢ T )  ≤ λ, and D , R | = ¬ P ≤ λ ( ♢ T ) ⇐ ⇒  min u ∈ R Pr D [ u ] ( ♢ T )  > λ. One w ay to determine the maximum reac hability probabilit y is to ﬁrst determine whic h u ∈ R induces the maximum, and then compute the probability on the instan tiated mo del D [ u ]. While we only discuss upp er-bounded speciﬁcations here, the results can b e analogously described for low er-b ounded sp eciﬁcations. Parameter Synthesis for Marko v Mo dels 43 Example 36 Consider D depicted in Figure 11(a) on page 36, φ = P ≤ 9 / 10 ( ♢ { s 2 } ), and R ′ = { ( p, q ) ∈ [ 2 / 5 , 3 / 5 ] × [ 1 / 5 , 1 / 2 ] } as in Example 27 on page 37. The maxim um is obtained at u = ( 2 / 5 , 1 / 2 ) (via some oracle). W e hav e D [ u ] | = P ≤ 9 / 10 ( ♢ { s 2 } ), and th us, D , R ′ | = P ≤ 9 / 10 ( ♢ { s 2 } ). Ho wev er, constructing an oracle that determines the u that induces the maximum is diﬃcult in general. W e fo cus on the essential idea an therefore make the following assumptions throughout the rest of this section: Assumption 1 – We r estrict the (gr aph-pr eserving) r e gion R to b e (i) r e ctangular, and (ii) close d. This r estriction makes the b ounds of the p ar ameters indep endent of other p ar ameter instantiations, and ensur es that the maximum over the r e gion exists. – We r estrict the pMC D to b e lo cally monotone –explaine d in Se ction 7.1.2– to exclude diﬃculties fr om analysing single tr ansitions. The ﬁrst assumption can b e a nuisance. In particular, it is not alwa ys clear ho w to create an adequate closed region from an op en region. The second assumption is v ery mild and can b e accomodated for using adequate prepro cessing [ 94 , Section 5.1] that in tro duced additional states. 7.1.2 L o c al ly Monotone pMCs Recall that the solution function is nonlinear. W e aim to appro ximate this u and therefore w an t to exploit the structure of the pMC. Therefore, w e w ant to make an assumption on the transition relation. Example 37 Consider a three-state pMC where the probabilit y from initial state s I to target state t is a non-linear, non-monotone transition function, as, e.g., the transition probability from s 0 to s 3 of the pMC in Figure 9(f ) on page 31. Finding the maximum requires an analysis of the deriv ative of the solution function, and is (appro ximately) as hard as the exact veriﬁcation problem. Instead, w e assume monotonic transition probabilities, and consider a slightly restricted class of pMCs. Deﬁnition 25 (Lo cally monotone pMCs) A pMC D = ( S , V , s I , P ) is lo c al ly monotone iﬀ for all s ∈ S there is a multilinear polynomial g s ∈ Q [ V ] satisfying P ( s, s ′ ) ∈ { f / g s | f ∈ Q [ V ] is multilinear } for all s ′ ∈ S . Lo cally monotone pMCs include most pMCs from the literature [ 124 ] 12 . Examples of the egligible transition probabilities are p, pq , 1 / p and their complements formed b y 1 − p etc. Thanks to monotonicit y , for a lo cally monotone pMC D = ( S , V , s I , P ), and a closed rectangular region R we hav e that for all s, s ′ ∈ S : max u ∈ R P ( s, s ′ ) = max u ∈ B ( V ) P ( s, s ′ ) 12 It ev en includes the embedded pMCs of parametric con tinuous-time Markov chains with multilinear exit rates. 44 Sebastian Junges et al. where B ( V ) = { u | ∀ p ∈ V .u ( p ) ∈ B R ( p ) } , i.e., all maxima of the individual transition probabilities are attained at the bounds of the region. How ever, the restriction to lo cal monotonicity do es not immediately o v ercome the challenge of constructing an oracle. The resulting solution function may still be highly nonlinear. In particular, Example 34 on page 41 uses a lo cally monotone pMC and a closed rectangular region. Ho wev er, as the example indicates, trade-oﬀs in locally monotone pMCs o ccur due to dependencies where parameters o ccur at multiple states. 7.2 Relaxation The idea of our approach, inspired b y [ 30 ], is to drop the aforementioned depen- dencies b et ween parameters b y means of a r elaxation of the pMC. W e w ant to highligh t that this relaxed pMC is v ery similar to so-called interv al MCs, a detailed discussion is giv en in [ 94 , Section 5.1.1.3]. In tuitively , the relaxation rel ( D ) is a pMC that arises from D to a pMC with the same state space but an up dated transition relation. In particular, it introduces a fresh copy of ev ery parameter in every state, thereb y eliminating parameter dependencies b et ween diﬀeren t states (if any). This step simpliﬁes ﬁnding an optimal instan tiation (in the relaxation). How ever, the set of instan tiated pMCs gro ws: some of the instantiations cannot b e obtained from the original pMC. In this subsection, we ﬁrst formalize the relaxation, then clarify the relation betw een prop erties b eing satisﬁed on the pMC and properties satisﬁed on the relaxation. W e ﬁnish the subsection by discussing how to eﬃcien tly analyze a relaxed pMC. Deﬁnition 26 (Relaxation) The r elaxation of pMC D = ( S , V , s I , P ) is the pMC rel ( D ) = ( S, rel D ( V ) , s I , P ′ ) with rel D ( V ) = { p s i | p i ∈ V , s ∈ S } and P ′ ( s, s ′ ) = P ( s, s ′ )[ p 1 , . . . , p n /p s 1 , . . . , p s n ]. W e extend an instantiation u for D to the r elaxe d instantiation rel D ( u ) for rel ( D ) b y rel D ( u )( p s i ) = u ( p i ) for every s . W e hav e that for all u , D [ u ] = rel ( D )[ rel D ( u )]. W e lift the relaxation to regions suc h that B ( p s i ) = B ( p i ) for all s , i. e., rel D ( R ) = × p s i ∈ rel D ( V ) I ( p i ). W e drop the subscript D , whenever it is clear from the context. Example 38 Figure 12(c) on page 41 depicts the relaxation rel ( D ) of the pMC D from Figure 12(a) on page 41. F or R = [ 1 / 10 , 4 / 5 ] × [ 2 / 5 , 7 / 10 ] and u = ( 4 / 5 , 3 / 5 ) ∈ R from Example 19 on page 24, we obtain rel ( R ) = [ 1 / 10 , 4 / 5 ] × [ 2 / 5 , 7 / 10 ] × [ 2 / 5 , 7 / 10 ] and rel ( u ) = ( 4 / 5 , 3 / 5 , 3 / 5 ). An instantiation rel ( D )[ rel ( u )] corresp onds to D [ u ] as depicted in Figure 5(d) on page 13. The relaxed region rel ( R ) con tains also instan tiations, e.g., ( 4 / 5 , 1 / 2 , 3 / 5 ) which are not realisable in R . F or a pMC D and a graph-preserving region R , relaxation increases the set of p ossible instantiations: {D [ u ] | u ∈ R } ⊆ { rel ( D )[ u ] | u ∈ rel ( R ) } . Thus, the maximal reac hability probabilit y o ver all instantiations of D within R is b ounded by the maxim um ov er the instantiations of rel ( D ) within rel ( R ). Lemma 3 F or pMC D and r e gion R : max u ∈ R  Pr D [ u ] ( ♢ T )  = max u ∈ R  Pr rel ( D )[ rel ( u )] ( ♢ T )  ≤ max u ∈ rel ( R )  Pr rel ( D )[ u ] ( ♢ T )  . Consequen tly , if rel ( D ) satisﬁes a reac habilit y prop erty , so does D . Parameter Synthesis for Marko v Mo dels 45 Corollary 4 F or pMC D and r e gion R : max u ∈ rel ( R )  Pr rel ( D )[ u ] ( ♢ T )  ≤ λ implies D , R | = P ≤ λ ( ♢ T ) . W e now formalise the earlier observ ation: Without p ar ameter dep endencies, ﬁnd- ing optimal instantiations in a pMC is simpler . Although rel ( D ) has (usually) more parameters than D , ﬁnding an instantiation u ∈ rel ( R ) that maximises the reacha- bilit y probabilit y is simpler than in u ∈ R : F or an y p s i ∈ rel ( V ), w e can in state s pic k a v alue in I ( p s i ) that maximises the probability to reach T from state s . There is no (negativ e) eﬀect for the reachabilit y probability at the other states as p s i only o ccurs at s . Optimal instan tiations can th us b e determined lo c al ly (at the states). F urthermore, as b oth D is lo cally monotone, and there are no parameter dep endencies, the maxim um reachabilit y probability is relativ ely easy to ﬁnd: W e only need to consider instan tiations u that set the v alue of each parameter to either the lo west or highest p ossible v alue, i. e., u ( p s i ) ∈ B ( p s i ) for all p s i ∈ rel ( V ): Theorem 4 L et D b e a pMC with states S and T ⊆ S and R a r e gion subje ct subje ct to Assumption 1. Ther e exists an instantiation u ∈ rel ( R ) satisfying u ( p s i ) ∈ B ( p s i ) for al l p s i ∈ rel ( V ) such that: Pr rel ( D )[ u ] ( ♢ T ) = max v ∈ rel ( R ) Pr rel ( D )[ v ] ( ♢ T ) . T o prov e this statemen t, w e consider an instantiation which assigns a v alue to a parameter strictly b etw een its b ounds. Any such instantiation can b e mo diﬁed suc h that all parameters are assigned to its b ound, without decreasing the induced reac hability probabilit y . The essen tial statemen t is the monotonicity of a parameter without any further dep endencies. The num b er of instantiations that m ust b e analysed is therefore ﬁnite, compared for inﬁnitely many candidates for non-relaxed pMCs. Lemma 4 L et D b e a lo c al ly monotone pMC with a single par ameter p that only o c curs at one state s ∈ S , i.e. P ( ˆ s, s ′ ) ∈ [0 , 1] for al l ˆ s, s ′ ∈ S with ˆ s  = s . F or r e gion R and T ⊆ S , the pr ob ability Pr D ( ♢ T ) is monotonic on R . Pr o of W. l. o. g. let s / ∈ T b e the initial state of D and let T b e reachable from s . F urthermore, let U denote the standard until-modality and ¬ T denote S \ T . Using the characterisation of reac hability probabilities as linear equation system (cf. [ 13 ]), the reac hability probability w. r. t. T (from the initial state) in D is given b y: Pr D ( ♢ T ) = X s ′ ∈ S P ( s, s ′ ) · Pr D s ′ ( ♢ T ) = X s ′ ∈ S P ( s, s ′ ) ·  Pr D s ′ ( ¬ s U T ) + Pr D s ′ ( ¬ T U s ) · Pr D ( ♢ T )  = X s ′ ∈ S P ( s, s ′ ) · Pr D s ′ ( ¬ s U T ) + X s ′ ∈ S P ( s, s ′ ) · Pr D s ′ ( ¬ T U s ) · Pr D ( ♢ T ) . T ransp osing the equation yields Pr D ( ♢ T ) = P s ′ ∈ S P ( s, s ′ ) · Pr D s ′ ( ¬ s U T ) 1 − P s ′ ∈ S P ( s, s ′ ) · Pr D s ′ ( ¬ T U s ) . 46 Sebastian Junges et al. The denominator can not be zero as T is reac hable from s . Since D is lo cally monotone, we hav e P ( s, s ′ ) = f s ′ / g s for s ′ ∈ S and multilinear functions f s ′ , g s ∈ Q [ p ]. W e obtain: Pr D ( ♢ T ) = P s ′ ∈ S f s ′ · c onstant z }| { Pr D s ′ ( ¬ s U T ) g s − P s ′ ∈ S f s ′ · Pr D s ′ ( ¬ T U s ) | {z } c onstant . Hence, Pr D ( ♢ T ) = f 1 / f 2 is a fraction of tw o multilinear functions f 1 , f 2 ∈ Q [ p ] and therefore monotonic on R . Pr o of (The or em 4 on the pr evious p age) W e pro ve the statement b y contraposition. Let u ∈ rel ( R ) with Pr rel ( D )[ u ] ( ♢ T ) = max v ∈ rel ( R )  Pr rel ( D )[ v ] ( ♢ T )  . F or the contra- p osition, assume that there exists a parameter p ∈ rel ( V ) with u ( p ) ∈ I R ( p ) \ B R ( p ) suc h that all instantiations u ′ ∈ rel ( R ) that set p to a v alue in B R ( p ) induce a smaller reac hability probabilit y , i.e. u ′ ( p ) ∈ B R ( p ) and u ′ ( q ) = u ( q ) for q  = p implies Pr rel ( D )[ u ′ ] ( ♢ T ) < Pr rel ( D )[ u ] ( ♢ T ) . Consider the pMC ˆ D = ( S, { p } , s, ˆ P ) with the single parameter p that arises from rel ( D ) b y replacing all parameters q ∈ rel ( V ) \ { p } with u ( q ). W e hav e ˆ D [ u ] = rel ( D )[ u ]. Moreo ver, Pr ˆ D ( ♢ T ) is monotonic on I ( p ) according to Lemma 4 on the previous page. Th us, there is an instan tiation u ′ ∈ rel ( R ) with u ′ ( p ) ∈ B R ( p ) and u ′ ( q ) = u ( q ) for q  = p satisfying Pr ˆ D [ u ] ( ♢ T ) ≤ Pr ˆ D [ u ′ ] ( ♢ T ) = Pr rel ( D )[ u ′ ] ( ♢ T ) . This con tradicts our assumption for parameter p . 7.3 Replacing parameters by nondeterminism In order to determine max u ∈ rel ( R ) Pr rel ( D )[ u ] ( ♢ T ), it suﬃces to make a discrete c hoice ov er instantiations u : rel ( V ) → R with u ( p s i ) ∈ B ( p i ). This choice can b e made lo cally at ev ery state, whic h brings us to the k ey idea of c onstructing a (non-p ar ametric) MDP out of the pMC D and the r e gion R , where nondeterministic c hoices represent all instantiations that ha ve to be considered. In the following, it is con venien t to refer to the parameters in a given state s by: V s = { p ∈ V | p occurs in D ( s, s ′ ) for some s ′ ∈ S } . Deﬁnition 27 (Substitution (pMCs)) F or pMC D = ( S , V , s I , P ) and region R , let the MDP sub R ( D ) = ( S, s I , A ct sub , P sub ) with – A ct sub = U s ∈ S A ct s where A ct s = { u : V s → R | ∀ p ∈ V s . u ( p ) ∈ B ( p ) } , and – P sub ( s, u, s ′ ) = ( P ( s, s ′ )[ u ] if u ∈ A ct s , 0 otherwise. Parameter Synthesis for Marko v Mo dels 47 s 0 s 1 s 2 s 3 s 4 1 / 10 9 / 10 4 / 5 1 / 5 2 / 5 3 / 5 7 / 10 3 / 10 2 / 5 3 / 5 7 / 10 3 / 10 1 1 (a) sub r ( D ) s 0 s 1 s 2 s 3 s 4 4 / 5 1 / 5 2 / 5 3 / 5 7 / 10 3 / 10 1 1 (b) sub r ( D ) σ Fig. 13 Illustrating parameter-substitution. b e the (p ar ameter-)substitution of D and R . Th us, choosing action u in s corresp onds to assigning one of the extremal v alues B ( p i ) to the parameters p s i . The n umber of outgoing actions from state s is therefore 2 | V s | . Example 39 Consider pMC D – depicted in Figure 12(a) on page 41 – with R = [ 1 / 10 , 4 / 5 ] × [ 2 / 5 , 7 / 10 ] as b efore. The substitution of D and R is shown in Figure 13(a). In D , each outgoing transition of states s 0 , s 1 , s 2 is replaced by a nondeterministic c hoice in MDP sub R ( D ). That is, w e either pick the upp er or low er bound for the corresp onding v ariable. The solid (dashed) lines depict transitions that b elong to the action for the upp er (low er) b ound. F or the states s 3 and s 4 , the choice is unique as their outgoing transitions in D are constant. Figure 13(b) depicts the MC sub R ( D ) σ whic h is induced b y the strategy σ on MDP sub D ( R ) that c ho oses the upp er bounds at s 0 and s 2 , and the lo wer b ound at s 1 . Notice that sub R ( D ) σ coincides with rel ( D )[ v ] for a suitable instantiation v , as depicted in Fig. 12(c) on page 41. The substitution enco des the lo cal choices for a relaxed pMC. That is, for an arbitrary pMC, there is a one-to-one corresp ondence b et w een strategies σ in the MDP sub rel ( R ) ( rel ( D )) and instan tiations u ∈ rel ( R ) for rel ( D ) with u ( p s i ) ∈ B ( p i ). F or b etter readabilit y , we will omit the superscripts for sets of strategies Str . Com bining these observ ations with Theorem 4 on page 45, yields the following. Corollary 5 F or a pMC D , a gr aph-pr eserving r e gion R , and a set T of tar get states of D : max u ∈ R Pr D [ u ] ( ♢ T ) ≤ max σ ∈ Str Pr sub rel ( R ) ( rel ( D )) σ ( ♢ T ) min u ∈ R Pr D [ u ] ( ♢ T ) ≥ min σ ∈ Str Pr sub rel ( R ) ( rel ( D )) σ ( ♢ T ) . F urthermore, the nondeterministic choices introduced b y the substitution only dep end on the v alues B ( p i ) of the parameters p i in R . Since the ranges of the parameters p s i in rel ( R ) agree with the range of p i in R , we hav e sub rel ( R ) ( rel ( D )) = sub R ( D ) for all graph-preserving R. (15) A direct consequence of these statemen ts yields: 48 Sebastian Junges et al. Algorithm 4 P arameter lifting reac hability (pMC D , T ⊆ S , region R , speciﬁcation P ≤ λ ( ♢ T ) ) // Check whether D , R | = P ≤ λ ( ♢ T ) Construct sub R ( D ) if ∀ σ ∈ Str sub R ( D ) | = P ≤ λ ( ♢ T ) then // via standard MDP mo del chec king pro cedures return true else if ∀ σ ∈ Str sub R ( D ) | = P >λ ( ♢ T ) then // via standard MDP mo del chec king pro cedures return false else return unknown Theorem 5 L et D b e a pMC, R a gr aph-pr eserving r e gion, φ a r e achability pr op erty, subje ct to Assumption 1. Then it holds: ∀ σ ∈ Str . sub R ( D ) σ | = φ = ⇒ D , R | = φ ∧ ∀ σ ∈ Str . sub R ( D ) σ | = ¬ φ = ⇒ D , R | = ¬ φ. Hence, we can deduce via Algorithm 4 whether D , R | = φ b y applying standard tec hniques for MDP mo del chec king to sub R ( D ), such as v alue- and policy iteration, cf. [ 13 , 123 ]. W e stress that while the relaxation is key for sho wing the correctness, equation (15 on the previous page) pro ves that this step do es not actually need to b e performed. Example 40 Reconsider Example 39 on the previous page. F rom sub R ( D ) in Fig- ure 13(a) on the previous page, w e can deriv e max σ ∈ Str Pr sub R ( D ) σ ( ♢ T ) = 47 / 60 and, b y Theorem 5, D , R | = P ≤ 4 / 5 ( ♢ T ) follows. Despite the large region R , we establish a non-trivial upp er bound on the reachabilit y probability o ver all instan tiations in R . If the ov er-approximation by region R is to o coarse for a conclusive answer, region R can b e reﬁned, meaning that w e split R in to a set of smaller regions 13 [ 30 ]. W e discuss splitting strategies in Section 9. Intuitiv ely , as more p otential parameter v alues are excluded by reducing the region size, the actual choice of the parameter v alue has less impact on reac hability probabilities. The smaller the region gets, the smaller the o ver-appro ximation: The optimal instantiation on the pMC D is o ver- appro ximated b y some strategy on sub R ( D ). The appro ximation error originates from c hoices where an optimal strategy on sub R ( D ) c ho oses actions u 1 and u 2 at states s 1 and s 2 , respectively , with u 1 ( p s 1 i )  = u 2 ( p s 2 i ) for some parameter p i , and therefore intuitiv ely disagree on its v alue. The probability mass that is aﬀected by these c hoices decreases the smaller the region is. F or inﬁnitesimally small regions, the error from the o ver-appro ximation v anishes, as the actions for the upp er and the low er bound of a parameter b ecome equal up to an inﬁnitesimal. More formally , the diﬀerence in reachabilit y probability betw een tw o MCs corresp onding to instan tiations in a region tends is b ounded and tends to zero if the region gets smaller [45, Lemma 9]. 13 Strictly speaking, these regions will ov erlap as we alw ays consider closed regions. This is not a concern for correctness. When splitting, we may take this information into account, see Section 9.2.2. Parameter Synthesis for Marko v Mo dels 49 7.4 Exp ected reward prop erties The reduction of b ounding reac habilit y probabilities on pMCs to oﬀ-the-shelf MDP mo del chec king can also b e applied to bound exp ected rewards. T o see this, we ha ve to extend the notion of lo cally monotone parametric Marko v chains. Deﬁnition 28 (Lo cally monotone rew ard pMC) A pMC D = ( S , V , s I , P ) with rew ard function rew : S → Q ( V ) is lo c al ly monotone iﬀ for all s ∈ S , there is a m ultilinear p olynomial g s ∈ Q [ V ] with { rew( s ) , P ( s, s ′ ) | s ′ ∈ S } ⊆ { f / g s | f ∈ Q [ V ] multilinear } . W e now generalise relaxation and substitution to the reward mo dels, and obtain analogous results. Deﬁnition 29 (Substitution for reward pMCs) Let D = ( S , V , s I , P ) be a pMC, rew : S → Q ( V ) a reward function, T ⊆ S a set of target states, and R a region. F or s ∈ S , let V rew s = V s ∪ { p i ∈ V | p i o ccurs in rew( s ) } . The MDP sub rew R ( D ) = ( S, s I , A ct rew sub , P rew sub ) with reward function rew sub is the (p ar ameter-)substitution of D , rew on R , where – A ct rew sub and P rew sub are analogous to Deﬁnition 27 on page 46, but o v er V rew s . – rew sub is giv en by: ( s, u ) 7→ ( rew( s )[ u ] if u ∈ A ct rew s , 0 otherwise, where A ct rew s is deﬁned analogously to A ct s in Deﬁnition 27. The rew ard approximation of a pMC can b e used to identify regions as accepting or rejecting for exp ected rew ard prop erties. Theorem 6 L et D b e a pMC with lo c al ly monotone r ewar ds rew , R a r e gion, and φ an exp e cte d r ewar d pr op erty, subje ct to Assumption 1: ∀ σ ∈ Str . sub rew R ( D ) | = φ implies D , R | = φ and ∀ σ ∈ Str . sub rew R ( D ) | = ¬ φ implies D , R | = ¬ φ. The pro of is analogous to the proof of Theorem 5 on the previous page. 8 Mo del-chec king-based Region V eriﬁcation of P arametric MDPs In the previous section, we approximated reachabilit y probabilities in (lo cally- monotone) pMCs by considering the substitution MDP , see Deﬁnition 27 on page 46. The non-determinism in the MDP enco des the ﬁnitely man y parameter v aluations that approximate the reachabilit y probabilities in the pMC. By letting an adversary play er resolve the non-determinism in the MDP , we obtain b ounds on the reac habilit y probabilities in the pMC. These bounds can eﬃciently be computed b y standard MDP mo del c hecking. 50 Sebastian Junges et al. In this section, w e generalise the approach to pMDPs, which already contain non-determinism. The result naturally leads to a 2-play er sto chastic game: One pla yer controls the nondeterminism inherent to the MDP , while the other play er con trols the (abstracted) parameter v alues. Letting the tw o play ers adequately minimise and/or maximise the reachabilit y probabilities in the SG yields b ounds on the minimal (and maximal) reac hability probabilities in the pMDP . F or example, if the play er for the original non-determinism maximises and the parameter play er minimises, w e obtain a lo wer b ound on the maximal probability . These b ounds can eﬃcien tly b e computed b y standard SG mo del chec king pro cedures. In our presen tation b elow, w e discuss the interpla y of the t w o sources of non- determinism. In particular, w e show ho w the generalisation of the method yields an additional source of (ov er-)approximation. Then, we formalise the construction of the substitution with nondeterminism, analogous to the pMCs from the previous section. In particular, Deﬁnition 30 on page 52 is analogous to Deﬁnition 27 on page 46 and Theorem 7 on page 53 is analogous to Theorem 5 on page 48. W e do not rep eat relaxation, describ ed in Section 7.2, as—as also discussed in the previous section—it is not a necessary ingredien t for the correctness of the approac h. 8.1 Two types of approximation In the follo wing, let M = ( S , V , s I , A ct , P ) b e a pMDP and R a graph-preserving, rectangular, closed region. Demonic str ate gies W e analyse R with resp ect to the demonic relation | = d . W e ha ve: M , R | = d φ ⇐ ⇒ ∀ u ∈ R. ∀ σ ∈ Str M . M [ u ] σ | = φ. The t w o univ ersal quantiﬁers can b e reordered, and in addition M [ u ] σ = M σ [ u ]. W e obtain: M , R | = d φ ⇐ ⇒ ∀ σ ∈ Str M . ∀ u ∈ R . M σ |{z} a pMC [ u ] | = φ In tuitively , the reform ulation states that we ha ve to apply pMC region veriﬁcation on M σ and R for all σ ∈ Str M . W e no w w ant to employ parameter lifting for eac h strategy . Thus, w e wan t to consider the v eriﬁcation of the substituted pMCs sub R ( M σ ). As these substituted pMCs share most of their structure, the set of all suc h substituted pMCs can b e concisely represen ted as an SG, in which b oth play ers co op erate (as witnessed b y the same quantiﬁers). In the scope of this paper, an SG with coop erating play ers can b e concisely represen ted as an MDP . Consequently , for the demonic relation, pMDP veriﬁcation can b e appro ximated by MDP model c hecking. Angelic str ate gies W e now turn our attention to the angelic relation | = a , cf. Deﬁni- tion 14 on page 20. M , R | = a φ ⇐ ⇒ ∀ u ∈ R. ∃ σ ∈ Str M . M [ u ] σ | = φ. Parameter Synthesis for Marko v Mo dels 51 s α β (a) M s (b) sub R ( M σ ) s s, α s, β α β (c) M ′ s s, α s, β α β (d) sub R ( M ′ ) s s, α s, β α (e) sub R ( M ′ ) σ Fig. 14 Illustration of the substitution of a pMDP . Here, w e cannot simply reorder the quantiﬁers. How ever: ∃ σ ∈ Str M . ∀ u ∈ R . M σ [ u ] | = φ = ⇒ M , R | = a φ. No w, the left-hand side expresses again that we wan t to do region veriﬁcation for pMCs induced by a strategy , as in the demonic case, and that w e likewise w ant to represen t b y a stochastic game. As witnessed by the quantiﬁer alternation, this SG do es not reduce to an MDP; the tw o pla yers ha ve opposing ob jectives. Nev ertheless, w e can eﬃciently analyse this SG (with a v ariant of v alue iteration), and thus the left-hand side of the implication abov e. Observ e that the o ver-appro ximation actually computes a robust strategy , as discussed in Remark 7 on page 22. In particular, we now hav e tw o sources of appro ximation: – The appro ximation that originates from dropping parameter dep endencies (as also in the demonic case). – The application of the substitution of parameters with non-determinism on robust strategies rather than of the actual angelic relation. Both o ver-appro ximations v anish with declining region size. 8.2 Replacing parameters by nondeterminism Example 41 Consider the pMDP M in Figure 14(a), where the state s has tw o enabled actions α and β . The strategy σ giv en b y { s 7→ α } applied to M yields a pMC, whic h is sub ject to substitution, cf. Figure 14(b). The parameter substitution of a pMDP (cf. Figure 14(a)) yields an SG—as in Fig- ure 14(d). It represents, for all strategies of the pMDP , the parameter-substitution (as in Deﬁnition 27 on page 46) of eac h induced pMC. T o ensure that in the SG eac h state can b e assigned to a unique play er, we split states in the pMDP whic h ha ve b oth (parametric) probabilistic branching and non-determinism, suc h that states ha ve either probabilistic branching or non-determinism, but not both. The reform ulation is done as follo ws: After each choice of actions, auxiliary states are in tro duced, such that the outcome of the action b ecomes deterministic and the probabilistic c hoice is dela yed to the auxiliary state. This construction is similar to the conv ersion of Segala’s probabilistic automata in to Hansson’s alternating mo del [129]. More precisely , we – split each state s ∈ S in to { s } ⊎ {⟨ s, α ⟩ | α ∈ A ct ( s ) } , 52 Sebastian Junges et al. – add a transition with probability one for each s ∈ S and α ∈ Act ( s ). The transition leads from s to ⟨ s, α ⟩ , and – mov e the probabilistic choice at s w. r. t. α to ⟨ s, α ⟩ . Applying this to the pMDP from Figure 14(a) on the previous page, we obtain the pMDP M ′ in Figure 14(c) on the previous page, where the state s has only nondeterministic choices leading to states of the form ⟨ s, α ⟩ with only pr ob abilistic c hoices. The subsequent substitution on the probabilistic states yields the SG sub R ( M ′ ), where one pla yer represents the nondeterminism of the original pMDP M , while the other pla yer decides whether parameters should b e set to their lo wer or upper b ound in the region R . F or the construction, we generalise V s to state-action pairs: F or a pMDP , a state s and action α , let V s,α = { p ∈ V | p occurs in P ( s, α, s ′ ) for some s ′ ∈ S } . Deﬁnition 30 (Substitution (pMDPs)) F or pMDP M = ( S , V , s I , A ct , P ) and region R , let SG sub R ( M ) = ( S  ⊎ S 2 , s I , A ct sub , P sub ) with – S  = S – S 2 = {⟨ s, α ⟩ | α ∈ Act ( s ) } , – A ct sub = A ct ⊎  U ⟨ s,α ⟩∈ S 2 A ct α s  where A ct α s = { u : V s,α → R | u ( p ) ∈ B ( p ) ∀ p ∈ V s,α } , and, – P sub ( t, β , t ′ ) =      1 if t ∈ S  , β ∈ A ct ( t ) , t ′ = ⟨ t, β ⟩ ∈ S 2 , P ( s, α, t ′ )[ β ] if t = ⟨ s, α ⟩ ∈ S 2 , β ∈ A ct α s , t ′ ∈ S  , 0 otherwise. b e the (p ar ameter-)substitution of M and R . W e relate the SG sub R ( M ) under diﬀeren t strategies for play er  with the sub- stitution in the strategy-induced pMCs of M . W e observ e that the strategies for pla yer  in sub R ( M ) coincide with strategies in M . Consider the induced MDP ( sub R ( M )) σ with a strategy σ for play er  .The MDP ( sub R ( M )) σ is obtained from sub R ( M ) by erasing transitions not agreeing with σ . In ( sub R ( M )) σ pla yer  -state ha ve a single enabled action, while play er 2 -states hav e multiple a v ailable enabled actions. Example 42 Con tinuing Example 41 on the previous page, applying strategy σ to sub R ( M ) yields ( sub R ( M )) σ , see Figure 14(e) on the previous page. The MDP ( sub R ( M )) σ matc hes the MDP sub R ( M σ ) apart from in termediate states of the form ⟨ s, α ⟩ : The outgoing transitions of s in sub R ( M σ ) coincide with the outgoing transitions of ⟨ s, α ⟩ in ( sub R ( M )) σ , where ⟨ s, α ⟩ is the unique successor of s . The following corollary formalises that ( sub R ( M )) σ and sub R ( M σ ) induce the same reac hability probabilities. Parameter Synthesis for Marko v Mo dels 53 Corollary 6 F or pMDP M , gr aph-preserving r e gion R , tar get states T ⊆ S , and str ate gies σ ∈ Str sub R ( M )  and ρ ∈ Str sub R ( M σ ) , it holds that Pr ( sub R ( M σ )) ρ ( ♢ T ) = Pr sub R ( M ) σ, b ρ ( ♢ T ) with b ρ ∈ Str sub R ( M ) 2 satisﬁes b ρ ( ⟨ s, σ ( s ) ⟩ ) = ρ ( s ) . Instead of performing the substitution on the pMC induced by M and σ , w e can p erform the substitution on M directly and preserv e the reac hability probability . Consequen tly , and analogously to the pMC case (cf. Theorem 5 on page 48), w e can deriv e whether M , R | = ♣ φ b y analysing a sto chastic game. F or this, we consider v arious standard v ariants of mo del chec king on sto chastic games. Deﬁnition 31 (Mo del-relation on SGs) F or an SG G , prop erty φ , and quantiﬁers Q 1 , Q 2 , w e deﬁne G | = Q 1 , Q 2 φ as: Q 1 σ  ∈ Str sub R ( M )  . Q 2 σ 2 ∈ Str sub R ( M ) 2 G σ  ,σ 2 | = φ The order of pla yers, for these games, do es not inﬂuence the outcome [48, 130]. Theorem 7 L et M b e a pMDP, R a r e gion, and φ a r e achability pr op erty, subje ct to Assumption 1 14 . Then: sub R ( M ) | = ∀ , ∀ φ implies M , R | = d φ , and sub R ( M ) | = ∃ , ∀ φ implies M , R | = a φ. Pr o of W e only prov e the second statement using φ = P >λ ( ♢ T ), other reac hability prop erties are similar. A proof for the (simpler) ﬁrst statement can b e derived in an analogous manner. W e ha ve that M , R | = a P >λ ( ♢ T ) iﬀ for all u ∈ R there is a strategy σ of M for whic h the reac habilit y probability in the MC M σ [ u ] exceeds the threshold λ , i. e., M , R | = a P >λ ( ♢ T ) ⇐ ⇒ min u ∈ R max σ ∈ Str M Pr M σ [ u ] ( ♢ T ) > λ. A lo wer b ound for this probability is obtained as follo ws: min u ∈ R max σ ∈ Str M  Pr M σ [ u ] ( ♢ T )  ≥ max σ ∈ Str M min u ∈ R  Pr M σ [ u ] ( ♢ T )  ∗ ≥ max σ ∈ Str M min ρ ∈ Str sub R ( M σ )  Pr ( sub R ( M σ )) ρ ( ♢ T )  ∗∗ = max σ ∈ Str sub R ( M )  min ρ ∈ Str sub R ( M ) 2  Pr sub R ( M ) σ,ρ ( ♢ T )  . 14 straightforw ardly lifting lo cally monotone pMCs to lo cally monotone pMDPs 54 Sebastian Junges et al. The inequality ∗ is due to Corollary 5 on page 47. The equality ∗∗ holds by Corollary 6 on the previous page. Then: sub R ( M ) | = ∃ , ∀ P >λ ( ♢ T ) ⇐ ⇒ ∃ σ ∈ Str sub R ( M )  . ∀ ρ ∈ Str sub R ( M ) 2 G σ,ρ | = P >λ ( ♢ T ) ⇐ ⇒ max σ ∈ Str G   min ρ ∈ Str G 2  Pr G σ,ρ ( ♢ T )   > λ = ⇒ min u ∈ R max σ ∈ Str M  Pr M σ [ u ] ( ♢ T )  > λ ⇐ ⇒ M , R | = a P >λ ( ♢ T ) . 9 Approximate Synthesis by Parameter Space Partitioning P arameter space partitioning is our iterative approac h to the approximate syn thesis problem. It builds on top of region v eriﬁcation, discussed abov e, and is, conceptually , indep enden t of the metho ds used for v eriﬁcation discussed later. P arameter space partitioning is b est viewed as a counter-example guided abstraction reﬁnement (CEGAR)-like [ 47 ] approach to successiv ely divide the parameter space in to accepting and rejecting regions. The main idea is to compute a sequence  R i a  i of simple accepting regions that successiv ely extend each other. Similarly , an increasing sequence  R i r  i of simple rejecting regions is computed. At the i -th iteration, R i = R i a ∪ R i r is the cov ered fragment of the parameter space. The iterativ e approach halts when R i is at least c % of the entire parameter space. T ermination is guaran teed: in the limit a solution to the exact syn thesis problem is obtained as lim i →∞ R i a = R a and lim i →∞ R i r = R r . Let us describ e the synthesis loop for the appro ximate synthesis as depicted in Figure 4 on page 10 in detail. In particular, w e discuss ho w to generate c andidate r e gions that can b e dispatc hed to the v eriﬁer along with a hyp othesis whether the candidate region is accepting or rejecting. W e fo cus on r e ctangular r e gions for sev eral reasons: – the automated generation of rectangular regions is easier to generalise to m ultiple dimensions, – earlier exp erimen ts [ 64 ] rev ealed that rectangular regions lead to a more eﬃcient SMT-based v eriﬁcation of regions (describ ed in Section 6), and – mo del-c hecking based region v eriﬁcation (describ ed in Section 7) requires rectangular regions. A downside of rectangular regions is that they are neither well-suited to approximate a region partitioning given by a diagonal, nor to co ver well-deﬁned regions that are not rectangular themselv es. R emark 11 In the following, w e assume that the parameter space is given by a rectangular w ell-deﬁned region R . If the parameter space is not rectangular, we o ver- appro ximate R b y a rectangular region ˆ R ⊇ R . If the p otential o v er-approximation of the parameter space ˆ R is not w ell-deﬁned, then we iterativ ely approximate ˆ R b y Parameter Synthesis for Marko v Mo dels 55 0.0 0.2 0.4 0.6 0.8 1.0 x 0.0 0.2 0.4 0.6 0.8 1.0 y Fig. 15 Parameter space partitioning into safe and unsafe regions. a sequence of w ell-deﬁned and ill-deﬁned 15 regions. The regions in the sequence of well-deﬁned regions are then sub ject to the syn thesis problem. Constructing the sequence of regions is done analogously to the partitioning in to accepting and rejecting regions. Before w e presen t the procedure in full detail, we ﬁrst outline a naive reﬁnemen t pro cedure b y means of an example. Example 43 (Naive r eﬁnement lo op) Consider the parametric die from Example 5 on page 6. Supp ose we wan t to syn thesise the partitioning as depicted in Figure 2 on page 7. W e start b y verifying the full parameter space R against φ . The veriﬁer returns false , as R is not accepting. Since R (based on our knowledge at this p oint) migh t b e rejecting, w e inv oke the v eriﬁer with R and ¬ φ , yielding false to o. Th us, the full parameter space R is inconsisten t. W e now split R in to four equally-sized regions, all of which are inconsisten t. Only after splitting again, w e ﬁnd the ﬁrst accepting and rejecting regions. After v arious iterations, the procedure leads to the partitioning in Figure 15. Algorithm 5 on the next page describes this naive region partitioning procedure. It tak es a pSG, a region R , a sp eciﬁcation φ , and a (demonic or angelic) satisfaction relation as input. It ﬁrst initialises a (priorit y) queue Q with R . In each iteration, a subregion R ′ of R is taken from the queue, the counter i is incremented, and the sequence of accepted and rejected regions is up dated. There are three p ossibilities. Either R ′ is accepting (or rejecting), and R i a ( R i r ) extends R i − 1 a ( R i − 1 r ) with R ′ , or R ′ is inconsistent. In the latter case, we split R ′ in to a ﬁnite set of subregions that are inserted in to the queue Q . Regions that are not extended are unc hanged. The algorithm only terminates if R a and R r are a ﬁnite union of hyper-rectangles. Ho wev er, the algorithm can b e terminated after any iteration yielding a sound appro ximation. The algorithm ensures lim i →∞ R i = R , if we order Q according to the size of the regions. W e omit the technical proof here; the elementary property 15 A region R is ill-deﬁned if no instantiation in R is w ell-deﬁned. 56 Sebastian Junges et al. Algorithm 5 Naiv e reﬁnement lo op naiv e-reﬁnement (pSG G , rectangular region R , ♣ ∈ { a, d } , sp eciﬁcation φ ) i : = 0 Q : = { R } , R i a : = ∅ , R i r : = ∅ while Q  = ∅ do i : = i + 1 R ′ : = Q .pop if G , R ′ | = ♣ φ then R i a : = R i − 1 a ∪ R ′ , R i r : = R i − 1 r else if G , R ′ | = ♣ ¬ φ then R i a : = R i − 1 a , R i r : = R i − 1 r ∪ R ′ else R i a : = R i − 1 a , R i r : = R i − 1 r Q : = Q ∪ split ( R ′ ) return Accepting region R i a , Rejecting region R i r is that the regions are Leb esgue-measurable (and hav e a p ositiv e measure by construction). The naiv e algorithm has a couple of structural w eaknesses: – It inv okes the v eriﬁcation algorithm twice to determine that the full parameter space is inconsisten t. – It do es not provide any (diagnostic) information from a veriﬁcation inv o cation yielding false . – It c hecks whether a region is accepting b efore it chec ks whether it is rejecting. This order is sub optimal if the region is rejecting. – If the region is inconsistent, it splits the region in to 2 n equally large regions. Instead, it might b e b eneﬁcial to select a smaller n um b er of regions (only split in one dimension). – Uninformed splitting yields many inconsistent subregions. Splitting in only one dimension ev en increases the num b er of veriﬁcation calls yielding false . In the remainder of this section, we discuss w ays to alleviate these w eaknesses. The prop osed improv ements are based on empirical observ ations ab out the b ench- marks and are in line with the implementation in our tool PROPhESY . In particular, w e tailor the heuristics to “w ell-b ehav ed” mo dels and sp eciﬁcations, whic h reﬂect the b enc hmarks from v arious domains. The notion of b eing well-behav ed refers to – a limited n umber of connected accepting and rejecting regions with smo oth (alb eit highly non-linear) borders b etw een these regions. – a limited n umber of accepting (rejecting) instantiations that are close to a rejecting (accepting) instan tiations. W e call instantiations that form a b order b et ween R a and R r b or der instan tiations. The parameter space depicted in Figure 15 on the previous page is w ell-b ehav ed. It features only tw o connected regions, with a smo oth border b et ween them. F urther- more, the regions hav e a considerable interior, or equiv alently , man y instantiations are not too close to the b order. W e remark that we do rely on these assumptions to hold, but PROPhESY will be slow on models that are not well-behav ed. Parameter Synthesis for Marko v Mo dels 57 0.0 0.2 0.4 0.6 0.8 1.0 x 0.0 0.2 0.4 0.6 0.8 1.0 y (a) (Uniform) Sampling 0.0 0.2 0.4 0.6 0.8 1.0 p 0.0 0.2 0.4 0.6 0.8 1.0 q (b) Generating candidates 0.0 0.2 0.4 0.6 0.8 1.0 x 0.0 0.2 0.4 0.6 0.8 1.0 y (c) Preliminary result Fig. 16 Parameter space partitioning in progress: Images generated by PROPhESY . 9.1 Sampling A simple but eﬀective improv ement is to verify an instan tiated mo del G [ u ] for some instan tiation (a sample ) u ∈ R . The veriﬁcation result either reveals that the region is not accepting, if G [ u ] | = ♣ φ , or not rejecting, if G [ u ] | = ♣ φ . Tw o samples within a region R ma y suﬃce to conclude that R is inconsisten t. In order to quickly ﬁnd inconsisten t regions b y sampling, it is b eneﬁcial to seek for border instantiations. T o this end, a go o d strategy is to start with a coarse (random) sampling to get a ﬁrst indication of b order instantiations. W e then select additional instan tiations b y in tra-/extrap olation b et ween these samples. Example 44 W e discuss ho w sampling ma y improv e the naiv e reﬁnemen t loop as discussed in Example 43 on page 55. Figure 16(a) shows a uniform sampling. Red crosses indicate that the instan tiated pMC satisﬁes ¬ φ , while green dots indicate that the instan tiation satisﬁes φ . The blue rectangle is a candidate region (with the h yp othesis ¬ φ , indicated b y the hatching), which is consisten t with all samples. 9.2 Finding region candidates W e use the sampling results to steer the selection of a candidate region that may either b e accepting or rejecting. A simple strategy is to split regions that w e found to b e inconsisten t via sampling. Example 45 Consider the parameter space with six samples depicted in Figure 17(a) on the next page. After verifying only six instantiated models, we conclude that the parameter space is inconsisten t. The use of samples allo ws to improv e the naive reﬁnemen t scheme as giv en in Algorithm 5 on the previous page. This impro vemen t is giv en in Algorithm 6 on the next page. F or each region R , w e ha ve a ﬁnite set X of samples. F or each sample u ∈ X , it is kno wn whether G [ u ] | = ♣ φ . The queue Q no w contains pairs ( R, X ). In each iteration, a pair ( R ′ , X ′ ) where R ′ is (as b efore) a subregion of R is tak en from the queue. Then, we distinguish (again) three p ossibilities. Only when all samples in X ′ satisfy φ , it is veriﬁed whether R ′ is accepting. If R ′ is accepting, w e pro ceed as b efore: R i a is extended b y R ′ while R i r remains unc hanged. In the symmetric case that all samples in X ′ refute φ , we pro ceed in a similar wa y by v erifying whether R ′ rejects φ . Otherwise, R ′ is split into a ﬁnite set of subregions with corresp onding subsets of X ′ , and added to the queue Q . In case the v eriﬁcation 58 Sebastian Junges et al. 1 / 2 1 1 / 2 1 p q (a) 1 / 2 1 1 / 2 1 p q (b) 1 / 2 1 1 / 2 1 p q (c) 1 / 2 1 1 / 2 1 p q (d) Fig. 17 Creating region candidates based on samples. Algorithm 6 Sampling-based reﬁnemen t lo op sampling-reﬁnemen t (pSG G , rectangular region R , ♣ ∈ { a, d } , speciﬁcation φ ) i : = 0, Q : = { ( R, sample ( R )) } , R i a : = ∅ , R i r : = ∅ while Q  = ∅ do i : = i + 1 ( R ′ , X ′ ) : = Q .pop if G , X ′ | = ♣ φ and G , R ′ | = ♣ φ then R i a : = R i − 1 a ∪ R ′ , R i r : = R i − 1 r else if G , X ′ | = ♣ ¬ φ and G , R ′ | = ♣ ¬ φ then R i a : = R i − 1 a , R i r : = R i − 1 r ∪ R ′ else R i a : = R i − 1 a , R i r : = R i − 1 r Q : = Q ∪ split ( R ′ , X ′ ) return Accepting region R i a , Rejecting region R i r engine pro vides a counterexample, w e can add this counterexample as a new sample. W e thus ensure that for all ( R ′ , X ′ ) ∈ Q , u ∈ X ′ implies u ∈ R ′ . The algorithm can b e easily extended such that sampling is also done once a region without samples is obtained: rather than inserting ( R ′ , ∅ ) in to Q , w e insert the en try ( R ′ , sample ( R ′ )). Example 46 After sev eral more iterations, the reﬁnemen t loop started in Example 44 on the previous page has proceeded to the state in Figure 16(b) on the previous page. First, w e see that the candidate region from Figure 16(a) on the previous page was not rejecting. The veriﬁcation engine ga ve a coun terexample in form of an accepting sample (around p 7→ 0 . 45 , q 7→ 0 . 52). F urther iterations with smaller regions had Parameter Synthesis for Marko v Mo dels 59 some successes, but some additional samples w ere generated as counterexamples. The curren t blue candidate is to b e c heck ed next. In Figure 16(c) on page 57, w e see a further con tinuation, with ev en smaller regions b eing veriﬁed. Note the white b o x on the righ t b order: It has b een chec ked, but the veriﬁcation timed out without a conclusiv e answ er. Therefore, w e do not ha ve a counterexample in this subregion. It remains to discuss some metho ds to split a region, and how w e may discard some of the constructed regions. W e outline more details b elo w. 9.2.1 How to split Splitting of regions based on the av ailable samples can b e done using diﬀerent strategies. W e outline tw o basic approaches. These approac hes can be easily mixed and extended, and their p erformance heavily dep ends on the concrete example at hand. Equal splitting. This approach splits regions in equally-sized regions; the main rationale is that this generates small regions with concise b ounds (the b ounds are t ypically p ow ers of tw o). Splitting in equally sized regions can b e done recursively: One pro jects all samples down to a single dimension, and splits if b oth accepting and rejecting samples are in the region. The pro cedure halts if all samples in a region are either accepting or rejecting. The order in which parameters are considered pla ys a crucial role. T ypically , it is a go o d idea to ﬁrst split along the larger dimensions. Example 47 A split in equally-sized regions is depicted in Figure 17(b) on the previous page, where ﬁrst the left region candidate is created. The remaining region can b e split either horizon tally or v ertically to immediately generate another region candidate. A horizon tal split in the remaining region yields a region without an y samples. The do wnside of equal splitting is that the p osition of the splits are not adapted based on the samples. Therefore, the num b er of splits migh t b e signiﬁcantly larger than necessary , leading to an increased num b er of veriﬁcation calls. Gr owing r e ctangles. This approach attempts to gradually obtain a large region candidate 16 . The underlying rationale is to quickly co ver v ast amounts of the parameter space. This is illustrated in Figure 17(d) on the previous page (notice that w e adapted the samples for a consistent but concise description) where from an initial sampling a large rectangle is obtained as region candidate. Example 48 Consider the shaded regions in Figure 17(c) on the previous page. Starting from v ertex v = (1 , 1), the outer rectangle is maximised to not con tain an y accepting samples. T aking this outer rectangle as candidate region is v ery optimistic, it assumes that the accepting samples are on the b order. A more p essimistic v ariant of gro wing rectangles is given b y the inner shaded region. It takes a rejecting sample as v ertex v ′ suc h that the v and v ′ span the largest region. 16 The approach shares its rationale with the approach formerly implemented in PROPhESY [ 64 ], but is realised slightly diﬀerently to ov ercome challenges for n-dimensional h yp er-rectangles. 60 Sebastian Junges et al. The gro wing rectangles algorithm iterates o ver a subset of the h yp er-rectangle’s v ertices: F or each v ertex (referred to as anchor ), among all p ossible sub-hyper- rectangles containing the anc hor and only accepting or only rejecting samples, the largest is constructed. Example 49 The growing rectangles approac h pessimistically tak es anchor (0 , 0) as anc hor and yields the candidate region in Figure 17(d) on page 58. The v eriﬁcation fails more often on large regions (either due to time-outs or due to the ov er-approximation). Consequen tly , choosing large candidate regions comes at the risk of failed veriﬁcation calls, and fragmentation of the parameter space in more subregions. F urthermore, growing rectangles requires a fall-back splitting strategy: T o see wh y , consider Figure 15 on page 55. The accepting (green) region do es not contain an y anchors of the full parameter space, therefore the h yp othesis for an y created subregion is alw ays rejection. Thus, no subregion con taining a (known) accepting sample is ev er considered as a region candidate. 9.2.2 Neighb ourho o d analysis Besides considering samples within a region, w e would lik e to illustrate that analysis of a region R can and should take information from outside of R in to account. First, tak e Figure 17(b) on page 58, and assume that the left region is indeed accepting. The second generated region contains only rejecting samples, but it is only rejecting if all p oints, including all those on the b order to the left region, are rejecting. In other w ords, the b order b et ween the accepting and rejecting regions needs to exactly follow the border b etw een the generated region candidates. The latter case do es not o ccur often, so it is reasonable to shrink or split the second generated region. Secondly , a sensible h yp othesis for candidate regions without samples inside is helpful, especially for small regions or in high dimensions. Instead of spa wning new samples, we take samples and decided regions outside of the candidate region in to account to create a hypothesis. Concretely , w e infer the hypothesis for regions without samples via the closest kno wn region or sample. 9.3 Requirements on veriﬁcation bac k-ends In this section, we hav e describ ed tec hniques for iterativ ely partitioning the param- eter space in to accepting and rejecting regions. The algorithms rely on verifying regions (and sets of samples) against the sp eciﬁcation φ . The w ay in which v eriﬁ- cation is used in the iterative parameter space partitioning sc heme imp oses the follo wing requirements on the veriﬁcation bac k-end: i) The v eriﬁcation should work incr emental ly . That is to sa y , veriﬁcation results from previous iterations should be re-used in successive iterations. V erifying diﬀeren t regions share the same mo del (pMC or pMDP). A simple example of working incrementally is to reuse minimisation techniques for the mo del o ver several calls. If a subregion is chec k ed, the problem is even incremental in a more narro w sense: any bounds etc. obtained for the super-region are also v alid for the subregion. Parameter Synthesis for Marko v Mo dels 61 ii) If the v eriﬁcation pro cedure fails, i.e. if the veriﬁer returns false , obtaining additional diagnostic information in the form of a counterexample is beneﬁcial. A coun terexample here is a sample whic h refutes the veriﬁcation problem at hand. This wish list is v ery similar to the typical requiremen ts that theory solvers in lazy SMT framew orks should fulﬁl [ 23 ]. Therefore, SMT-based veriﬁcation approaches naturally matc h the wish-list. Parameter-lifting can work incremen tally: it reuses the graph-structure to a void rebuilding the MDP , and it ma y use previous mo del c hecking results to improv e the time until the model chec ker con verges. P arameter- lifting, due to its approximativ e nature, do es provide only limited diagnostic information: In particular, it provides information which parameters w ould b e assigned with the upp er or low er bounds based on the strategy that optimizes the MDP/SG. 10 Implementation All the algorithms and constructions in this pap er ha ve b een implemen ted, and are publicly a v ailable via PROPhESY 17 . In particular, PROPhESY supp orts algorithms for: – the exact synthesis problem: via computing the solution function , using either of the three v ariants of state elimination, discussed in Section 5. – the v eriﬁcation problem: via an enc oding to an SMT-solver as in Section 6 or b y employing the p ar ameter lifting metho d as in Section 7 and 8. – the approximate synthesis problem: via p arameter sp ac e p artitioning , that iterativ ely generates veriﬁcation calls as describ ed in Section 9. PROPhESY is implemented in python , and designed as a ﬂexible to olb ox for dev eloping and experimenting with parameter syn thesis. PROPhESY in ternally hea vily relies on high-performance routines of the probabilistic model chec ker Storm [ 65 ] and the SMT Z3. PROPhESY is built in a modular wa y , suc h that it is easy to use diﬀeren t back end solvers. The computation of the solution function and the parameter lifting presen ted in the exp erimen ts hav e b een implemented in Storm . PROPhESY can b e divided in three parts: i) First and foremost, it presents a library consisting of: a) data structures for parameter spaces and instantiations, solution functions, sp eciﬁcations, etc., built around the python bindings of the library carl 18 (featuring computa- tions with p olynomials and rational functions), b) algorithms suc h as guided sampling, v arious candidate region generation procedures, decomp osition of regions, etc., metho ds that require tight in tegration with the mo del are realised via the python bindings of Storm 19 , c) abstract interfaces to bac k end to ols, in particular probabilistic model chec kers, and SMT-chec kers, together with some concrete adapters for the diﬀeren t solvers, see Figure 18 on the next page. ii) An extensive command-line in terface which provides simple access to the diﬀeren t core functionalities of the library , ranging from sampling to full parameter syn thesis. 17 github.com/moves- rwth/prophesy , archiv ed at doi.org/10.5281/zenodo.7697154 18 https://moves- rwth.github.io/pycarl/ 19 https://moves- rwth.github.io/stormpy/ 62 Sebastian Junges et al. CLI W ebservice W ebsite PROPhESY library model chec king adapter SMT adapter Storm python carl PRISM [102] Storm [65] Storm (python) z3 [112] SMT-RAT [52] Fig. 18 High-level architecture of PROPhESY and its back ends iii) A protot ypical web-service running on top of the library , which allows users to in teract with the parameter synthesis via a web-in terface. PROPhESY is constructed in a mo dular fashion: b esides the python bindings for carl , all non-standard pack ages and to ols (in particular model chec kers and SMT solvers) are optional. Naturally , the full p ow er of PROPhESY can only b e used if these pac k ages are a v ailable. Besides the methods presented in this paper, PROPhESY con tains tw o further mature parameter synthesis metho ds: i) particle- sw arm optimisation inspired by [43], and ii) conv ex optimisation from [57]. The information in the remainder details the implementation and the possibili- ties pro vided by PROPhESY . The section con tains some notions from probabilistic mo del c hecking [ 13 , 16 , 98 ]. W e refrain from pro viding detailed descriptions of these notions, as it w ould go b ey ond the scop e of this pap er. 10.1 Mo del construction and prepro cessing (R ealise d in Storm ) The model chec ker Storm supp orts the creation of pMCs and pMDPs from b oth PRISM-language mo del descriptions [ 102 ] and JANI-sp eciﬁcations [ 32 ]. The latter can b e used as in termediate format to supp ort, e.g., digital-clo c k PT As with param- eters written in Modest [ 80 ], or to supp ort exp ected time prop erties of generalised sto c hastic Petri nets [ 109 ] with parametric rates and/or weigh ts. Parametric models can b e built using the matrix-based, explicit representation, as w ell as the sym- b olic, decision diagram (dd)-based engine built on top of sylvan [ 68 ]. Both engines supp ort the computation of qualitative properties, an essen tial prepro cessing step, and bisimulation minimisation on parametric mo dels, as describ ed in [ 78 ]. W e adv o cate the use of the Storm - python API adapter: Its in teractive nature a voids the rep etition of exp ensive steps. In particular, it allows for the incremen tal usage of parameter lifting and sampling. The support for rational functions is realised via the library carl 20 . The rational function is stored as a tuple consisting of multiv ariate p olynomials. These p olynomials are b y default stored in a partially factorised fashion, cf. [ 91 ]. Eac h factor (a p olynomial) is stored as an ordered sparse sum of terms, each term 20 https://github.com/moves- rwth/carl- storm Parameter Synthesis for Marko v Mo dels 63 consists of the coeﬃcient and a sparse represen tation of v ariables with their non- zero exp onen ts. F or manipulating the (rational) co eﬃcients, w e exploit gmp 21 or cln 22 . The former is thread-safe, while the latter p erforms slightly b etter with single-thread usage. Computation of GCDs in m ultiv ariate p olynomials is done either via ginac [22] or cocoa [2]. 10.2 Solution function computation (R ealise d in Storm ) The computation of solution functions for pMCs as discussed in Section 5 is implemen ted for a v ariety of sp eciﬁcations: – reachabilit y and reach-a void probabilities, – exp ected rewards, including exp ected time of contin uous-time Marko v chains, – step-b ounded reachabilit y probabilities, and – long-run av erage probabilities and rewards. The computation is realised either via state elimination, or via Gaussian elimination. An implementation of set-based transition elimination is a v ailable for sym b olic representations of the pMC. 10.2.1 State elimination As the standard sparse matrix representation used b y Storm is not suitable for fast remo v al and insertion of entries, a ﬂexible sparse matrix with faster delete and insert op erations is used. The order in which states are eliminated has a sev ere impact on the per- formance [ 64 ]. Storm supp orts a v ariety of static (pre-computed) and dynamic orderings for the elimination: – sev eral static orders (forward (rev ersed), bac kward (rev ersed)) based on the order of state-generation by the mo del construction algorithms. This latter order is t ypically determined by a depth-ﬁrst search through the high-level mo del description 23 , – orders based on the top ology of the pMC, e.g., based on the decomp osition in strongly connected comp onen ts, – orders (Regex) which tak e into accoun t the in-degree (the num b er of incoming transitions at a state), inspired b y [84, 127], – orders (SPen, DPen) whic h take in to account the complexity of the rational function corresp onding to the transition probability . The complexit y is deﬁned b y the degree and num b er of terms of the o ccurring p olynomials. The orders are computed as p enalties for states, and the order prefers states with a lo w p enalty . F or dynamic orderings (Regex, DPen), the p enalties are recomputed as the in-degree of states and complexity of transition probabilities change during state elimination. 21 https://gmplib.org/ 22 https://www.ginac.de/CLN/ 23 this order is destroy ed during the computation of a bisimulation quotien t 64 Sebastian Junges et al. 10.2.2 Gaussian elimination Storm supp orts Eigen [ 76 ] as a linear equation system solv er o ver the ﬁeld of rational functions. It uses the “sup ernodal” (sup ernodes) LU factorisation. The matrix is p ermuted b y the column approximate minimum degree p ermutation (COLAMD) algorithm to reorder the matrix. One adv an tage is that this solver is based on sparse model-chec king algorithm for parameter-free models. The solver therefore, in addition to the prop erties supp orted b y state elimination, supp orts the construction in [15] for conditional probabilities and rew ards. 10.2.3 Set-b ase d tr ansition elimination This elimination metho d is targeted for symbolic representations of the Mark ov c hain. Set-based transition elimination is implemen ted via matrix-matrix multipli- cations. In ev ery m ultiplication, a cop y of the dd-represen tation of a matrix o ver v ariables (  s,  t ) is made. The cop y uses renamed dd-v ariables (  t,  t ′ ). Then, a m ulti- plication of the original matrix with the cop y can b e done on the dd level yielding a matrix (  s,  t ′ ). Renaming  t ′ to  t yields a matrix on the original dd-v ariables. 10.3 Parameter lifting (Re alise d in Storm ) F or parameter lifting (Section 7 and 8), the ma jor eﬀort b eyond calling standard mo del-c hecking pro cedures is the construction of the substituted (lifted) model. As parameter lifting for diﬀeren t regions does not change the topology of the lifted mo del, it is b eneﬁcial to create a template of the lifted model once, and to substitute the v alues according to the region at hand. The substitution operation can be sp ed up b y exploiting the following observ ation: Typically , transition probabilit y functions coincide for many transitions. Thus, we ev aluate each occurring function once and substitute the outcome directly at all o ccurrences. Moreov er, for a gro wing num b er of regions to b e chec k ed, any one-time preprocessing of the lifted mo del ev entually pays oﬀ. In particular, w e apply minimisation techniques b efore construction of the lifted model. W e use b oth bisimulation minimisation as well as state elimination of parameter-free transitions. These minimisations drastically reduce the run-time of chec king a single region. W e use numerical metho ds ﬁrst: for regions that w e w ant to classify as accepting (or rejecting) we resort to the analysis of MDPs using policy iteration with rational n umbers. F or that, w e initialise the p olicy iteration with a guess based on the earlier n umerical results. 10.4 SMT-based region veriﬁcation (Re alise d in PROPhESY ) This complete region c hecking pro cedure is realised b y constructing SMT queries, as elab orated in Section 6. When inv oking the SMT solver, w e use some features of the SMT-lib standard [ 18 ]. First of all, when chec king several regions, we use backtrac k- p oin ts to only partly reset the solver: More precisely , the problem description is giv en by a conjunction of subformulae, where the conjunction is represen ted by a stac k. W e ﬁrst push the constraints for the problem to the stack, sa ve a backtrac k p oin t, and then store the region. Once we hav e chec ked a particular region, we Parameter Synthesis for Marko v Mo dels 65 bac ktrack to the b acktr ack p oint , that is, we remo ve the constrain ts for the particular region from the problem description. This w ay , w e reuse simpliﬁcations and data structures the solv er constructed for the problem description co vering the mo del (and not the region). T o supp ort b oth verifying the prop ert y and its negation, the problem description is slightly extended. W e add tw o Bo olean v ariables ( accepting and rejecting ). The following gives an example of the enco ding together with c hecking whether a region R 1 is accepting, and a region R 2 is rejecting, using the notation of Section 6. x = f D ,φ ∧  accepting = ⇒ x ≥ λ  ∧  rejecting = ⇒ x < λ  ( push ) accepting ∧ Φ ( R 1 ) ( p op ) ( push ) rejecting ∧ Φ ( R 2 ) 10.5 Sampling (R ealise d in PROPhESY ) W e accelerate the selection of regions by getting a rough picture through sampling, as discussed in Section 9. W e support t wo engines for computing the samples: Either via mo del chec king, or by instantiating the solution function. Sampling on the solution function should alwa ys b e done exactly , as the ev aluation of the typically highly-nonlinear solution functions is (again typically) n umerically unstable. In eac h iteration, based on the curren t set of samples, a new set of sampling candidates is computed. The choice of the new samples can b e modiﬁed in sev eral wa ys. The standard used here is via linear interpolation b etw een accepting and rejecting samples. 10.6 Partitioning (Re alise d in PROPhESY ) F or the construction of region candidates, we split the initial regions according to our heuristic (quads or gro wing rectangles, cf. Section 9.2) un til none of the regions is inconsistent. W e sort the candidate regions based on their size in descending order. F urthermore, we prefer regions where we deem v eriﬁcation to b e less costly: Candidate regions that are supp osed to be accepting and are further a wa y from samples or regions that are rejecting are preferred ov er those regions which hav e rejecting samples or regions in their neigh b ourho od. 11 Exp erimental Ev aluation In this section, w e review the scalabilit y of the presen ted approaches based on a selection of b enc hmarks. 66 Sebastian Junges et al. 11.1 Set-up 11.1.1 Benchmarks W e consider ﬁve case studies from the literature. The selection represents v arious application domains. NAND multiplexing. With integrated circuits being built at ever smaller scale, they are more prone to defects and/or to exhibit transien t failures [ 85 ]. One w ay to o vercome these deﬁciencies is the implementation of redundancy at gate-level. In particular, one aims to construct reliable devices from unreliable comp onents. NAND m ultiplexing is such a tec hnique, originally due to von Neumann [ 115 ]. Automated analysis of NAND m ultiplexing via Marko v chain mo del chec king was considered ﬁrst in [ 117 ]. They also studied the inﬂuence of gate failures in either of the stages of the m ultiplexing by sampling v arious v alues. W e use the pMC from [ 64 ], that replaced ﬁxed probabilities in the original formulation with parameters. W e analyse the eﬀect of changing failure probabilities of the gates on the reliability of the m ultiplexed NAND. Herman ’s self-stabilising pr oto c ol. In distributed systems, tokens are used to grant privileges (e.g., access to shared memory) to pro cesses. Randomisation is an essential tec hnique to break the symmetry among sev eral pro cesses [ 7 ]. Herman’s probabilistic algorithm [ 88 ] is a tok en circulation algorithm for ring structures. In each step, ev ery pro cess p ossessing a tok en passes the tok en along with probabilit y p and k eeps the token with probabilit y 1 − p . The algorithm is self-stabilising, i.e., started from any illegal conﬁguration with more than one token the algorithm reco vers to a legal conﬁguration with a unique token. The reco very time crucially dep ends on the probability of passing the tok en, and an optimal v alue for p dep ends on the size of the system [ 105 ]. W e inv estigate the exp ected recov ery time by parameter syn thesis, inspired by [3]. Me an-time-to-failur e of a c omputer system. In reliability engineering, fault trees are a prominen t mo del to describ e ho w a system may fail based on faults of its v arious comp onen ts [ 24 , 125 ]. Dynamic fault trees (DFTs, [ 71 ]) extend these fault trees with a notion of a state, and allow to model spare management and temp oral dep endencies in the failure b ehaviour. State-of-the-art approaches for dynamic fault trees translate such fault trees in to Marko v chains [ 27 , 50 , 140 ]; ev aluation of the mean-time-to-failure boils do wn to the analysis of the underlying Mark o v chain. Probabilities and rewards originate from the failure rate of the comp onents in the describ ed system. Such failure rates are often not known (precisely), esp ecially during design time. Therefore, they may be represented by parameters. W e tak e the HECS DFT [ 138 ] b enc hmark describing the failure of a computer system with an unknown failure rate for the softw are interface and the spare pro cessor, as ﬁrst describ ed in [ 139 ]. W e analyse how this failure rate aﬀects the exp ected time until the failure (mean-time-to-failure) of the complete computer system. Network scheduling. This benchmark [ 143 ] concerns the wireless do wnlink sc heduling of traﬃc to diﬀerent users, with hard deadlines and prioritised pack ets. The system Parameter Synthesis for Marko v Mo dels 67 is time-slotted: time is divided in to perio ds and eac h p erio d is divided into an equal n umber of slots. At the start of each time p erio d, a new pack et is generated for each user with a randomly assigned priority . The goal of sc heduling is to, in eac h perio d, deliv er the pack ets to each user b efore the p erio d ends. Pac k ets not delivered b y the end of a p erio d are dropp ed. Sc heduling is non-trivial, as successful transmissions are not sto chastically indep enden t, i.e., channels hav e a (hidden) internal state. The system is describ ed as a partially observ able Marko v decision pro cess [ 126 ], a prominen t formalism in the AI communit y . W e take the Net wo rk mo del from [ 118 ], and consider the pMC that describ es randomised ﬁnite memory con trollers that solv e this scheduling problem, based on a translation from [ 96 ]. Concretely , the parameters represen t how the ﬁnite memory controller randomises. W e ev aluate the eﬀect of the randomisation in the sc heduling on the exp ected pac ket loss. Bounde d r etransmission pr oto c ol. The b ounded retransmission protocol ( BRP , [ 61 , 87 ]) is a v arian t of the alternating bit proto col. It can be used as part of an OSI data link la yer, to implement retransmitting corrupted ﬁle ch unks b etw een a sender and a receiver. The system con tains tw o channels; from sender to receiver and vice v ersa. BRP is a famous benchmark in (non-parametric) probabilistic mo del chec king, based on a mo del in [ 62 ]. W e consider the parametric version from [ 78 ]. The parameters naturally reﬂect the channel qualities. The mo del contains non-determinism as the arriv al of ﬁles on the link lay er cannot b e inﬂuenced. This non-determinism hamp ers a man ual analysis. The combination of parametric probabilities and non- determinism naturally yields a pMDP . W e analyse the maxim um probability that a sender ev entually do es not rep ort a successful transmission. R emark 12 Other benchmarks and a thorough performance ev aluation hav e b een presen ted b efore in [ 64 ] (for state elimination and parameter space partitioning) and [124] (for parameter lifting). 11.1.2 Benchmark statistics T able 1 on the next page summarises relev ant information about the concrete instances that we to ok from the b enc hmarks. The id is used for reference. The b enchmark refers to the name of the b enchmark-set, while the instanc e describ es the particular instance from this b enchmark set. W e give the total num b er of p ar ameters | V | b oth in the transition matrix as well as in the reward structure whenev er applicable. F or the remainder of the columns, we give tw o num b ers p er benchmark instance: The upper row describes the original mo del, the latter describ es the (strong) bisimulation quotient. The columns give the n umber of states and tr ansitions . The last row giv es the time (in seconds) required for constructing the mo del (top) and constructing the bisimulation quotient (b ottom). W e remark that all b enc hmarks hav e a limited num b er of parameters: Systems with many parameters are b eyond the reac h of the metho ds discussed here, but can b e analysed with resp ect to simpler syn thesis questions (such as ﬁnding one suitable instan tiation). W e refer to the related work for a discussion of such methods. 11.1.3 Evaluation W e conducted the empirical ev aluation on an HP BL685C G7 with Debian 9.6. Eac h ev aluation run could use 8 cores with 2.1GHz each. How ever, unless sp eciﬁed 68 Sebastian Junges et al. T able 1 Detailed information for mo dels in the b enchmark set id benchmark instance | V | states transitions time 1 BRP MAX=2,N=16 2 1439 1908 0.06 664 928 0.22 2 MAX=2,N=256 20639 27348 0.57 10264 14368 370.83 3 MAX=2,N=512 41119 54484 1.11 20504 28704 197.69 4 MAX=5,N=16 2801 3783 0.10 1354 1912 1.23 5 MAX=5,N=256 40721 55143 1.15 21034 29752 3305.07 6 MAX=5,N=512 81169 109927 2.25 42026 59448 345.21 7 HECS m=1,k=1,i=1 2 129 489 0.02 25 71 0.00 8 m=1,k=1,i=2 145 589 0.02 49 173 0.00 9 Herman N=3 1 9 36 0.02 3 5 0.00 10 N=5 33 276 0.03 5 15 0.00 11 N=7 129 2316 0.11 16 137 0.02 12 N=9 513 20196 0.92 347 15009 0.12 13 NAND K=2,N=2 2 178 243 0.03 125 167 0.00 14 K=2,N=20 154942 239832 2.81 102012 154722 0.91 15 K=2,N=30 681362 1065797 12.56 474847 732768 4.65 16 K=5,N=10 35112 52647 0.63 23603 34093 0.21 17 K=5,N=20 384772 594792 7.04 288102 436332 3.17 18 K=5,N=30 1697732 2653937 31.45 1345507 2074758 18.49 19 Netw ork c=2,K=2,T=2 8 52 133 0.00 52 133 0.00 20 c=2,K=2,T=3 16 106 269 0.01 106 269 0.00 21 c=2,K=2,T=4 24 164 411 0.01 164 411 0.00 22 c=2,K=4,T=2 20 136 365 0.01 136 365 0.00 23 c=2,K=4,T=3 36 262 691 0.01 262 691 0.00 24 c=2,K=4,T=4 52 392 1023 0.01 392 1023 0.00 otherwise, algorithms use a single core. W e set the timeout to 1 hour and the memory limit to 16GB. W e used PROPhESY v ersion 2.0, together with the Storm - p ython bindings version 1.3.1, z3 v ersion 4.8.4. All b enchmark ﬁles are made a v ailable via PROPhESY 24 . 24 Benchmarks are in the subfolder benchmark files Parameter Synthesis for Marko v Mo dels 69 T able 2 Empirical p erformance of computing the solution function id degree degree # terms # terms success time time num denom num denom mc total 7 23 24 234 247 16 2.00 2.09 0.64 0.72 8 31 32 408 425 16 9.12 9.21 3.00 3.08 9 0 2 1 2 18 0.00 0.09 0.00 0.08 10 4 6 5 6 17 0.04 1.55 0.00 0.10 11 28 30 29 30 11 0.62 0.82 0.37 0.56 12 150 152 151 152 8 247.00 248.14 114.49 115.64 13 10 0 32 1 18 0.00 0.11 0.00 0.09 14 100 0 2106 1 15 43.05 46.88 15.46 19.35 15 150 0 4653 1 13 469.29 486.74 110.54 128.48 16 110 0 1220 1 15 6.30 7.24 3.30 4.25 17 200 0 4640 1 13 245.47 256.05 88.18 98.71 18 330 0 10260 1 1 3031.34 3083.88 3031.34 3083.88 19 1 0 23 1 16 0.00 0.07 0.00 0.06 20 1 0 111 1 16 0.01 0.08 0.01 0.07 21 1 0 519 1 16 0.04 0.11 0.03 0.09 22 1 0 65 1 16 0.01 0.08 0.01 0.08 23 1 0 289 1 16 0.07 0.15 0.03 0.10 24 1 0 1377 1 16 0.40 0.48 0.12 0.20 11.2 Exact synthesis via the solution function T o ev aluate the exact synthesis approach, we use state elimination with 7 dif- feren t heuristics, set-based transition elimination, and Gaussian elimination. All conﬁgurations are ev aluated with and without strong bisim ulation. First, w e show the sizes of the solution function: The results are summarised in T able 2. The id references the corresponding b enchmark instance in T able 1 on the previous page. The BRP pMDP is not included. The set of all strategies preven ts the computation of the solution function for all induced pMCs. The next four columns displa y properties of the resulting rational function. W e giv e the degree of b oth the n umerator ( de gr e e num ) and denominator ( de gr e e denom ), as well as the num b er of terms in b oth p olynomials ( # terms num , # terms denom ). The next column giv es the n umber of conﬁgurations (out of the 18) whic h suc c essful ly ﬁnished within the time limit. The last tw o columns indicate timings. W e give the times (in seconds) to compute the solution function ( time mc ) and the total time including mo del building, (optional) bisimulation minimisation and computing the solution function. F or these timings we giv e t w o num b ers p er b enchmark instance: The upper row describ es the median v alue o ver all successful conﬁgurations and the lo w er ro w describ es the best result obtained. Th us, while functions often gr ow 70 Sebastian Junges et al. 10 0 10 1 10 2 10 3 5 10 15 No. b enc hmark Time (in seconds) Set-based (bisim) Gaussian (bisim) DPen (bisim) FwRev (bisim) Regex (bisim) SPen (bisim) SPen Fig. 19 Cumulativ e solving times for solution function computation pr ohibitively lar ge, me dium-size d functions c an stil l b e c ompute d. Con trary to model c hecking for parameter-free mo dels, mo del building is typically not the bottleneck. F urthermore, we see that the selected heuristic is indeed crucial. Consider instance 11: 11 heuristics successfully compute the solution function (and most of them within a second). How ever, 7 others yield a timeout. That leads us to compare some heuristics in Figure 19. The plot depicts the cumulativ e solving times for selected conﬁgurations ov er all 18 b enc hmark instances (excluding BRP ). Gaussian and set-based refer to these approac hes, resp ectively , all other conﬁgurations are v ariants of state elimination, cf. Section 10.2.1, (bisim) denotes that bisimulation minimisation is used. The x-axis represen ts the num b er of solved instances and the (logarithmic) y-axis represen ts the time in seconds. A p oin t ( x, y ) in the plot represen ts the x fastest instances whic h could be solved within a total time of y seconds. F or 15 instances, one of the depicted conﬁgurations w as the fastest ov erall. Regex based conﬁgurations were the fastest eight times, DP en based ones four times and three times conﬁgurations based on FwRev were fastest. F rom these n umbers, w e conclude that the sele ction of the heuristic is essential, and dep ending on the mo del to b e analyse d. F rom the graph, we further observe that although using a Gaussian elimination yields go o d p erformance, state-elimination approaches can (signiﬁcan tly) outp erform the Gaussian elimination on some b enchmarks. The DP en solv es all instances (the only conﬁguration to do so), but Regex is o verall (sligh tly) faster. The uninformed FwRev with bisim ulation works surprisingly well for these b enc hmarks (but that is mostly coincidence). The set-based elimination is clearly inferior on the b enchmarks considered here, but allows to analyse some mo dels with a very regular structure and a gigantic state space, e.g., a parametric Marko v c hain for the analysis of the blueto oth protocol [70]. 11.3 Three types of region veriﬁcation W e ev aluate region veriﬁcation using t wo SMT-based approac hes (SF: based on ﬁrst computing the Solution F unction, or ETR: enco ding the equations into Existen tial Theory of the Reals), and PLA. In particular, we present some results for the Herman b enchmark: it features a single parameter, and therefore is well-suited for the illustration of some concepts. W e visualised the results for instance 11 in Figure 20 on the following page. The x-axis represents the probabilit y p and the Parameter Synthesis for Marko v Mo dels 71 0 0 . 2 0 . 4 0 . 6 0 . 8 1 4 6 8 5 7 p Exp ected recov ery time Fig. 20 Plot for Herman mo del with seven pro cesses and parameter p (Benchmark Id: 11) T able 3 Empirical p erformance of region veriﬁcation algorithms. id λ region techn. result time 11 5 [0.20, 0.27] ETR inconsistent 12,11 PLA unknown 0.01 SF unknown TO [0.27, 0.28] ETR reject 20.68 PLA reject 0.01 SF unknown TO [0.28, 0.35] ETR reject 53.47 PLA unknown 0.01 SF unknown TO [0.35, 0.50] ETR reject 23.41 PLA reject 0.00 SF unknown TO [0.54, 0.55] ETR reject 22.35 PLA reject 0.01 SF unknown TO [0.80, 0.90] ETR unknown TO PLA accept 0.01 SF unknown TO 13 0.3 [0.01, 0.99] × [0.70, 0.90] ETR accept 16.20 PLA unknown 0.01 SF accept 0.16 [0.01, 0.99] × [0.90, 0.99] ETR inconsistent 19.41 PLA unknown 0.01 SF inconsistent 0.04 [0.01, 0.50] × [0.65, 0.70] ETR accept 45.61 PLA unknown 0.01 SF accept 0.13 [0.01, 0.50] × [0.75, 0.90] ETR accept 4.58 PLA accept 0.01 SF accept 0.12 [0.01, 0.99] × [0.40, 0.50] ETR reject 19.82 PLA reject 0.00 SF reject 0.08 y-axis the expected recov ery time. W e indicate the solution function in blue. The threshold in the following is set to λ = 5 and indicated by the orange horizontal line. The black columns depict six diﬀerent regions 25 that are ev aluated with region c hecking. F or eac h region we w ant to v erify whether the exp ected reco very time is at least 5. The results are summarised in (the upp er part of ) T able 3. The ﬁrst 25 Strictly speaking, regions are giv en b y the in terv als for the parameter, w e depict the columns for better visibility . 72 Sebastian Junges et al. 0 0 . 2 0 . 4 0 . 6 0 . 8 1 0 0 . 5 1 0 0 . 5 1 prob1 p err Fig. 21 Plotting the solution function for NAND K = 2 , N = 2 (Benchmark Id: 13) and parameters prob1 and perr. column id references the b enchmark instance and the second column gives the threshold λ . The next columns indicate the considered re gion and the te chnique . The last columns giv e the r esult of the region v eriﬁcation and the time (in seconds) needed for the computation. The timeout (TO) w as set to 120 seconds. F or b enchmark instance 11, P arameter lifting (PLA) computes a result within milliseconds and the computation time is indep enden t of the considered region. The SMT-based tec hniques tak e longer and the SF technique in particular do es not terminate within t wo min utes. How ever, the ETR technique could yield a result for region [0 . 28 , 0 . 35] whereas PLA could not give a conclusive answ er due to its inheren t ov er-approximation. W e no w consider the region v eriﬁcation on the NAND mo del with tw o parameters. W e visualised the solution function for instance 13 in Figure 21. The considered threshold is λ = 0 . 3. Green coloured parts indicate parameter instantiations leading to probabilities abov e λ and red parts lie b elow λ . The results of the veriﬁcation for diﬀeren t regions are given in (the low er part of ) T able 3 on the previous page. PLA is again the fastest technique, but for larger regions close to the threshold PLA can often not provide a conclusiv e answer. Con trary to b efore, SF is sup erior to ETR. The p erformance of the SMT-based techniques (again) greatly dep ends on the considered region. It is only natural that the size of the r e gion, and the diﬀer ence to the threshold have a signiﬁc ant inﬂuenc e on the p erformanc e of re gion veriﬁc ation . These observ ations are general and do hold on al l other b enchmarks . F urthermore, parameter lifting seems broadly applicable, and in the setting ev aluated here, clearly faster than SMT-based approac hes. Parameter lifting ov er-approximates and therefore migh t only giv e a decisive result in a reﬁnement lo op such as parameter space partitioning. The SMT-based approaches are a v aluable fallbac k. When relying on the SMT techniques, it is hea vily mo del-dep endent whic h p erforms better. T able 4 on page 75 at the end of the next section giv es some additional results, indicating the p erformance of the diﬀeren t v eriﬁcation techniques. Parameter Synthesis for Marko v Mo dels 73 0 0 . 2 0 . 4 0 . 6 0 . 8 1 0 2 4 6 8 5 p Exp ected recov ery time Fig. 22 Parameter space partitioning for Herman N = 5 (Benc hmark Id: 10) with parameter p 11.4 Approximativ e synthesis via parameter space partitioning W e no w ev aluate the parameter space partitioning. W e use the implementation in PROPhESY with the three veriﬁcation pro cedures ev aluated ab o ve. Therefore, w e fo cus here on the actual parameter space partitioning. First, consider again Herman for illustration purp oses. Region v eriﬁcation is not applicable for instance 10 (with threshold 5), as neither all instan tiations accept nor all reject the speciﬁcation. Instead, parameter space partitioning deliv ers whic h of these instantiations accept, and which reject the sp eciﬁcation. The resulting parameter space partitioning is visualised in Figure 22. Next, we compare the three veriﬁcation tec hniques—each with tw o diﬀerent metho ds for selecting candidate regions–in Figure 23 on the next page. Figure 23(a) on the next page depicts the computation on the Herman mo del with 5 pro cesses and threshold λ = 5. The plot depicts the cov ered area for all three tec hniques with b oth quads (straight lines) and rectangles (dashed lines) as regions. The x-axis represen ts the computation time (in seconds) on a logarithmic scale and the y-axis represen ts the percentage of cov ered area. A p oint ( x, y ) in the plot represents y p ercen t of the parameter space whic h could b e cov ered within x seconds. F or Herman , SMT-based techniques perform b etter than PLA. PLA was able to co ver 64% of the parameter space within milliseconds. Ho wev er, in the remaining hour only 2% more space w as co v ered. The SMT-based tec hniques w ere able to co ver at least 99% of the parameter space within 15 seconds. Moreov er, the rectangles co ver the parameter space faster than quads. W e also p erform the parameter space partitioning on the NAND model with tw o diﬀerent thresholds: W e compare the parameter space partitioning tec hniques for threshold λ = 0 . 1 in Figure 23(b) on the next page, and for threshold λ = 0 . 3 in Figure 23(c) on the next page. F or NAND , the PLA technique performs b etter than the SMT-based techniques. F or threshold λ = 0 . 1, PLA could co ver at least 99% of the parameter space within 1 second. The main reason is that the border is in a corner of the parameter space. Additionally , the SMT-based techniques with rectangles are signiﬁcantly faster than the quads for this threshold. F or threshold λ = 0 . 3, more region v eriﬁcation steps were necessary . PLA still outp erforms ETR and SF. How ever, the use of rectangles ov er quads do es not lead to a b etter p erformance for this threshold. A t an y p oint in time, there can b e very signiﬁcant diﬀerences betw een the heuristics 74 Sebastian Junges et al. 1 60 600 3600 0 0 . 2 0 . 4 0 . 6 0 . 8 1 Time (in seconds) Cov ered area ETR PLA SF (a) Herman , N = 5, with λ = 5 1 60 600 3600 0 0 . 2 0 . 4 0 . 6 0 . 8 1 Time (in seconds) Cov ered area ETR PLA SF (b) NAND K = 2 , N = 2 with λ = 0 . 1 1 60 600 3600 0 0 . 2 0 . 4 0 . 6 0 . 8 1 Time (in seconds) Cov ered area ETR PLA SF (c) NAND K = 2 , N = 2 with λ = 0 . 3 Fig. 23 Cov ered areas for parameter space partitioning on diﬀerent models and thresholds for candidate generation, esp ecially in settings where single region veriﬁcation calls b ecome expensive. Finally , we summarise an o verview of the p erformance in T able 4 on the following page. F or brevity , w e pruned some rows, especially if the present approac hes already struggle with smaller instances. The id is a reference to the benchmark instance. The te chnique is giv en in the next column. In the next three columns w e giv e for each tec hnique the time (in seconds) needed to co ver at least 50%, 90% and 98% of the complete parameter space. The next t wo columns giv e the complete c over e d ar e a —i.e. the sum of the sizes of all accepting or rejecting regions— when terminating the parameter space partitioning after 1h, together with the safe ar e a , i.e. the sum of the sizes of all accepting regions. The last tw o columns indicate the p ercentage of the total time sp ent in generating the regions ( time r e g gen ) and v erifying the regions ( time analysis ). PLA is almost always sup erior, but not on al l b enchmarks (and not on al l (sub)r e gions. Depending on the mo del, SF or ETR are the b est SMT-based technique. There might b e ro om for improv ement by p ortfolios and mac hine-learned algorithm selection schemes. 12 Related W ork and discussion W e discuss related work with resp ect to v arious relev ant topics. Complexity. F or graph-preserving pMCs, many complexit y results are collected in [ 97 ], including results from [ 45 ]. In particular, the complemen t of the veriﬁcation Parameter Synthesis for Marko v Mo dels 75 T able 4 Empirical p erformance of parameter space partitioning v ariations. id tec hn. time time time area area p ercent percent 50% 90% 98% cov safe reg gen analysis 1 ETR — — — 0.20 0.20 0.00 % 99.82 % PLA 0.04 0.19 3.09 0.99 0.83 31.65 % 12.36 % SF — — — 0.00 0.00 — — 2 ETR — — — 0.00 0.00 0.00 % 89.50 % PLA 0.33 0.34 — 0.97 0.97 0.00 % 81.18 % SF — — — 0.00 0.00 — — 4 ETR — — — 0.03 0.03 0.00 % 99.61 % PLA 0.06 0.30 7.86 0.99 0.63 38.45 % 10.83 % SF — — — 0.00 0.00 — — 5 ETR — — — 0.00 0.00 0.00 % 9.73 % PLA 0.70 — — 0.87 0.87 0.00 % 7.72 % SF — — — 0.00 0.00 — — 7 ETR — — — 0.47 0.00 0.00 % 99.64 % PLA 0.00 0.00 0.00 1.00 0.00 0.00 % 0.00 % SF — — — 0.00 0.00 — — 8 ETR — — — 0.00 0.00 0.00 % 99.77 % PLA 0.01 0.01 0.01 1.00 1.00 0.00 % 0.91 % SF — — — 0.00 0.00 — — 9 ETR 0.02 30.19 70.41 0.99 0.05 0.00 % 98.89 % PLA 0.08 — — 0.55 0.06 0.15 % 73.89 % SF 0.02 0.09 0.23 0.99 0.05 0.00 % 18.60 % 10 ETR 0.12 0.45 1.29 0.99 0.16 0.00 % 57.09 % PLA 0.03 — — 0.66 0.17 0.15 % 74.84 % SF 0.24 1.20 11.30 0.99 0.16 0.00 % 90.63 % 12 ETR — — — 0.00 0.00 0.00 % 99.66 % PLA 1.75 — — 0.56 0.43 0.15 % 75.39 % SF — — — 0.00 0.00 0.00 % 96.20 % 13 ETR — — — 0.28 0.28 0.00 % 99.80 % PLA 0.05 0.09 0.49 0.99 0.98 8.22 % 15.53 % SF 28.70 202.98 357.90 0.98 0.98 0.00 % 96.26 % 14 ETR — — — 0.00 0.00 0.00 % 85.15 % PLA 3.08 16.08 152.36 0.99 0.15 32.01 % 47.66 % SF — — — 0.00 0.00 0.00 % 98.68 % 15 ETR — — — 0.00 0.00 — — PLA 20.27 91.18 854.48 0.99 0.14 30.27 % 61.95 % SF — — — 0.00 0.00 0.00 % 92.56 % 16 ETR — — — 0.00 0.00 0.00 % 98.95 % PLA 0.55 4.65 55.99 0.99 0.19 33.04 % 25.99 % SF — — — 0.00 0.00 0.00 % 99.42 % 17 ETR — — — 0.00 0.00 0.00 % 18.75 % PLA 8.79 40.99 326.12 0.99 0.16 33.23 % 54.62 % SF — — — 0.00 0.00 0.00 % 94.39 % 18 ETR — — — 0.00 0.00 — — PLA 53.69 254.13 1861.31 0.99 0.16 33.21 % 60.37 % SF — — — 0.00 0.00 — — 19 ETR — — — 0.00 0.00 0.00 % 99.28 % PLA — — — 0.12 0.12 0.00 % 99.54 % SF — — — 0.32 0.32 0.00 % 98.22 % problem, i.e., the question whether there exists an instan tiation in a region that satisﬁes a reac hability property , is ETR-complete for b oth pMDPs and pMCs 26 . F or any ﬁxed num b er of parameters, the problem can b e solved in p olynomial time [17]. This pap er also considers a ric her fragment of the logic PCTL. Computing a solution function. This approach was pioneered b y [ 63 ] and signiﬁcan tly impro ved by [ 78 ]. Both PRISM [ 102 ] and PARAM [ 77 ] supp ort the computation of a solution function based on the latter metho d. It has b een adapted in [ 91 ] to 26 It holds that P ⊂ ETR ⊆ PSP ACE. A prominent ETR-complete problem is whether a multiv ariate p olynomial has a real-v alued ro ot. 76 Sebastian Junges et al. an elimination of SCCs and a more clev er representation of rational functions. This represen tation has b een adapted by Storm [ 65 ]. In [ 72 ], computing a solution function via a computer algebra system was considered. That method targets small, randomly generated pMCs with man y parameters. Recently , [ 17 ] explored the use of one-step fraction-free Gaussian elimination to reduce the num b er of GCD computations. F or pMDPs, [ 79 ] exp erimented with the introduction of discrete parameters to reﬂect strategy choices—this method, how ever, scales po orly . In [ 66 ] and [ 67 ], v ariants of v alue iteration with a dd-based representation of the solution function are presen ted. F ast sampling on (concise representations of ) the solution function is considered in [73, 89]. Equation system formulation. Regarding pMDPs, instead of in tro ducing a Bo olean structure, one can lift the linear program form ulation for MDPs to a nonlinear program (NLP). This lifting has b een explored in [ 20 ], and shown to b e not feasible in general. A string of results rely on conv ex programming approac hes. F or instance, although the general NLP do es not lie in the class of conv ex problems, a v ariety of veriﬁcation related problems can be expressed by a sequence of geometric programs, which is exploited in [ 56 ]. Alternatively , ﬁnding satisfying parameter instan tiations in pMDPs under demonic non-determinism and with aﬃne transition probabilities can be approached b y iteratively solving a conv ex-concav e program that appro ximates the original NLP [57]. A comprehensiv e ov erview of exploiting con vex programming is presen ted in [ 60 ]. Alternatively , more eﬃcien t solvers can b e used [ 42 ] for subclasses of pMDPs. An alternativ e parametric mo del with a ﬁnite set of parameter instan tiations, but without the assumption that these instan tiations are graph preserving is considered in [41]. Mo del r ep air. The problem of mo del repair is related to parameter synthesis. In particular, for a Marko v mo del and a refuted sp eciﬁcation the problem is to transform the mo del such that the sp eciﬁcation is satisﬁed. In the sp ecial case where repair amounts to c hanging transition probabilities, the underlying mo del is parametric as in this pap er: the parameters are addive factors to b e added to the original transition probabilities. The problem w as ﬁrst deﬁned and solved either by a nonlinear program or parameter synth esis in [ 20 ]. A greedy approac h was given in [ 119 ] and eﬃcien t sim ulation-based metho ds are presen ted in [ 43 ]. In addition, parametric mo dels are used to rank patc hes in the repair of softw are [107]. Interval Markov chains. Instead of parametric transitions, interv al MCs or MDPs feature in terv als at their transitions [ 10 , 74 , 92 , 141 ]. These models do not allo w for parameter dep endencies, but v eriﬁcation is necessarily “robust” against all probabilities within the in terv als, see for instance [ 122 ], where con vex optimization is utilised, and [ 81 , 82 ], where eﬃcient v eriﬁcation of multiple-ob jectives is in tro duced. In [6, 19], these mo dels are extended to so-called parametric in terv al MCs, where in terv al b ounds themselv es are parametric. Extensions to richer models suc h as partially observ able MDPs are considered in [59, 134]. Derivatives and monotonicity. Many systems b eha v e monotonically in some of their system parameters. F or example, most netw ork protocols b ecome more reliable if the comm unication channel reliability increases. If the solution function is monotonic, Parameter Synthesis for Marko v Mo dels 77 then parameter space partitioning can b e accelerated [ 131 ]. Assessing monotonicity can be tightly in tegrated in a loop that uses parameter lifting [ 132 ]. Finally , the deriv ative of the solution function can b e used for gradien t descent whenev er the goal is to ﬁnd a coun terexample for region v eriﬁcation [86]. Sensitivity analysis. Besides analysing in which regions the system b ehav es correctly w. r. t. the sp eciﬁcation, it is often desirable to p erform a sensitivity analysis [ 44 , 133 ], i. e., to determine in whic h regions of the parameter space a small perturbation of the system leads to a relativ ely large change in the considered measure. In our setting, such an analysis can b e conducted with little additional eﬀort. Given a rational function for a measure of interest, its deriv ations w. r. t. all parameters can b e easily computed. P assing the deriv ations with user-sp eciﬁed thresholds to the SMT solver then allo ws for ﬁnding parameter regions in whic h the system b ehav es robustly . Adding the safety constrain ts describ ed earlier, the SMT solv er can ﬁnd regions that are b oth safe and robust. Par ameters with distributions. Rather than a mo del in whic h the parameter v alues are c hosen from a set, they can be equipp ed with a distribution. The v eriﬁcation outcome consists then of conﬁdence in terv als rather than absolute guaran tees. In [ 111 ], simulation based methods are used, whereas [ 33 , 34 ] use statistical metho ds on a solution function. pMDPs with a distribution ov er the parameters are considered in [ 9 ]. Sampling-based metho ds that rely on the so-called scenario-approach [ 36 , 37 ] are presen ted in [11, 58]. Ensuring gr aph pr eservation. Chec king graph-preserv ation is closely related to chec k- ing whether a well-deﬁned p oin t instantiation exists, which has an exp onential run time in the num b er of parameters [ 106 ]. F or parametric interv al Marko v chains, the question whether there exists a w ell-deﬁned instantiation is referred to as c onsistency and receiv ed attention in [6, 120]. R obust str ate gies. Robust strategies for pMDPs, as mentioned in Remark 7 on page 22, are considered in, among others, [ 108 , 141 ]. These and other v ariants of syn thesis problems on pMDPs were compared in [ 8 ]. A v ariant where parameters are not non-deterministically chosen, but go verned b y a prior ov er these parameters, has recently b een considered [ 9 ]. In [ 121 ], data-driven b ounds on parameter ranges are obtained, and prop erties are v alidated using parameter syn thesis tec hniques. Continuous time. Parametric CTMCs w ere ﬁrst considered b y [ 83 ]. A method using relaxations similarly to parameter lifting has b een prop osed in [ 30 ]. The method w as improv ed in [ 39 ] and implemented in PRISM-PSY [ 40 ]. A combination with sampling-based algorithms to ﬁnd go o d parameter instantiations is explored in [ 35 ]. Parameter synthesis with statistical guaran tees has b een explored in [ 25 , 26 ]. Moreo ver, a sampling-based approach for so-called uncertain parametric CTMCs that hav e a distribution ov er the parameter v alues obtains statistical guaran tees on reac hability probabilities [ 12 ]. Finally , in [ 75 ], ﬁnding go od parameter instantiations is considered b y iden tifying subsets of parameters that ha v e a strictly p ositiv e or negativ e inﬂuence on the prop ert y at hand. 78 Sebastian Junges et al. Conne ction to other mo dels. F urthermore, [ 96 ] establishes connections to the com- putation of strategies in partially observ able MDPs [ 126 ], a prominen t mo del in AI. In [ 142 ], the connection to concurren t sto chastic games is sho wn. pMCs can b e used to accelerate solving hierarchical Mark ov models [ 95 , 114 ] and for parameter syn thesis in Ba yesian net works [ 128 ]. Finally , in [ 53 ], a method that main tains a b elief o ver parameter v alues is introduced in a rob otics context. 13 Conclusion and F uture W ork This pap er gives an extensive account of parameter synthesis for discrete-time Mark ov chain mo dels. In particular, we considered three diﬀerent v arian ts of parameter synthesis questions. F or eac h problem v ariant, w e give an account of the av ailable algorithms from the literature, together with sev eral extensions from our side. All algorithms are a v ailable in the op en-source to ol PROPhESY . F utur e work F uture work in v arious directions is p ossible. Many of the results here can b e p orted to the more general setting of weigh ted automata ov er the adequate semiring [ 69 ], which can b e interesting from a theoretical p ersp ective. Algorithmically , w e would like to develop metho ds which identify and exploit structural prop erties that are common to standard b enchmarks for Mark ov c hains and Mark ov decision pro cesses. First steps in this direction hav e b een taken, e.g., b y exploiting monotonicity [ 131 ]. While graph-preserv ation is common in many applications, this restriction is not alwa ys natural. The decomposition presented in this paper yields an exp onential blow-up in the n umber of parameters that we w ould lik e to av oid whenev er p ossible. How ever, algorithms that do not rely on graph- preserv ation hav e not yet b een integrated. The tec hniques to cov er the parameter space by sets of smaller and easy-to-verify regions are still rather naive: This is true b oth for region veriﬁcation, where w e split due to the approximation, and for parameter space partitioning. The abov e men tioned monotonicity is one p ossibility to accelerate the wa y we split. In general, we plan to exploit parametric mo dels in a data-driv en con text, where the structure pro vided b y parameter dependencies can b e exploited to accelerate learning of probabilistic models [135, 136]. Ac knowledgemen ts The authors would lik e to thank Harold Bruin tjes and Florian Corzilius for their contributions to PROPhESY 1.0, T om Janson and Lutz Klink enberg for their help in developing PROPhESY 2.0, Gereon Kremer as a long-term main tainer of carl and the anonymous reviewers for their thorough feedback. References 1. (1999) IEEE wireless LAN Medium Access Con trol (MA C) and Ph ysical La y er (PHY) sp eciﬁcation 2. Abb ott J, Bigatti AM (2022) CoCoALib: a c++ library for doing Computations in Comm utative Algebra. Av ailable at http://cocoa.dima.unige.it/cocoalib 3. Aﬂaki S, V olk M, Bonakdarpour B, Kato en JP , Storjohann A (2017) Auto- mated ﬁne tuning of probabilistic self-stabilizing algorithms. In: SRDS, IEEE Computer So ciet y , pp 94–103 Parameter Synthesis for Marko v Mo dels 79 4. Amparore EG, Beccuti M, Donatelli S (2014) (Sto c hastic) mo del chec king in GreatSPN. In: P etri Nets, Springer, LNCS, vol 8489, pp 354–363 5. Ando v a S, Hermanns H, Kato en JP (2003) Discrete-time rew ards mo del- c heck ed. In: FORMA TS, Springer, LNCS, v ol 2791, pp 88–104 6. Andr ´ e ´ E, Delahay e B (2016) Consistency in parametric interv al probabilistic timed automata. In: TIME, IEEE Computer Society , pp 110–119 7. Angluin D (1980) Lo cal and global properties in netw orks of processors (extended abstract). In: STOC, A CM, pp 82–93 8. Arming S, Barto cci E, Sokolo v a A (2017) SEA-P ARAM: exploring schedulers in parametric MDPs. In: QAPL@ET APS, EPTCS, vol 250, pp 25–38 9. Arming S, Bartocci E, Chatterjee K, Kato en JP , Sok olov a A (2018) P arameter- indep enden t strategies for pMDPs via POMDPs. In: QEST, Springer, LNCS, v ol 11024, pp 53–70 10. Bacci G, Delahay e B, Larsen KG, Mariegaard A (2021) Quantitativ e analysis of interv al mark ov c hains. In: Mo del Chec king, Syn thesis, and Learning, Springer, LNCS, v ol 13030, pp 57–77 11. Badings TS, Cubuktepe M, Jansen N, Junges S, Kato en J, T op cu U (2022) Scenario-based v eriﬁcation of uncertain parametric mdps. Int J Softw T o ols T echnol T ransf 24(5):803–819 12. Badings TS, Jansen N, Junges S, Stoelinga M, V olk M (2022) Sampling-based v eriﬁcation of ctmcs with uncertain rates. In: CA V (2), Springer, LNCS, vol 13372, pp 26–47 13. Baier C, Kato en JP (2008) Principles of Mo del Chec king. MIT Press 14. Baier C, Clarke EM, Hartonas-Garmhausen V, Kwiatko wsk a MZ, Ry an M (1997) Sym b olic mo del c hecking for probabilistic pro cesses. In: ICALP, Springer, LNCS, v ol 1256, pp 430–440 15. Baier C, Klein J, Kl ¨ upp elholz S, M¨ arc k er S (2014) Computing conditional probabilities in Marko vian models eﬃciently . In: T ACAS, Springer, LNCS, v ol 8413, pp 515–530 16. Baier C, de Alfaro L, F orejt V, Kwiatko wsk a M (2018) Mo del c hecking probabilistic systems. In: Handb o ok of Mo del Checking, Springer, pp 963–999 17. Baier C, Hensel C, Hutschenreiter L, Junges S, Kato en J, Klein J (2020) P arametric mark ov c hains: PCTL complexit y and fraction-free gaussian elimi- nation. Inf Comput 272:104,504 18. Barrett C, F on taine P , Tinelli C (2016) The Satisﬁability Modulo Theories Library (SMT-LIB). www.SMT-LIB.org 19. Bart A, Delaha ye B, F ournier P , Lime D, Monfroy E, T ruc het C (2018) Reac hability in parametric interv al Marko v chains using constraints. Theor Comput Sci 747:48–74 20. Barto cci E, Grosu R, Katsaros P , Ramakrishnan C, Smolk a SA (2011) Mo del repair for probabilistic systems. In: T ACAS, LNCS, v ol 6605, Springer, pp 326–340 21. Basu S, P ollac k R, Ro y MF (2006) Algorithms in Real Algebraic Geometry (Algorithms and Computation in Mathematics). Springer-V erlag New Y ork 22. Bauer C, F rink A, Kreck el R (2002) Introduction to the ginac framework for sym b olic computation within the C++ programming language. J Symb Comput 33(1):1–12 23. Biere A, Heule M, v an Maaren H, W alsh T (eds) (2009) Handb o ok of Sat- isﬁabilit y , F rontiers in Artiﬁcial Intelligence and Applications, vol 185, IOS 80 Sebastian Junges et al. Press 24. Bobbio A, T rivedi KS (2017) Reliability and Av ailabilit y Engineering: Mo del- ing, Analysis, and Applications. Cam bridge Universit y Press 25. Bortolussi L, Silvetti S (2018) Bay esian statistical parameter synthesis for linear temp oral prop erties of sto chastic mo dels. In: T ACAS (2), Springer, LNCS, v ol 10806, pp 396–413 26. Bortolussi L, Milios D, Sanguinetti G (2016) Smoothed mo del c hecking for uncertain con tinuous-time Marko v chains. Inf Comput 247:235–253 27. Boudali H, Crouzen P , Stoelinga M (2010) A rigorous, comp ositional, and extensible framework for dynamic fault tree analysis. IEEE T rans Dep endable Sec Comput 7(2):128–143 28. Bozzano M, Villaﬁorita A (2010) Design and Safet y Assessmen t of Critical Systems. CR C Press 29. Bozzano M, Cimatti A, Kato en JP , Katsaros P , Mokos K, Nguy en VY, Noll T, Postma B, Ro veri M (2014) Spacecraft early design v alidation using formal metho ds. Rel Eng & Sys Safet y 132:20–35 30. Brim L, Cesk a M, Drazan S, Safr´ anek D (2013) Exploring parameter space of sto c hastic biochemical systems using quan titative model chec king. In: CA V, Springer, LNCS, v ol 8044, pp 107–123 31. Bruttomesso R, Cimatti A, F ranz ´ en A, Griggio A, Sebastiani R (2008) The MathSA T 4 SMT solver. In: CA V, Springer, LNCS, v ol 5123, pp 299–303 32. Budde CE, Dehnert C, Hahn EM, Hartmanns A, Junges S, T urrini A (2017) JANI: quantitativ e mo del and to ol interaction. In: T ACAS (2), LNCS, vol 10206, pp 151–168 33. Calinescu R, Ghezzi C, Johnson K, P ezz` e M, Raﬁq Y, T am burrelli G (2016) F ormal veriﬁcation with conﬁdence interv als to establish quality of service prop erties of soft wa re systems. IEEE T rans Reliability 65(1):107–125 34. Calinescu R, Johnson K, Paterson C (2016) F A CT: A probabilistic mo del c heck er for formal veriﬁcation with conﬁdence in terv als. In: T A CAS, Springer, LNCS, v ol 9636, pp 540–546 35. Calinescu R, Cesk a M, Gerasimou S, Kwiatko wsk a M, Paoletti N (2018) Eﬃcien t synthesis of robust mo dels for stochastic systems. Journal of Systems and Soft ware 143:140–158 36. Campi MC, Garatti S (2008) The Exact F easibility of Randomized Solutions of Uncertain Conv ex Programs. SIAM Journal on Optimization 19(3):1211–1230 37. Campi MC, Garatti S (2011) A Sampling-and-Discarding Approac h to Chance- Constrained Optimization: F easibility and Optimality. Journal of Optimization Theory and Applications 148(2):257–280 38. Cerotti D, Donatelli S, Horv´ ath A, Sproston J (2006) CSL mo del chec king for generalized sto c hastic Petri nets. In: QEST, IEEE Computer So ciety , pp 199–210 39. Cesk a M, Dannenberg F, Kwiatk o wsk a MZ, Paoletti N (2014) Precise param- eter syn thesis for sto c hastic bio chemical systems. In: CMSB, Springer, LNCS, v ol 8859, pp 86–98 40. Cesk a M, Pilar P , P aoletti N, Brim L, Kwiatko wsk a MZ (2016) PRISM-PSY: precise gpu-accelerated parameter synthesis for stochastic systems. In: T ACAS, Springer, LNCS, v ol 9636, pp 367–384 41. Cesk a M, Jansen N, Junges S, Katoen J (2019) Shepherding hordes of marko v c hains. In: T ACAS (2), Springer, Lecture Notes in Computer Science, vol Parameter Synthesis for Marko v Mo dels 81 11428, pp 172–190 42. Chatzieleftheriou G, Katsaros P (2018) Abstract model repair for probabilistic systems. Inf Comput 259(1):142–160 43. Chen T, Hahn EM, Han T, Kwiatko wsk a M, Qu H, Zhang L (2013) Mo del repair for Mark ov decision pro cesses. In: T ASE, IEEE Computer So ciet y , pp 85–92 44. Chen T, F eng Y, Rosen blum DS, Su G (2014) Perturbation analysis in v eriﬁcation of discrete-time Mark o v c hains. In: CONCUR, Springer, LNCS, v ol 8704, pp 218–233 45. Chonev V (2017) Reac hability in augmen ted interv al Mark ov chains. CoRR abs/1701.02996 46. Clarke EM, Grumberg O, Peled D (1999) Model Checking. MIT press 47. Clark e EM, Grum b erg O, Jha S, Lu Y, V eith H (2000) Coun terexample-guided abstraction reﬁnemen t. In: CA V, Springer, LNCS, v ol 1855, pp 154–169 48. Condon A (1990) On algorithms for simple stochastic games. In: Adv ances In Computational Complexity Theory , DIMA CS/AMS, DIMACS Series in Discrete Mathematics and Theoretical Computer Science, v ol 13, pp 51–72 49. Co ok B (2018) F ormal reasoning ab out the security of Amazon web services. In: CA V, Springer, LNCS, vol 10981, pp 38–47 50. Coppit D, Sulliv an KJ, Dugan JB (2000) F ormal semantics of mo dels for computational engineering: a case study on Dynamic F ault T rees. In: ISSRE, IEEE Computer So ciet y , pp 270–282, DOI 10.1109/ISSRE.2000.885878 51. Cormen TH, Leiserson CE, Riv est RL, Stein C (2009) In tro duction to Algo- rithms, 3rd Edition. MIT Press 52. Corzilius F, Kremer G, Junges S, Sch upp S, ´ Abrah´ am E (2015) SMT-RA T: an open source C++ to olb ox for strategic and parallel SMT solving. In: SA T, Springer, LNCS, v ol 9340, pp 360–368 53. Costen C, Rigter M, Lacerda B, Ha wes N (2023) Planning with hidden parameter p olynomial mdps. In: AAAI, AAAI Press, pp 11,963–11,971 54. Courcoub etis C, Y annak akis M (1988) V erifying temp oral prop erties of ﬁnite- state probabilistic programs. In: FOCS, IEEE Computer So ciet y , pp 338–345 55. Cousineau D (2009) Fitting the three-parameter Weibull distribution: review and ev aluation of existing and new metho ds. IEEE T ransactions on Dielectrics and Electrical Insulation 16(1):281–288 56. Cubuktep e M, Jansen N, Junges S, Katoen JP , Papusha I, Poonaw ala HA, T op cu U (2017) Sequential con vex programming for the eﬃcien t veriﬁcation of parametric MDPs. In: T ACAS (2), LNCS, vol 10206, pp 133–150 57. Cubuktep e M, Jansen N, Junges S, Kato en JP , T opcu U (2018) Synthe sis in pMDPs: A tale of 1001 parameters. In: A TV A, Springer, LNCS, v ol 11138, pp 160–176 58. Cubuktep e M, Jansen N, Junges S, Katoen J, T op cu U (2020) Scenario-based v eriﬁcation of uncertain mdps. In: T ACAS (1), Springer, LNCS, vol 12078, pp 287–305 59. Cubuktep e M, Jansen N, Junges S, Marandi A, Suilen M, T op cu U (2021) Robust ﬁnite-state con trollers for uncertain p omdps. In: AAAI, AAAI Press, pp 11,792–11,800 60. Cubuktep e M, Jansen N, Junges S, Kato en J, T op cu U (2022) Conv ex op- timization for parameter synthesis in mdps. IEEE T rans Autom Control 67(12):6333–6348 82 Sebastian Junges et al. 61. D’Argenio PR, Kato en JP , Ruys TC, T retmans J (1997) The b ounded re- transmission proto col m ust b e on time! In: T A CAS, Springer, LNCS, vol 1217, pp 416–431 62. D’Argenio PR, Jeannet B, Jensen HE, Larsen K G (2001) Reac hability analysis of probabilistic systems by successiv e reﬁnemen ts. In: P APM-PROBMIV, Springer, LNCS, v ol 2165, pp 39–56 63. Da ws C (2004) Sym b olic and parametric mo del chec king of discrete-time Mark ov chains. In: ICT A C, Springer, LNCS, vol 3407, pp 280–294 64. Dehnert C, Junges S, Jansen N, Corzilius F, V olk M, Bruintjes H, Kato en JP , ´ Abrah´ am E (2015) Prophesy: A probabilistic parameter synthesis tool. In: CA V, Springer, LNCS, vol 9206, pp 214–231 65. Dehnert C, Junges S, Kato en JP , V olk M (2017) A storm is coming: A mo dern probabilistic mo del c heck er. In: CA V, Springer, LNCS, v ol 10427, pp 592–600 66. Delgado KV, Sanner S, de Barros LN (2011) Eﬃcien t solutions to factored MDPs with imprecise transition probabilities. Artif Intell 175(9-10):1498–1527 67. Delgado KV, de Barros LN, Dias DB, Sanner S (2016) Real-time dynamic programming for Marko v decision pro cesses with imprecise probabilities. Artif In tell 230:192–223 68. v an Dijk T, v an de P ol J (2017) Sylv an: multi-core framework for decision diagrams. STTT 19(6):675–696 69. Droste M, Kuic h W, V ogler H (2009) Handb o ok of weigh ted automata. Springer Science & Business Media 70. Duﬂot M, Kwiatk owsk a MZ, Norman G, P arker D (2006) A formal analysis of blueto oth device disco v ery . STTT 8(6):621–632 71. Dugan JB, Bavuso SJ, Bo yd MA (1992) Dynamic fault-tree mo dels for fault- toleran t computer systems. T rans Reliability 41(3):363–377, DOI 10.1109/24. 159800 72. Filieri A, T amburrelli G, Ghezzi C (2016) Supporting self-adaptation via quan titative veriﬁcation and sensitivity analysis at run time. IEEE T rans Soft ware Eng 42(1):75–99 73. Gainer P , Hahn EM, Schew e S (2018) Accelerated mo del chec king of para- metric mark ov chains. In: A TV A, Springer, LNCS, vol 11138, pp 300–316 74. Giv an R, Leac h SM, Dean TL (2000) Bounded-parameter Marko v decision pro cesses. Artif In tell 122(1-2):71–109 75. Goub erman A, Siegle M, T ati B (2019) Marko v chains with p erturb ed rates to absorption: Theory and application to mo del repair. P erformance Ev aluation 76. Guennebaud G, Jacob B, et al (2010) Eigen v3. http://eigen.tuxfamily .org 77. Hahn EM, Hermanns H, W ach ter B, Zhang L (2010) P ARAM: A mo del c heck er for parametric Mark ov mo dels. In: CA V, Springer, LNCS, vol 6174, pp 660–664 78. Hahn EM, Hermanns H, Zhang L (2010) Probabilistic reachabilit y for para- metric Mark ov mo dels. STTT 13(1):3–19 79. Hahn EM, Han T, Zhang L (2011) Syn thesis for PCTL in parametric Mark ov decision pro cesses. In: NASA F ormal Metho ds, Springer, LNCS, v ol 6617, pp 146–161 80. Hahn EM, Hartmanns A, Hermanns H, Kato en JP (2013) A comp ositional mo delling and analysis framework for sto chastic h ybrid systems. F ormal Metho ds in System Design 43(2):191–232 Parameter Synthesis for Marko v Mo dels 83 81. Hahn EM, Hashemi V, Hermanns H, Lahijanian M, T urrini A (2017) Multi- ob jectiv e robust strategy synthesis for in terv al marko v decision pro cesses. In: QEST, Springer, LNCS, v ol 10503, pp 207–223 82. Hahn EM, Hashemi V, Hermanns H, Lahijanian M, T urrini A (2019) In terv al mark ov decision processes with m ultiple ob jectives: F rom robust strategies to pareto curv es. ACM T rans Model Comput Simul 29(4):27:1–27:31 83. Han T, Kato en JP , Mereacre A (2008) Approximate parameter synthesis for probabilistic time-bounded reachabilit y . In: R TSS, IEEE Computer Society , pp 173–182 84. Han Y (2013) State elimination heuristics for short regular expressions. F un- dam Inform 128(4):445–462 85. Haselman M, Hauc k S (2010) The future of in tegrated circuits: A surv ey of nano electronics. Proceedings of the IEEE 98(1):11–38 86. Hec k L, Sp el J, Junges S, Mo erman J, Kato en J (2022) Gradient-descen t for randomized con trollers under partial observ ability . In: VMCAI, Springer, LNCS, v ol 13182, pp 127–150 87. Helmink L, Sellink MP A, V aandrager FW (1993) Pro of-chec king a data link proto col. In: TYPES, Springer, LNCS, v ol 806, pp 127–165 88. Herman T (1990) Probabilistic self-stabilization. Inf Pro cess Lett 35(2):63–67 89. Holtzen S, Junges S, V azquez-Chanlatte M, Millstein TD, Seshia SA, den Bro ec k GV (2021) Model chec king ﬁnite-horizon marko v chains with proba- bilistic inference. In: CA V (2), Springer, LNCS, vol 12760, pp 577–601 90. Hop croft JE, Motw ani R, Ullman JD (2003) Introduction to Automata Theory , Languages, and Computation. Addison-W esley 91. Jansen N, Corzilius F, V olk M, Wimmer R, ´ Abrah´ am E, Kato en JP , Beck er B (2014) Accelerating parametric probabilistic veriﬁcation. In: QEST, Springer, LNCS, v ol 8657, pp 404–420 92. Jonsson B, Larsen KG (1991) Sp eciﬁcation and reﬁnement of probabilistic pro cesses. In: LICS, IEEE Computer Society , pp 266–277 93. Jo v anovic D, de Moura LM (2013) Cutting to the chase - solving linear integer arithmetic. J Autom Reasoning 51(1):79–108 94. Junges S (2020) Parameter synthesis in mark o v mo dels. PhD thesis, R WTH Aac hen Universit y , Germany 95. Junges S, Spaan MTJ (2022) Abstraction-reﬁnement for hierarc hical proba- bilistic mo dels. In: CA V (1), Springer, LNCS, vol 13371, pp 102–123 96. Junges S, Jansen N, Wimmer R, Quatmann T, Winterer L, Kato en JP , Beck er B (2018) Finite-state controllers of p omdps using parameter synthesis. In: UAI, A UAI Press, pp 519–529 97. Junges S, Katoen J, P ´ erez GA, Winkler T (2021) The complexity of reachabil- it y in parametric marko v decision pro cesses. J Comput Syst Sci 119:183–210 98. Kato en JP (2016) The probabilistic mo del c hec king landscape. In: LICS, A CM 99. Kn uth D, Y ao A (1976) Algorithms and Complexity: New Directions and Recen t Results, Academic Press, chap The complexity of non uniform random n umber generation 100. Kozine I, Utkin L V (2002) Interv al-v alued ﬁnite Marko v chains. Reliable Computing 8(2):97–113 101. Kurshan RP (2018) T ransfer of model chec king to industrial practice. In: Handb ook of Mo del Checking, Springer, pp 763–793 84 Sebastian Junges et al. 102. Kwiatk owsk a M, Norman G, Park er D (2011) Prism 4.0: V eriﬁcation of probabilistic real-time systems. In: CA V, Springer, LNCS, vol 6806, pp 585– 591 103. Kwiatk owsk a M, Norman G, Park er D (2012) The PRISM b enchmark suite. In: QEST, IEEE Computer So ciet y , pp 203–204 104. Kwiatk owsk a MZ, Norman G, P ark er D (2008) Using probabilistic mo del c hecking in systems biology . SIGMETRICS P erformance Ev aluation Review 35(4):14–21 105. Kwiatk owsk a MZ, Norman G, Park er D (2012) Probabilistic veriﬁcation of Herman’s self-stabilisation algorithm. F ormal Asp Comput 24(4-6):661–670 106. Lanotte R, Maggiolo-Schettini A, T roina A (2007) Parametric probabilis- tic transition systems for system design and analysis. F ormal Asp ects of Computing 19(1):93–109 107. Long F, Rinard M (2016) Automatic patc h generation b y learning correct co de. In: POPL, A CM, pp 298–312 108. Mannor S, Meb el O, Xu H (2012) Lightning do es not strike t wice: Robust MDPs with coupled uncertain ty . In: ICML, icml.cc / Omnipress 109. Marsan MA, Balbo G, Con te G, Donatelli S, F rancesc hinis G (1998) Modelling with generalized sto c hastic petri nets. SIGMETRICS P erformance Ev aluation Review 26(2):2 110. McGlynn MJ, Borbash SA (2001) Birthda y proto cols for low energy deploy- men t and ﬂexible neighbor disco very in ad hoc wireless netw orks. In: MobiHo c, A CM, pp 137–145 111. Meedeniy a I, Moser I, Aleti A, Grunske L (2014) Ev aluating probabilistic mo dels with uncertain mo del parameters. Soft ware and System Modeling 13(4):1395–1415 112. de Moura LM, Bjørner N (2008) Z3: An eﬃcient SMT solver. In: T ACAS, Springer, LNCS, v ol 4963, pp 337–340 113. Mushkin M, Bar-David I (1989) Capacity and co ding for the gilb ert-elliot c hannels. IEEE T rans Information Theory 35(6):1277–1290 114. Neary C, V erginis CK, Cubuktep e M, T op cu U (2022) V eriﬁable and comp o- sitional reinforcement learning systems. In: ICAPS, AAAI Press, pp 615–623 115. v on Neumann J (1956) Probabilistic logics and synthesis of reliable organisms from unreliable comp onents. In: Shannon C, McCarthy J (eds) Automata Studies, Princeton Univ ersity Press, pp 43–98 116. Norman G, Shmatik ov V (2006) Analysis of probabilistic con tract signing. Journal of Computer Securit y 14(6):561–589 117. Norman G, Park er D, Kwiatko wsk a M, Shukla S (2005) Ev aluating the relia- bilit y of NAND multiplexing with PRISM. IEEE T ransactions on Computer- Aided Design of In tegrated Circuits and Systems 24(10):1629–1637 118. Norman G, Park er D, Zou X (2017) V eriﬁcation and control of partially observ able probabilistic systems. Real-Time Systems 53(3):354–402 119. P athak S, ´ Abrah´ am E, Jansen N, T acc hella A, Kato en J (2015) A greedy approac h for the eﬃcient repair of stochastic mo dels. In: NFM, Springer, LNCS, v ol 9058, pp 295–309 120. P etrucci L, v an de Pol J (2018) Parameter syn thesis algorithms for parametric in terv al Marko v chains. In: F OR TE, Springer, LNCS, v ol 10854, pp 121–140 121. P olgreen E, Wijesuriy a VB, Haesaert S, Abate A (2016) Data-eﬃcien t ba yesian v eriﬁcation of parametric marko v chains. In: QEST, Springer, LNCS, vol 9826, Parameter Synthesis for Marko v Mo dels 85 pp 35–51 122. Puggelli A, Li W, Sangiov anni-Vincentelli AL, Seshia SA (2013) P olynomial- time v eriﬁcation of PCTL prop erties of MDPs with conv ex uncertainties. In: CA V, Springer, LNCS, vol 8044, pp 527–542 123. Puterman ML (1994) Mark ov Decision Processes: Discrete Sto chastic Dynamic Programming. John Wiley and Sons 124. Quatmann T, Dehnert C, Jansen N, Junges S, Katoen JP (2016) Parameter syn thesis for Mark ov models: F aster than ever. In: A TV A, LNCS, v ol 9938, pp 50–67 125. Ruijters E, Stoelinga M (2015) F ault tree analysis: A survey of the state-of- the-art in mo deling, analysis and to ols. Computer Science Review 15-16:29–62 126. Russell SJ, Norvig P (2010) Artiﬁcial Intelligence - A Mo dern Approach (3. in ternat. ed.). Pearson Education 127. Sak arovitc h J (2005) The language, the expression, and the (small) automaton. In: CIAA, Springer, LNCS, v ol 3845, pp 15–30 128. Salmani B, Katoen J (2021) Fine-tuning the o dds in ba y esian netw orks. In: ECSQAR U, Springer, LNCS, vol 12897, pp 268–283 129. Segala R, T urrini A (2005) Comparative analysis of bisimulation relations on alternating and non-alternating probabilistic mo dels. In: QEST, IEEE Computer So ciet y , pp 44–53 130. Shapley LS (1953) Stochastic games. Pro ceedings of the National Academy of Sciences 39(10):1095–1100 131. Sp el J, Junges S, Kato en J (2019) Are parametric marko v chains monotonic? In: A TV A, Springer, LNCS, v ol 11781, pp 479–496 132. Sp el J, Junges S, Katoen J (2021) Finding prov ably optimal marko v chains. In: T ACAS (1), Springer, LNCS, vol 12651, pp 173–190 133. Su G, F eng Y, Chen T, Rosen blum DS (2016) Asymptotic p erturbation bounds for probabilistic model c hec king with empirically determined probabilit y parameters. IEEE T rans Softw are Eng 42(7):623–639 134. Suilen M, Jansen N, Cubuktep e M, T op cu U (2020) Robust p olicy syn thesis for uncertain pomdps via conv ex optimization. In: IJCAI, ijcai.org, pp 4113–4120 135. Suilen M, Sim˜ ao TD, Park er D, Jansen N (2022) Robust anytime learning of mark ov decision pro cesses. In: NeurIPS 136. T appler M, Aichernig BK, Bacci G, Eichlseder M, Larsen KG (2019) L * -based learning of marko v decision pro cesses. In: FM, Springer, Lecture Notes in Computer Science, v ol 11800, pp 651–669 137. V ardi MY (1985) Automatic veriﬁcation of probabilistic concurren t ﬁnite-state programs. In: F OCS, IEEE Computer So ciet y , pp 327–338 138. V esely W, Stamatelatos M (2002) F ault tree handb o ok with aerospace appli- cations. T ech. rep., NASA Headquarters, USA 139. V olk M, Junges S, Kato en JP (2016) Adv ancing dynamic fault tree analysis - get succinct state spaces fast and synthesise failure rates. In: SAFECOMP, Springer, LNCS, v ol 9922, pp 253–265 140. V olk M, Junges S, Kato en JP (2018) F ast dynamic fault tree analysis by mo del c hecking techniques. IEEE T rans Industrial Informatics 14(1):370–379 141. Wiesemann W, Kuhn D, Rustem B (2013) Robust Mark ov decision pro cesses. Mathematics of Op erations Researc h 38(1):153–183 86 Sebastian Junges et al. 142. Winkler T, Junges S, P´ erez GA, Kato en J (2019) On the complexit y of reac hability in parametric marko v decision pro cesses. In: CONCUR, Schloss Dagstuhl - Leibniz-Zen trum f ¨ ur Informatik, LIPIcs, vol 140, pp 14:1–14:17 143. Y ang L, Murugesan S, Zhang J (2011) Real-time scheduling o ver Marko vian c hannels: When partial observ ability meets hard deadlines. In: GLOBECOM, IEEE, pp 1–5

Parameter Synthesis for Markov Models: Covering the Parameter Space

Original Paper

Comments & Academic Discussion

Leave a Comment

Original Paper

Related Papers

Comments & Academic Discussion

Leave a Comment