A Quantifier Elimination Algorithm for Linear Real Arithmetic

A Quan tifier Eliminatio n Algorithm for Linear Real Arithmetic Da vid Monniaux VERIMA G ∗ Octob er 22, 2018 Abstract W e prop ose a new quantifier elimination algori t hm for the theory of linear real arithmetic. This algorithm uses as subroutines satisfiability mod u lo this theory and polyh edral pro jection; there are good algorithms and implementations for b oth of th ese. The quantifier elimination al- gorithm presented in the pap er is compared, on examples arising from program analysis problems and on rand om examples, to several other im- plementatio n s, all of which cannot solv e some of the examples that our algorithm solve s easily . 1 In tro duction Consider a log ic formula F , p ossibly with quant ifier s, whose v ariables lay within a cer tain set S and whose a tomic predicates are relations over S . The mo dels of this formula are assignment s of v alues in S for the free v ar iables of F such that F ev aluates to “ true” . Quantifier elimination is the a ct of providing ano ther formula F ′ , without qua ntifiers, such that F and F ′ are e quivalent , that is, hav e exactly the same mo dels. F or instance, ∀ x ( x ≥ y ⇒ x ≥ 3) is equiv alen t to quantifier-free y ≥ 3. If F ha s no free v a riables, then F ′ is a gr ound (qua n tifier-fre e, v aria ble-free) formula. In most pr actical ca ses such formulas can be easily decided to b e true or false; qua n tifier elimination thus pr o v ides a de cision pr o c e dur e for quantified formulas. In this paper , we only co ns ider relations of the form L ( x, y , z , . . . ) ≥ 0 where L is a linear affine expressio n (an arithmetic express ion where m ultiplica tion is allowed only by a constant fac tor), int e r preted ov er the real n umbers (or, equiv alently , over the rationals). W e can th us deal with any formula o ver linear equalities or inequalities. Our a lgorithm transfor ms any formula of the form ∃ x 1 , . . . , x n F , wher e F has no quant ifiers, into a quantifier-free formula F ′ in disjunctiv e no rmal form. Nes ted quantifiers are dealt with by syntactic induc- tion: in o rder to eliminate quantifiers from ∃ x F o r ∀ x F , where F may co ntain quantifiers, one first e limina tes quantifiers from F . Univ er sal quantifiers are ∗ VERIMAG i s a join t l ab oratory of CNRS, Univ er sit ´ e Joseph F ourier and Grenoble INP . 1 conv erted to existent ia l o ne s ( ∀ x 1 , . . . , x n F ≡ ¬∃ x 1 , . . . , x n ¬ F ), yet o ur a lgo- rithm gener ally a voids the combinatorial explos ion ov er negations that hinders some other methods. Our metho d can b e understo o d as an impr o vement ov er the approa c h of conv erting to DNF thro ugh ALL-SA T and p erforming pro jection; we co mpared bo th approa c hes exp erimen tally (see § 5.2). W e co mpa red o ur implemen ta - tion with commer cial and no nco mmercial q ua n tifier elimination pr o cedure s ov er some examples ar is ing from practica l pro g ram analys is cas es, as well as ran- dom problems, and ours w a s the only one capable of pro cessing them without exhausting memory or time, or failing altogether due to the imp ossibility of handling large coefficients. 2 The Algorithm W e fir st describ e the datatypes o n which our algo rithm opera tes, then the off- the-shelf subroutines that it uses, then the algo rithm and its correctness pro o f, then pos sible a lterations. 2.1 Generalities W e op erate on unquantified formulas built using ∧ , ∨ , ⇒ , ¬ o r other lo gical connectives s uc h as exclusive-or (the exact set of connectives allow ed depe nds on the satisfiability tester b eing used, see b elow; in this paper w e s hall only us e ∧ , ∨ a nd ¬ ), a nd o n quantified formulas built with the sa me connectives and the existential ( ∃ ) and univ er sal ( ∀ ) quantifiers. It is pos sible to quantify not only on a single v a riable but also on a set of v a riables, repres en ted as a vector ~ v . The atoms are linear inequa lities, that is, formulas of the form c + c x x + c y y + c z z · · · ≥ 0 where c ∈ Q is the c onstant c o efficient and c v ∈ Q is the co efficient asso ciated with v ariable v . It is trivia lly po ssible to represent equalities o r str ict inequalities using this fo r m ula langua ge. The mo dels of a formula F are assig nmen ts a of rational num b ers to the free v ar iables o f F such that a satisfies F (written a | = F ). F is said to b e satisfiable if a mo del exists for it. If F ha s no free v ariables, then F is sa id to b e true if F is satisfiable, false other wise. Two formulas A and B are said to b e e quivalent , noted A ≡ B , if they have the s ame mo dels. F o rm ula A is s aid to imply formula B , noted A ⇛ B , if any mo del of A is a mo del of B . Consider a quantifier-free for m ula F , whose atomic predicates are line a r inequalities, and v ar iables x 1 , . . . , x n . W e wish to obtain a quantifier-free for- m ula F ′ equiv alent to ∃ x 1 , . . . , x n F . Let us temp orar ily forget ab out e ffi- ciency in order to convince our s elv es quickly that quantifier elimination is p os- sible. F ca n b e put into disjunctive nor mal form (DNF) C 1 ∨ · · · ∨ C m (b y recursive application of distributivity), and ∃ x 1 , . . . , x n F is th us equiv alent to ( ∃ x 1 , . . . , x n C 1 ) ∨ · · · ∨ ( ∃ x 1 , . . . , x n C m ). V a rious metho ds exist for finding a conjunction C ′ i equiv alent to ∃ x 1 , . . . , x n C i , among which we can cite F ourier- Motzkin elimination (see § 5.1). W e therefore obtain F ′ in DNF. F or a universal quantifier, thro ugh De Morg an’s laws, we o btain a formula in conjunctive nor mal form (CNF). Such a naive a lgorithm suffers from an o b vio us inefficiency , par ticularly if applied recurs iv ely to formulas with alter nating q uan tifiers . A first and obvious 2 step is to replace DNF conv ersio n thro ugh distr ibutivit y b y mo dern tec hniques (mo de l enumeration using satisfiabilit y modulo theory). W e show in this pap er than one ca n do better by interleaving the pro jection and the model numeration pro cesses. 2.2 Building blo cks If one has prop ositional for m ulas with a lar ge num be r of v ariables, one never conv erts formulas naively from CNF to DNF, but one uses techniques suc h as prop ositional satisfiability (SA T) solving . Even though SA T is NP -complete, there now exist a lgorithms and implementations that can deal efficiently with many lar ge problems a rising fro m pr ogram verification. In our case, we apply SA T mo dulo the theory of linear r eal inequalities (SMT), a problem for which there also exis t algorithms, implemen tations , standard b enc hmar ks and even a comp etition. Like SA T, SA T mo dulo linear inequa lities is NP-complete. A SMT solver takes as an input a formula F where the literals are linea r equa lities or inequalities, and a nsw er s either “ not satisfiable” , or a mo del of F , assigning a rational num b er to each v ar iable in F . W e assume we have suc h an a lgorithm Smt at our disp osal as a building block Another needed building blo ck is qua n tifier elimination ov er conjunctions, named Project ( C , ~ v ): given a conjunction C of linear inequalities ov er v ari- ables ~ v = v 1 , . . . , v N , o btain a co njunction C ′ equiv alent to ∃ v 1 , . . . , v n C . F or efficiency r easons, it is better if C ′ is minimal (no conjunct can be remov ed without adding more mo dels), or a t least “small” . F ourier-Mo tzkin elimina- tion is a simple algo r ithm, yet, when it eliminates a single v ariable, the output conjunction ca n have a quadr atic num b er o f conjuncts compare d to the in- put co njunction, th us a pass of simplification is needed for practical efficiency ; v arious algor ithms hav e b een prop osed in that re s pect [9]. F or our implemen ta- tions, w e used “black b ox” libra ries implemen ting g eometrical transfor mations, in particular po lyhedron pro jection: C defines a conv e x p olyhedron 1 in Q N , and finding C ′ amounts to computing the inequalities defining the pro jection of this po lyhedron into Q N − n . 3 Quan tifier Elimination Algorithm W e s hall first describ e subr outines Generalize1 and Generalize2 , then the main algorithm E xistElim . 3.1 Generalized Mo dels Consider a satisfiable qua n tifier-fr ee formula F . W e supp ose w e hav e a t our disp o sal a SMT-solv ing alg orithm that will output a mo del m | = F . W e wish to obta in instead a generalize d mo del: a conjunction C such that C = ⇒ F . 1 A go od bi bliograph y on con vex polyhedra and the asso ciated algorithms can b e f ound in the documen tation of the P arma Po l yhedra Li brary . [1] By c onve x p olyhe dr on , w e mean, in a finite-dimension affine linear real space, an int ersection of a finite num b er of half-spaces eac h delimited b y a linear inequality , that is, the set of solutions of a finite system of linear inequalities. In particular, such a p olyhedron can b e unbounded. In the rest of the paper , the wo r ds “polyhedron” must b e understoo d to mean “con v ex polyhedron” with that definition. 3 Algorithm 1 Generalize1 ( a, F ): Gener alize a mo del a of a for mula F to a conjunction Require: a | = F M ← true for all P ∈ A tomicPredica tes ( F ) do if a | = P then M ← M ∧ P else M ← M ∧ ¬ P end if end for Ensure: M ⇛ F Algorithm 2 Generalize2 (G, M): Remo ve useless constraints from conjunc- tion M so that G ∧ M ≡ false Require: G ∧ M is not satisfia ble for all c conjunct in M d o if ( G \ { c } ) ∧ M is not satisfiable (call Smt ) then remov e c fr o m M end if end for Ensure: G ∧ M is not satisfia ble Ideally , w e would like C to hav e as few conjuncts a s p ossible. W e shall no w see algorithms in order to o bta in such g eneralized mo dels. The truth v a lue of F on an assignment a o f its v a riables o nly dep ends on the truth v alue of the atomic predicates of F over a . Let us note N F = | A tomicPredica tes ( F ) | , wher e | X | denotes the cardinality of the se t X . These truth assignments therefor e define at most 2 N F equiv alence classes ov er the v al- uations o f the v ariables a ppear ing in F . There can be few er than 2 N F equiv a- lence classe s, b ecause so me truth ass ignmen ts can b e c o n tra dictory (for instance, x ≥ 1 assigned to true a nd x ≥ 0 assigned to false ). One can immediately ge ne r - alize a mo del o f a formula to its equiv alence class, which motiv ates our algorithm Generalize1 . Its output is a conjunction o f liter als fro m F . This conjunction may itself b e insufficiently g e neral. Consider the formula F = ( x ≥ 0 ∧ y ≥ 0) ∨ ( ¬ x ≥ 0 ∧ y ≥ 0). x 7→ 0 , y 7→ 0 is a mo del of F . Generalize1 will output the conjunction x ≥ 0 ∧ y ≥ 0. Y et, the first co njunct could b e safely removed. Generalize2 ( ¬ ( F ∨ O ) , M ) will remov e unnecessa ry conjuncts from M while pr eserving the pro perty that M ⇛ F ∨ O . Figure 3 illustrates wh y it is b e tter to gener alize the conjunctions. The problem of obtaining a minimal (or a t leas t, “re asonably small” ) in- consistent subse t o ut of an inconsis ten t conjunction has alrea dy b een studied. In DPLL(T) algor ithms [8] for SMT-so lving, the problem is to find out, given a cons isten t c o njunction of literals L 1 ∧ · · · ∧ L n and a new liter al L ′ , whether L 1 ∧ · · · ∧ L n ⇒ L ′ , L 1 ∧ · · · ∧ L n ⇒ ¬ L ′ , or neither ; and if one of the implicatio ns holds, pro duce a minimal explanation wh y it holds, that is, a subset L i 1 , . . . , L i m of the L i such that L i 1 ∧ · · · ∧ L i m ⇒ L ′ (resp ectiv e ly , ⇒ ¬ L ′ ). Since this de- cision and explanation pro cedure is calle d o ften, it should b e fa s t and muc h 4 projection of A A B C Figure 1: Subsumption of one generalized mo del by ano ther effort has b een devoted in that re spect by implementors of SMT- solvers (e.g. [13] for cong ruence theories). It is how e ver not straightforward to use such ex- planation pro cedures for o ur pur poses , since we do not consider conjunctions of literals only: when algo r ithm ExistEl im inv okes Generalize2 ( ¬ F, M 1 ), ¬ F is in general a complex formula, not a literal. W e therefore present here a stra igh tfor w ar d incons is ten t set minimization algorithm similar to the one found in [6, § 6]. General ize2 ( G, M ), where M is a conjunction suc h that G ∧ M is unsatisfiable, works as follows: • It attempts removing the first co njunct fro m M (thus relaxing the M constraint). If G ∧ M stays unsa tisfiable, the conjunct is remov ed. If it bec omes satisfia ble, then the conjunct is necessary and is kept. • The pro cess is co n tinued with the following conjuncts. Unsurprisingly , the re sults of this pro cess de p end on the or de r of the con- juncts inside the conjunction M . So me order s may per form b etter than others; the resulting set of co njuncts is minimal with resp ect to inclusion, but not nec- essarily with resp ect to cardina lit y . 2 3.2 Main A lgorithm 2 This is the case even if we consider a purely prop ositional case. As an example, consider F = A ∨ ( B ∧ C ). M = A ∧ B ∧ C ⇛ F , otherwise said M ∧ ¬ F is not satisfiable. If one first relaxes the constrain t A , one gets the conjunction B ∧ C , whic h still impli es F ; this conjunction has tw o propositional mo dels ( A ∧ B ∧ C and ¬ A ∧ B ∧ C ). Y et, one could hav e c hosen to relax B a nd obtain A ∧ C , and then to relax C a nd obtain A (which stil l implies F ); this f orm ul a has four propositional mo dels. 5 A B C O D Figure 2: The gr a y area is the set of po in ts matched by for m ula F = y ≥ − 1 ∨ ( y ≥ − 2 ∧ x ≥ − 1 ∧ x ≤ 1). Poin t O = (0 , 0) is found as a mo del. This mo del is first ge ne r alized to y ≥ − 1 ∧ y ≥ − 2 ∧ x ≥ − 1 ∧ x ≤ 1 acco r ding to its v aluations on the atomic b o olean form ulas . Depending on whether one first tries to r elax x ≥ − 1 or y ≥ − 1, one g ets either a half pla ne (one conjunct) or a v er tical band (three conjuncts); the for mer is “ simpler” than the second. The simplicity o f the formula output by Generalize2 thu s dep ends o n the or dering of the input co njuncts. Algorithm 3 ExistElim : Ex is ten tial quan tifier elimina tio n H ← F O ← false while H is satis fia ble (call S mt ) do { ( ∃ ~ v F ) ≡ ( O ∨ ∃ ~ v H ) and H ∧ O ≡ false and O do es not men tion v ariables from ~ v } a ← a mo del o f H { a | = H } M 1 ← Generalize1 ( F , a ) { M 1 ⇛ F } M 2 ← Generalize2 ( ¬ F, M 1 ) {¬ ( M 2 ∧ G ) } π ← Project ( M 2 , ~ v ) { π ≡ ∃ ~ v M 2 } O ← O ∨ π H ← H ∧ ¬ π end while Ensure: O ≡ ∃ ~ v F The main a lgorithm is ExistE lim ( F, ~ v ) which co mputes a DNF formula equiv alent to ∃ ~ v F . ~ v is a v e c to r of v ariables. ~ v can be empty , and then the a l- gorithm simply c omputes a “simple” DNF form for F . The a lgorithm computes generalized mo dels of F and pro jects them one by one, un til exhaustion. It maintains three fo r m ulas H and O . O is a DNF form ula containing the pro jec- tions o f the mo dels pr o cess ed so far. H contains the mo dels yet to be pro cessed; it is initially equal to F . F or each ge ne r alized mo del M , its pro jection π is added to O and remov ed from H . ExistEl im can thu s b e unders too d as an ALL-SA T implementation coupled with a pro jection, wher e the pro jection is per formed inside the lo op so as to simplify the problem (a s o pposed to w a iting 6 A B C Figure 3: A is the fir st gener a lized mo del se le c ted. If G 0 def = ¬ F , the initial v alue of G , is replaced at the next iteration by G 1 def = ¬ F ∧ ¬ π 0 where π 0 is the pr o jection of A , then it is p ossible to generate a s ing le genera lized model encompassing b oth B and C (for instance x ≥ − 1 ∧ y ≥ 0 ∧ y ≤ 2 . If G stays constant, then the x ≥ 1 constr ain t defining the le ft edge o f C cannot b e rela xed. for all models to b e o utput a nd pro jecting them). The par tial correctness of the algo rithm ensues from the loo p condition a nd the following lo op inv ar ian ts : ( ∃ ~ v F ) ≡ O ∨ ( ∃ ~ v H ), H ⇛ F and O do es not men tion v ar iables fro m ~ v . Given a formula φ , we denote by W ( φ ) the num b er of equiv alence classe s induced by the atomic predica tes of F with nonempty intersection with the mo dels of φ . T er mination is ensured b ecause W ( H ) decrea ses by at leas t o ne at each itera tio n: M 1 defines exactly one equiv alence class, M 2 defines a union of equiv a lence cla sses which includes the one defined b y M 1 , and the mo dels of π include tho se of M 2 th us also at leas t one equiv alence class . The n umber of iterations is thus at most 2 N F . Note that Generalize2 is needed neither for correctnes s nor for terminatio n, but only for efficiency: otherwise, the n umber of iterations would always be the num b er of eq uiv alence clas ses, which can be hu g e. 4 P ossible Changes and Extensions W e inv estiga ted tw o v ar iations of the same algor ithm, b oth of which p erform significantly worse. In addition, we extended the algo rithm to quantifier elimi- nation modulo a user-sp ecified theory . 7 4.1 ALL-SA T t hen pro ject (Mo d1) The algor ithm would still b e corr ect if M was remov ed from H instead of π . It then beco mes equiv alent to p erforming ALL-SA T (obta ining all satisfying assignments) then pr o jection. On the one ha nd, with this mo dified algo rithm, the set of atomic for m ulas of H would stay included in that o f F throughout the iterations, while this set ca n g row larger with the or iginal algorithm since the set of atomic formulas o f the pr o jection of F can b e much larg er than the set of atomic formulas in F (see § 5 .1). O n the o ther hand, the o riginal a lg orithm may need few er iterations b ecause π may subsume several genera lized mo dels, as shown by Fig. 1 : A is the first g eneralized mo del b eing gener ated, and its pro jection subsumes B ; thus, the orig inal algor ithm will not hav e to gener ate B , while the mo dified algor ithm will gener ate B . Our exp eriments ( § 5.2) s howed that the unmo dified algorithm often p erforms muc h be tter in practice than this approach. 4.2 Remo v als from Negated Set (Mo d2) Algorithm 4 ExistElim (Mo d2): Existential quantifier eliminatio n H ← F G ← ¬ F O ← false while H is satisfiable (call Smt ) do { ( ∃ ~ v F ) ≡ ( O ∨ ∃ ~ v H ) and G ≡ ¬ ( F ∨ O ) and H ∧ O ≡ false and O do es not mention v ariables from ~ v } a ← a mod el of H { a | = H } M 1 ← G eneralize1 ( F, a ) { M 1 ⇛ F } M 2 ← G eneralize2 ( G, M 1 ) {¬ ( M 2 ∧ G ) } π ← Project ( M 2 , ~ v ) { π ≡ ∃ ~ v M 2 } O ← O ∨ π H ← H ∧ ¬ π G ← G ∧ ¬ π end while Ensure: O ≡ ∃ ~ v F The algor ithm given previo usly was not the first we exp erimented; we had originally a slightly more complicated one, given as ExistElim (Mo d2), which we wro ngly tho ugh t would be more efficient. Instead of using ¬ F to check for inappropriate gene r alizations, we used a formula G initially equa l to ¬ F , and then progr essively altered. The terminatio n pro of stays the same, while correctnes s relies on the additional inv ariant G ≡ ¬ ( F ∨ O ). ExistElim can b e thought of as iden tical to ExistEl im (Mod2) except that G stays c onstant . W e thoug ht this scheme a llo wed more g eneralization of mo dels than the algorithm we gav e earlie r in the article, a s shown by Fig. 3. ExistE lim tries to generalize M to a conjunction that implies F , but in fact this is too strict a condition to succeed, whereas ExistElim (Mo d2) succeeds in generalizing F to a conjunction that implies F ∨ O . If at least one v ar iable is pro jected out, and F a ctually depe nds on that v ariable, then the mo dels o f F are strictly included in those of the final v alue of O , which is eq uiv alent to ∃ ~ v F . Exp eriment s ( § 5.2) how ever showed that this “more clever” algor ithm is slow er by approximately a facto r o f tw o, b ecause adding extra a ssertions to 8 G is costly fo r the SMT-s olv e r . 4.3 Extra Mo dulo Theory The alg orithm ca n b e easily ex tended to quantifier elimination mo dulo an a s- sumption T on the free v ar iables of F . All definitions stay the same except that ⇛ is r eplaced by ⇛ T , defined a s P ⇛ T Q def = ( P ∧ T ) ⇛ ( Q ∧ T ) and ≡ is re pla ced b y ≡ T , defined as ( P ≡ T Q ) def = ( P ∧ T ≡ Q ∧ T ). ExistElim is mo dified by replacing the initia lization o f G and H by ¬ F ∧ T and F ∧ T resp ectively . Intuitiv ely , T defines a universe of v alidit y suc h that v alues o utside of the models T ar e ir r elev an t to the pr oblem b eing studied. 5 Comparison with Other Algorithms The “cla s sical” algo rithm for qua n tifier elimination ov er linear inequa lities is F errante and Rack off ’s [7]. Another algorithm based on similar idea s , but with better p erforma nce, w a s pr opo sed by Lo os and W eispfenning [10]. W e shall therefore co mpare o ur method to these algor ithms, b oth theoretically and ex - per imen tally . W e also compared o ur algorithm with other av a ilable pa c k ages using other quan tifier elimination tec hnique s . 5.1 Complexit y b ounds Benc h mark r. lim. R r. lim. float prsb23 blowup5 Mjollnir 1.4 17 0.06 negligi b le Mjollnir (mo d1) 1.6 77 a 0.06 negligi b le Mjollnir (mo d2) 1.5 34 0.07 negligi b le Mjollnir Lo os-W eispfenning o-o-m o-o-m o-o-m negligible Proof-of-concept n/a 823 n/a n/a Mjollnir F erran te-R ac koff o-o-m o-o-m o-o-m negligible Proof-of-concept n/a 823 n/a n/a Lira o-o-m o-o-m 8.1 0.6 Redlog rlqe 182 o-o-m 1.4 negligible Redlog rlqe + rldnf o-o-m o-o-m n/a n/a Ma thema tica Reduce ( > 12000) o-o-m ( > 780) 7.36 a Memory consumption grows to 1.1 GiB. T able 1 : Timings (in seconds, o n an AMD T urion TL- 58 64-bit Linux system) for e liminating quantifiers from our be nc hmark s. The first line is the algorithm describ ed in this pap e r, the t wo following linear v ariants from § 4, then other pack age s . Reduce has rlqe (quantifier eliminatio n) and rlqe + rldnf (same, follow ed by co n version to DNF). ( > t ) means that the computatio n was killed after t seconds b ecause it was running to o long. The prs b23 and following are decision pro blems, the output is true o r fa lse , thus DNF form do es not matter. Out-of-memory is noted “o-o-m” . W e cons ider in this section that inequalities are written using in teger coeffi- cients in binary notation. W e shall prove tha t a c omplexit y bo und 2 n 2 q where n is the nu mber of atomic formulas and q is the num b er of qua n tifiers to b e 9 eliminated. This y ields an overall complexity of 2 2 2 | F | where | F | is the size of the formula. Let us co ns ider a co njunction of inequalities taken from a set of n ineq ualities. The F o urier-Motzkin algor ithm [4, 9] eliminates v ariable x from this conjunction as follows. It first partitions these inequalities into those where x do es no t app ear, which are retained verbatim, a nd those where x a pp ears p ositively ( E + ) and neg ativ ely ( E − ). F rom each couple o f inequalities ( e + , e − ) in E + × E − , an inequality wher e x does not app ear is obtained b y cancellation between e + and e − . The size in bits of the co efficients in the output inequalities can b e at most 2 s + 1 where s is the ma x imal size of the input coe fficie n ts. The inequalities output ther efore belo ng to a set of size as ymptotically at most n 2 / 4 (the worst-case o ccurs when the inequalities split evenly b et ween those in which x app ears po sitiv ely and those where it app ears negatively). The output conjunction is in gener al to o large: many inequa lities in it are sup e rfluous; yet it is guaranteed to include all inequalities defining the facets of the pro jection of the polyhedr on. Consider a formula F written with inequalities A 1 , . . . , A n as atomic for - m ula s, with max ima l co efficient size s . Our alg o rithm eliminates the quantifier from ∃ x F and outputs a DNF formula F ′ built with inequalities found in the output o f the F ourier -Motzkin a lgorithm op erating on the set A 1 , . . . , A n and v ariable x . It follows that F ′ is built from at mo st, as ymptotically , n 2 / 4 in- equalities as atomic formulas. The running time for this quantifier elimination comes from: • The SMT s olving passes. There are a t most 2 n branches to explor e in to ta l. F or ea c h br anc h, SMT has to test whether the solution set of a conjunction of poly nomial ine q ualities is empty o r no t, which is a pa rticular ca se of linear progr amming, with p olynomial complexity . The overall SMT cost is therefore b ounded by O (2 n .P ( n )) for some polynomia l P ; • The pro jections, with complexity O ( n 2 .s ), applied to each of a t mos t 2 n po lyhedra. This giv e s a n ov er all complexity of O (2 cn ) where c is a cons ta n t. Consider no w a succession of quan tifier eliminations (with or without a lter- nations). W e now have F consisting of a sequence of quantifiers follow ed b y a quantifier-free formula built out of atomic form ula s A 1 , . . . , A n . Our algor ithm per forms eliminatio ns in seq ue nce , star ting from the rightmost quantifier. Let us no te A ( k ) the set of atomic formulas that can be obtained after k eliminations; A (0) = { A 1 , . . . , A n } . Clea rly , | A ( k ) | ≤ | A (0) | 2 k asymptotically , since at ea c h iteration the size of the set of atomic for m ulas can at most ge t squared by F ourier- Motzkin elimination. The size of the co efficients grows at most as s. 2 k . This yields the promised bound. It is p ossible that the b ound | A ( k ) | ≤ | A (0) | 2 k , obta ine d by observ ation of the F ourier -Motzkin algorithm, is to o p essimistic. The literature do es not show examples of such doubly exp onential blowups, while po lyhedra with single exp onen tial blowups can b e constructed. The “classica l” algorithm for quantifier e limination over real or rationa l arith- metic is F err a n te and Rack off ’s metho d [7][4, § 7.3][1 4 , § 4.2]. A related alg orithm was pr o pos e d b y L o os and W eispfenning [10][14, § 4.4]. Both these a lgorithms are based on the idea that a n existent ia lly qua n tified formula ∃ x F ( x ) with fr e e 10 v ariables y , z , . . . can be replaced b y F ( x 1 ) ∨ · · · ∨ F ( x m ) where x 1 , . . . , x m are expressed a s functions of y , z , . . . . In the ca se of F erra nte and Rack off, m is quadratic in the worst cas e in the leng th of the for mula, while for Lo os and W eispfenning it is linear. In both case s , the ov era ll complexity b ound is 2 2 cn . The w eak nes s of both algo rithms is that they never simplify formulas. This may explain that while their theo retical b ounds are better than ours, our algo- rithm is in practice mor e efficient, a s s hown in the next subsectio n. One co uld at first assume that the complexity bounds for our algo rithm are asymptotically worse than F er r an te and Rackoff ’s. Our algorithm, howev er, outputs res ults in CNF or DNF form, while F errante and Rack off ’s algo rithm do es not. If we add a step of tra nsformation to CNF or DNF to their algor ithm, then w e also obtain a triple exponential b ound. 5.2 Practical results depth 14 depth 15 depth 16 Solv ed Avg O-o-m Solv ed Avg O-o-m Solved Avg O- o-m Mjollnir 100 1.6 0 94 9.8 0 73 35.3 0 Mjollnir (mo d1) 94 8.2 3 80 27.3 7 39 67.1 25 Mjollnir (mo d2) 100 3.8 0 91 13.9 0 65 39.2 0 Mjollnir Lo os-W. 93 1.77 4 90 6.42 5 62 17.65 27 Pro of-of-concept 94 1.4 0 86 2.2 0 55 17.7 0 Mjollnir F erran te-R. 51 18.2 41 23 23.2 65 3 7.3 85 Pro of-of-concept 94 1.4 0 86 2.2 0 55 17.7 0 Lira 14 102.4 83 3 77.8 94 1 8 9 5 Redlog ( rlqe ) 92 13.7 0 53 27.4 0 27 33.5 0 Ma thema tica 6 30.2 0 1 255.7 0 1 19.1 0 T able 2: Benchmarks on 3 × 10 0 random insta nces genera ted using randpr sb , with form ula depths n re spectively 14, 1 5 and 16 (obtained by ran dprsb 0 7 - 10 10 n i ) where i range s in [0 , 9 9 ]). The ta ble shows the n umber of instances solved within the timeout pe riod out o f the prop osed 100 , the av era ge time sp en t per so lv ed instance, and the n umber of instances resulting in out-o f-memory . W e benchmarked several v ariants of our method a gainst other alg orithms: Mjollni r is the algo rithm des c ribed in § 3, implement ed o n top of SMT solver Yices 3 and the NewPolk a p olyhedron pack a ge fro m Apron 4 , or o ption- ally the P a rma Polyhedra Library (PPL 5 ). Profiling show ed that most of the time is sp en t in the SMT solver, so p erforma nc e differences b etw een NewPolk a and PPL ar e neg ligible. Pro of-of-concept is an early version of the same algorithm, implemen ted on top of a rudimentary SMT solver a nd the PPL. The SMT algo rithm used is simple a nd lazy: the SMT problem is turned into SA T b y replacing each atomic inequality b y a prop ositional v ariable, and the SA T problem is input into Minisa t . A full SA T solution is o btained, then tested for emptiness by solving a linear pr o gramming problem: finding a vector of 3 http://y ices.csl.sri.com / 4 http://a pron.cri.ensmp.f r /library/ 5 http://w ww.cs.unipr.it/p p l/ 11 co efficien ts suitable a s a contradiction witness for F ark as’ lemma . If a witness is found, it yields a contradictory conjunction, whose negation is added to the SA T pro blem and SA T is restar ted. Mjollni r (mo d1) is the ALL-SA T then pro jection algo rithm from § 4.1. It is inv oked by option -no- block-pr ojected-m o del . Mjollni r (mo d2) is the a lgorithm fr o m § 4 .2; it is inv oked by o ption - add- blocki ng-to-g . Mjollni r F erran te-Rac koff implements [7][4, § 7.3 ]. Mjollni r Lo os-W eis pfenning implemen ts [10]. Lira 6 is based on B ¨ uchi automata and handles both P resburger a rithmetic (in- teger linear inequalities) a nd ra tional linear inequalities. Mathematica 7 is a g eneral-purp ose symbolic alg ebra pa ck age. Its R educe fonction app ears to implement CAD [5], an algo rithm suitable for non- linear inequa lities int e rpreted in the theory of real clo sed fields, though it is difficult to know what exactly is implemented b ecause this pro g ram is closed source. Redlog 8 is a symbo lic formula pack ag e implemen ted o n top of the co mputer algebra system Reduce 3.8 . 9 Redlog implements v ario us a lgorithms due to V o lk er W eispfenning and his group [11]. T able 1 compar es these v ar ious implemen tations on a few benchmark exam- ples 10 coming from t wo s ources: 1. Examples pro duced from pro blems o f program analysis following o ur metho d for the parametric computatio n of leas t inv aria n ts. [12] T o s umma- rize, each formula express es the fact that a set of progra m sta tes (such as a pro duct of in terv als for the n umerica l v ariables) is the least inv ar ian t o f a progra m, o r the stronges t p ostcondition if ther e is no fixed p oin t inv olved. Most of the exa mples, b eing extra cted by hand from simple s ubprograms, were eas ily solved a nd thus did not constitute go od b enc hmar ks, but o ne of them, defining the least inv ariant of a rate limiter, proved to b e toug her to so lv e, and we s elected it as a b enc hma r k. W e hav e t wo versions of this example: the first for a rate limiter op erating ov er real num b ers ( “r. lim R ” ) the s econd ov er flo ating-po in t num b ers, abstracted using real num b ers ( “r. lim float” ), and considera bly tougher to pro cess than the real example. 2. Examples pro cured fro m the Lira designers ( prsb23 and b lowup5 ). Memory consumption stayed mo dest for all examples ( < 15 MiB), except for r. lim float. P r ofiling sho wed that most of the time is s pent in the SMT-solver and only a few per cen ts in the pro jection algo rithm. The fact that the pro of-of- concept implementation, with a very naive SMT-so lver, pe r forms decently on an 6 http://l ira.gforge.avacs . org/ 7 http://w ww.wolfram.com/ 8 http://w ww.algebra.fim.u n i- passau.de/~redlog/ 9 http://w ww.uni- koeln.de/REDUCE/ 10 Av ailable from http://ww w- ver imag.imag.fr/~monniaux/download/linear_qe_benchmarks.zip . 12 example where o ther algorithms exhaust memory s ho ws that the p erformance of our algorithm ca nnot b e solely explained b y the go o d quality of Yices . T able 2 co mpares the v arious algorithms on ra ndom examples. W e then used the LIRA team’s randp rsb to ol 11 to genera te 1 00 ra ndom insta nces, b y changing the seed of the rando m n umber generato r from 0 to 99 , for each of three v a lues (1 4 , 15, 16) of the depth para meter, which measures complexity . 12 The pr o grams were then tested with b oth a 1.8 GiB memory limit and a timeout of five minutes. It is clear from T ab. 2 that Mjol lnir -no-a dd-blocking-to-g is the most efficient of the tested to ols. 6 Conclusion and F u ture W ork W e hav e pr opos ed a new quantifier elimination algorithm fo r the theory o f linear inequalities over the real or ra tional num b ers, a nd inv estig a ted p ossible v ariants. Our mo tiv ation was the practical a pplication of a recent result of ours on pro- gram analys is, stating that formulas for computing the leas t inv a riants of certain kinds of systems can b e obtained through quan tifier elimination [12]. This alg orithm is efficient on examples obtained from this pro gram analy- sis tec hniq ue, as well as other exa mples, whereas e arlier published a lgorithms, as well as several commercial pack ages, all exhaust time or memor y res ources. Our alg orithm leverages the recent pr ogresses on satisfiability modulo theor y solvers (SMT) a nd, cont r ary to older a lgorithms, p erforms on-the-fly simplifi- cations of formulas that keep for m ula sizes manag eable. Our a lgorithm also per forms b etter than a straight application of SMT solvers (ALL-SA T follow ed by pro jection). Our a lgorithm is describ ed for r ational or r eal linear arithmetic, but it can be extended to any theor y for which there is an e fficien t satisfiability testing algorithm for unquantified formulas and a re asonably e fficie n t pro jection al- gorithm for conjunctions . Among ex tensions that could be interesting from a practical p oint of v iew w o uld b e on the o ne hand the nonlinea r case for rea l arithmetic (p olynomials ), a nd on the other hand the mixed in teg e r / real prob- lems. Of course, nonlinear in tege r a rithmetic cannot b e considered, since P e ano arithmetic is undecidable. T arski show ed that the theory of the r e a l closed fields (inequa lities of p o lyno- mial expressions) a dmits quantifier elimination, [16] how ever his algorithm had impractical (non-ele men tary ) complexity . Later, the cylindric al algebr aic de- c omp osition (CAD) [2, Ch. 11] metho d was introduced, with doubly exp onential complexity , which is unavoidable in the worst case [2, § 1 1.4]. Our exp eriments with b oth Ma thema tica a nd Qepcad , bo th of which implemen t CAD, as well as with Reduce / Redlog , which implement v arious algorithms for quantifier elimination, show ed us that combinatorial blowup o ccurs very quickly . F o r such techn iques to be interesting in pra ctice, pra ctical complexity s hould b e low ered. Perhaps o ur technique could help. There ar e, how ever, significant difficulties in that resp ect. Our tech niq ue star ts with some single mo del of the target formula ov er the rational num b ers; but a system of nonlinear inequalities needs not hav e rational mo dels when it is not full-dimensio na l (for insta nce, X 2 = 2). Our tech- 11 http://l ira.gforge.avacs . org/toolpaper/randPrsb.hs 12 W e used the command line randprsb 0 7 -10 10 n i where n is the depth parameter (here, 14, 15 or 16) and i ranges in [0 , 99]. 13 nique reduce s the geometrical computations to computations on co njunctions; but in the nonlinear ca se, single inequalities can b e reduced to disjunctions. As an example, X 2 ≥ 4 is reduced to X ≤ − 2 ∨ X ≥ 2. Most impo rtan tly , our techn ique relies at se v er a l steps on the a v ailability of a decision pro cedure that stays efficient even when the answer is nega tive. Regarding the mixed integer / r e a l problems, the Lira to ol implements quantifier elimination using a weak form of B ¨ uchi automa ta matc hing the b -ary expression of the integers or reals, where b is an ar bitrary base. [3] The output of the pro cess is a n automaton and not a reada ble for mula. While it is possible to decide a closed for m ula, and to obtain one model from a satis fia ble non-closed formula, it is an op en problem how to efficiently reconstr uc t a quantifier-free formula from the resulting a utomaton. The automaton construct is unsuitable for large co efficients (as our examples obtained from the analysis of floating- po in t pro grams). Even on examples with small co efficien ts, the to ol was unable to complete qua ntifier elimination without blowing up. W e think ther efore that it would b e in ter esting to be able to apply our technique to the mixed integer / rea l problems, but ther e are difficulties: the algorithms on in teg er po ly hedra are considerably more complex tha n on rationa l p olyhedra. A classical ob jectio n to automatic progra m analysis to ols meant to prove the absence of bugs is that these to ols could themselves contain bugs. Our metho d uses complex a lgorithms (SMT-solving , poly hedron pro jection) as sub- pro cedures. W e c o nsider developing tec hniq ue s so that the algor ithm outputs easily-chec k able pro ofs or “pr oof witnesses ” of the cor rectness of its computation. F urthermore, we show ed in e arlier publica tions [12] that certain progra m ana ly- sis tasks were equiv alent to quan tifier elimination pro ble ms ; that is , an effective static analyzer ca n b e extracted fro m the qua n tifier-fr ee form of a n ana lyzer sp ecification. This therefore s ug gests a new w ay for writing safe static analyz- ers: instead of painstaking ly wr iting an analyzer , then pro ofs of corr ectness in a pro of assis ta n t [15], o ne could formulate the a nalysis problem as an equiv alent quantifier eliminatio n problem, with a re lativ ely simple pro of of equiv alence, then apply a “cer tified” quantifier elimination procedur e in o rder to extract the effective analyzer . References [1] Rob erto Bagnar a, Patricia M. Hill, and Enea Z affanella. The Parma Poly- he dr a Libr ary, version 0.9 . av ailable fr o m ht tp://www .cs.unipr.it/ppl . [2] Saugata Basu, Richard Pollack, and Marie-F r an¸ coise Roy . Algorithms in r e al algebr aic ge ometry . Algorithms and co mputation in mathematics. Springer, 2003. [3] Eisinger J. Becker B., Dax C. and Klae dtk e F. LIRA: handling constraints of linear arithmetics ov er the integers and the reals . In Computer Aide d V erific ation (CA V) , num b e r 45 90 in LNCS, pages 312–31 5, 2005. [4] Aaron R. Bradley and Zohar Manna. The Calculus of Computation: De ci- sion Pr o c e dur es with Applic ations to V erific ation . Springer, Octobe r 20 07. [5] Bob F. Caviness and J e r em y R Jo hnson, editor s. Quantifier elimination and cylindric al algebr aic de c omp osition . Springer, 1998. 14 [6] Leonar do de Moura, Harald Rueß, and Maria Sorea . Lazy theore m proving for b o unded model chec king ov er infinite domains. In Pr o c e e dings of the 18th International Confer enc e on Automate d De duct ion , volume 2392 of L e ctur e Notes in Computer Scienc e , pa ges 4 38–455 . Spring er-V erlag, July 2002. [7] Jeanne F errante a nd Charles Rack off. A dec is ion pro cedure fo r the first order theory of r eal addition with order . SIA M J ournal of Computation , 4(1):69–7 6, Ma rc h 1975. [8] H. Ganzing er, G. Hagen, R. Nieuw enhuis, A. Oliveras, and C. Tinelli. DPLL(T): F ast Decision P ro c edures. In R. Alur and D. Peled, editors, 16th International Confer enc e on Computer Aide d V erific ation, CA V’04 , volume 3114 of L e ctur e N otes in Computer Scienc e , page s 175 – 188. Spring e r, 200 4 . [9] Jean-Lo uis Imbert. F ourier ’s elimina tio n: Which to c ho ose? In Principles and Pr actic e of Constr aint Pr o gr amming , pa ges 117–129 , 19 93. [10] R ¨ udiger Loo s and V olker W eispfenning. Applying linear qua n tifier elimi- nation. The Computer Journal , 36(5):450 –462, 1993. [11] Ruediger Lo os a nd V olker W eispfenning. Applying linear qua n tifier elim- ination. The Computer J ournal , 36(5):45 0 –462, 199 3. Specia l issue o n computational quan tifier elimination. [12] David Monniaux . Optimal a bs traction on rea l- v a lue d progr ams. In Gilb erto Fil ´ e a nd Hanne Riis Niels o n, editors, Static analysis (SAS ’07) , num b er 4634 in LNCS, pa ges 10 4–120. Springer , 2007 . [13] Rob ert Nieu wenh uis and Alber t Oliveras. F as t Cong ruence Clos ure and Extensions. Inf. Comput. , 2005(4):557 –580, 20 07. [14] T obia s Nipk ow. Linear quantifier e limination. In Automate d re asoning (IJCAR) , v olume 519 5 of LNCS , pa ges 18–33. Spr inger, 20 08. [15] David Pichardie. Interpr´ etation abstr aite en lo gique intuitionniste : extra c- tion d’analyseurs Java c ertifi´ es . PhD thesis, Universit ´ e Rennes 1, 2005. [16] Alfred T arsk i. A D e cisio n Metho d for Elementary Algebr a and Ge ometry . Univ ersity of California Press, 1 951. 15