Pessimistic Testing

P essimist ic T esting Ernie Cohen Microsoft Octob er 26, 2018 Abstract W e prop ose a new app roac h to t esting conformance t o a n ondeterministic sp eci- fication, in whic h testing pro ceeds only as long as increased test cov erage is guaran- teed. In testing that a system meets a nondeter ministic s pec ific a tion [1], it is usually as- sumed that the system is fa ir to each transitio n of the sp ecificatio n (i.e., the system will make every p ossible nondeterministic c ho ice if given enough opp or tunities). But in fact, some trans itions mig h t b e unlikely or impo ssible for a given implementation. When this happ ens, common mode l- based testing practices (e.g., following a precomputed tour of the state spac e ) often lead to wasted test cycles and po o r test cov er age. W e prop ose an alternative appro ach, in whic h the tester uses a dyna mically computed stra teg y that is guar anteed to event ua lly increas e cov erag e; testing stops a s so on as the system ha s a strategy to av oid further coverage. As usual, we cast the test problem as a game; her e, it is conv eniently repr esented as a (directed) h yp erg raph. A hyp er gr aph is given b y a set o f vertices and a s e t of (hyper)edg es. Each edg e is given by a head vertex and a set o f tail vertices; we say it is incident to its head. An edge is r e achable iff a ll of its tail vertices are reachable, a nd a vertex is reachable iff one of its incident edges is r eachable (as usual, taking the minimal solution). The r ank of a rea chable edge is the maximum of the ra nks of its tail vertices (0 if the tail is empt y), and the rank of a reachable vertex is one plus the minimum rank of its rea chable incident edges. In the test context, the hyper g raph vertices ar e system states, ea ch hyperedg e repre- sents a po s sible test stimulus, the head of the hyp eredge is the state in which the s tim ulus can b e delivered, and the tail o f the h yp eredge gives the sta tes to which the system is allow ed to transition under the stim ulus. T o keep tra ck o f which states hav e b een ex- plored by the test, we add trivial edges (with empty tails) inciden t on each state (other than the initial state). When the system first visits a s tate, this incident edge is r emov ed, “marking” the state. Thu s , the test s ta te consists of a h yp erg raph and a current state (an unmar ked vertex of the hyper graph), and a mov e of the testing game consists of the tester cho osing an edge inciden t on the current state and the system cho osing a new current state fro m the tail of the edge (mar k ing the state if it is unmarked). Cov erag e is measured a s the num ber of mar ked states. A key o bserv ation is that the tes ter has a strategy to incr ease cov er age iff the curr ent state is reachable. If the curren t state is unreachable, the system can preven t further marking b y always choo sing a n unreachable s uccessor sta te. Conv ersely , if the current 1 state is r eachable, the tester’s str a tegy is to alwa ys choose a reachable edg e inciden t on the current state of minimal r ank; this res ults in either mov ement o f the sys tem to a state of lower r ank or marking o f the new state. Since rank is b ounded b elow by 0, so me state is ev entually marked. (In fa ct, this strategy is optima l in the sense of minimizing the upper b ound on the num b er o f mov e s b efor e the next marking.) T o op erationaliz e this test strategy , w e need to maintain (under game mo ves) the ranks of r e achable edges incident on the current state. The obvious decremental hyper- graph reachabilit y algor ithm [2] maintains ra nks for all states and edges in total time O ( E + S · H ), where E is the nu mber of states marked, S is the total n umber of states, and H is the s ize of the h yp ergr a ph. How ever, since ranks can only increa se, and this algorithm upda tes ranks in incr easing r ank order, we can delay up dating ranks for edges and states whenev er they excee d that of the current system state. W e c a n also delay adding to the g raph edges incident on unmar ked states (“dead” edges). This improv es the worst-case complexity to O ( E + R · H ′ ), where R is the maximum rank assumed by the s ystem state pr io r to termination (i.e., the maximum num b er of stimuli b etw een exploratio ns ), and H ′ is the total size of the live edg es. In the w or st case R is E , but in practice, R grows mu ch more slowly in E ; for example, fo r random gra phs o f b ounded degree, R is O (log ( E ) ). In exp e riments tes ting co nformance of the Microsoft Hyp erv i- sor to its functional specifica tion [3 ], the h yp erg raph computation is dominated by test instrumentation, even on tests with ov er 10 5 states. There a re several potentially useful trans formations of the test hypergr aph. First, other kinds of tes t coverage (edge coverage, bra nch coverage, etc.) ca n b e o btained using standard techniques. Second, in the co mmon case where an op era tion can fail without a state change, the head o f the h y p er edge a ppea rs in the tail, which makes the edge un us a ble in the strategy above. W e typically wan t to a llow s uch an edge to be used as if the failure were imp ossible, at least until a test hits the failing case at least once; we can achiev e this b y adding a new state and edg e to brea k the self-lo o p. Thir d, since the worst-case computationa l co st grows with the maximum r ank a s sumed b y the system state, it can b e adv antageous to compress a lo ng s equence of prepar atory op erations into a single edge. Finally , because dead edg es a re no t added to the gra ph during co mputation, the metho d here is fully compatible with la zy generation o f the state space, wher e edges incident on a state are g enerated only when the state is marked. References [1] [1] Lev Nachmanson, Mar gus V eanes , W olfram Sch ulte, Nikolai Tillmann, W olfgang Griesk amp. Optimal strategies for testing nondeter ministic sy stems . ISST A 2004 : 55- 64 [2] [2] G. Ausiello, P . G. F ranciosa, a nd D. F rigio ni, Partially Dynamic Maintenance of Minim um W eight Hyp er paths. J. of Discrete Algorithms, 3(1):27 -46, 2005. [3] [3] Micro soft Hyp erv is or F unctiona l Sp ecification. Av ailable from www.microsoft.co m. 2