New Types of Repair Analyses

The repair analysis presented in  and implemented in the prototype version of encodes a TDT as a constraint system in linear real arithmetic. It computes syntactic correct modifications of the underlying NTA by introducing bound variation variables $`\bvvar`$. For example, possible bound modifications for a clock bound $`x \leq 2`$ are expressed by a modified clock bound $`x\leq 2+\bvvar`$. The repairs are computed by solving a partial SMT problem on the TDT constraint system, involving soft-assert constraints on the bound variation variables. No repair is computed whenever the soft assertion $`\bvvar=0`$ holds, otherwise the computed value of $`\bvvar`$ characterizes the repair. In the following we sketch the new types of repairs implemented in . For a more comprehensive description, which space limitations do not allow us to provide here, we refer to .

Operator Variation Repair Analysis.

This analysis is motivated by the assumption that a wrong comparison operator in a location invariant or transition guard may cause a property violation. We assume for the repair encoding that the operators $`\sim`$ are indexed according to their order in the sequence $`\langle \; <, \leq, =, \geq, >\> \rangle`$. The possible repairs are encoded by a fresh variation variable $`\bvvar^{ov}_\idx`$ where the value of $`\bvvar^{ov}_\idx`$ is the index of the corresponding comparison operator. If $`x < 4`$ is computed as a repair, then $`\bvvar^{ov}_\idx = 1`$.

We define appropriate operator variation constraints $`\mathmbox{{\cal I}^{ov}}`$ and $`\mathmbox{{\cal G}^{\ov}}`$ with the help of an n-ary exclusive or operation $`\bigoplus\limits_{i = 0 \ldots n}f_i`$ which is satisfied iff exactly one of the formulas $`f_i`$ is true:

\begin{align*}
\mathmbox{{\cal I}^{\ov}} \equiv \; &
\bigwedge \limits_{(c \sim \beta) \in I(l_j)}
  \bigoplus\limits_{0 \leq k \leq 5}
  c\sim_k \beta \wedge
  c + \delta_j\sim_k \beta \wedge
   \bvvar^{\ov}_\idx = k.\\
%
\mathmbox{{\cal G}^{\ov}} \equiv \; &
\bigwedge \limits_{(c \sim \beta) \in g_j} 
\bigoplus\limits_{0 \leq k \leq 5} 
c + \delta_j\sim_k \beta \wedge \bvvar^{\ov}_\idx = k.
\end{align*}

Using this repair analysis, finds two admissible repairs for the example in Figures [fig:client] and [fig:db] that replace the comparison operator in the clock constraint $`w>=1`$ by $`<`$ or $`<=`$, respectively.

Clock Reference Repair Analysis.

This analysis aims to repair property violations resulting from errors that stem from the unintended use of a wrong clock variable. We enumerate all the positions of clock variables in clock bound constraints using index $`\idx`$ and all clock variables using index $`k`$. We then introduce for every position $`\idx`$, a fresh variation variable $`\bvvar^{\cv}_\idx`$ whose value $`k`$ indicates the clock $`c_k`$ to be used at that position in the repaired model. For example, if $`y \leq 2`$ is a repaired constraint, where the position of $`y`$ in the constraint has index $`3`$ and clock $`y`$ has index $`1`$, then $`\bvvar^{\cv}_3 = 1`$.

We define the appropriate clock variation constraints $`\mathmbox{{\cal I}^{\cv}}`$ and $`\mathmbox{{\cal G}^{\cv}}`$:

\begin{align*}
\mathmbox{{\cal I}^{\cv}} \equiv \; &
\bigwedge \limits_{(c \sim \beta) \in I(l_j)}
\bigoplus\limits_{0 \leq k \leq |C|} (c_{k}\sim
\beta)\wedge (c_{k} + \delta_j\sim \beta)\wedge(\bvvar^{\cv}_\idx = k)\\
%
\mathmbox{{\cal G}^{\cv}} \equiv \; &
\bigwedge \limits_{(c \sim \beta) \in g_j}
\bigoplus\limits_{0 \leq k \leq |C|} (c_{k} + \delta_j\sim
\beta)\wedge(\bvvar^{\cv}_\idx = k)
\end{align*}

Applying this repair analysis to the examples in Figures [fig:client] and [fig:db], finds 13 admissible clock reference modification repairs, each involving two modifications. Nine repairs exchange $`y`$ in the constraints $`y \leq 1`$ and $`y \geq 1`$ by a selection from the set of clocks $`z`$, $`x`$ and $`w`$. Four repairs exchange $`y`$ in the constraint $`y \leq 1`$ by $`w`$ or $`x`$, and $`w`$ in the constraint $`w\geq 1`$ by $`y`$ or $`z`$.

Reset Clock Repair Analysis.

This analysis aims to repair a property violation by adding or removing clock resets. We introduce a variation variable $`\bvvar^{\rv}_{\idx,j}`$ for each clock $`c_\idx`$ and the transition leaving location $`\lambda_j`$ in the TDT. The reset status in the extended constraint system is inverted when $`\bvvar^{\rv}_{\idx,j} \neq 0`$: if $`c_i`$ was not reset before, it will now be reset, and vice versa.

This is encoded by the clock reset variation constraints $`\mathmbox{{\cal R}^{\rv}}`$ and $`\mathmbox{{\cal D}^{\rv}}`$:

\begin{align*}
\mathmbox{{\cal R}^{\rv}} \equiv \; &
\bigwedge
\limits_{c_i \in \mathmbox{\textit{reset}}(\lambda_j)}
  c_{i, j+1} = \left \{
  \begin{aligned}
    &0, && \text{if}\ \bvvar^{\rv}_\idx = 0 \\
    &c_{i} + \delta_j, && \text{otherwise}
  \end{aligned}\quad . \right .
\\
%  
\mathmbox{{\cal D}^{\rv}} \equiv \; &
\bigwedge
\limits_{c_{i} \not\in \mathmbox{\textit{reset}}(\lambda_j)}
  c_{i, j+1} =  \left \{
  \begin{aligned}
   &c_{i} + \delta_j, && \text{if}\ \bvvar^{\rv}_\idx = 0 \\
    &0, && \text{otherwise}
  \end{aligned}\quad . \right.
\end{align*}

Applying the reset repair analysis to the examples in Figures [fig:client] and [fig:db], finds four admissible repairs. One repair removes the reset of clock $`y`$, another removes the reset of clock $`z`$ and two repairs add a reset of clock $`x`$ either on the transitions towards the state reqProcessing or the transition towards the state serReceiving.

Urgent Location Repair Analysis.

This analysis aims to repair cases where a faulty usage of urgent locations, which are always left with zero delay after entering, causes a property violation. Urgency of a location is modeled in the TDT constraint system by setting the location delay $`\delta_j`$ to $`0`$. We define a fresh variation variable $`\bvvar^{\uv}_\idx`$ for a location $`\lambda_j`$. For $`\bvvar^{\uv}_\idx \not= 0`$, the urgency for a location $`\lambda_j`$ is inverted.

We encode this idea using the following urgency variation constraint $`\mathmbox{{\cal U}^{\uv}}`$:

\begin{align*}
\mathmbox{{\cal U}^{\uv}} \equiv \; &
\bigwedge\limits_{j \in \mathmbox{\text{\em urgent}(S)}}
 \bvvar^{\uv}_{i} = 0 \implies \delta_j = 0
%\\
\ \ \wedge
\bigwedge\limits_{j \not\in \mathmbox{\text{\em urgent}(S)}}
 \bvvar^{\uv}_{i} \not= 0 \implies \delta_j = 0.
\end{align*}

Applying the urgency location repair analysis to the examples in Figures [fig:client] and [fig:db], finds two inadmissible repairs. The first one makes the state reqAwaiting urgent, and another repair makes the state serReceiving urgent.

Introduction

A reactive system with requirements pertaining to its timing behavior is often modeled as a network of timed automata (NTA) . Whether a timing requirement holds in an NTA can be analyzed by timed model checkers such as Uppaal  or opaal . In case of a requirement violation, a model checker returns a timed counterexample, also called a timed diagnostic trace (TDT). Until now, developers must manually identify and correct such violations by analyzing the generated TDTs. It is therefore desirable to support this process by an automated tool set that not only determines whether timing requirements are met, but also proposes syntactic repairs of the NTA in case they are not.

In  we presented an automated repair analysis that analyzes a TDT obtained from the violation of a timed safety property and returns syntactic repair suggestions that avoid the concrete executions of the TDT violating the property. The analysis performs an additional admissibility check ensuring that the repaired model is functionally equivalent with the original NTA, which means that no action traces are added or omitted by the repair.

To illustrate the repair analysis consider the NTA in Figures [fig:client] and [fig:db]. It describes a client that sends a request req to a database db and expects to receive a response ser within $`4`$ time units after sending the request. The client contains a clock $`x`$ that measures the time delay between the request creation and the receiving of a response in location serReceiving. The NTA allows to execute a TDT that violates the property, illustrated as a sequence diagram with time intervals in Figure [fig:tdt]. A time interval in the sequence diagram denotes the minimal and maximal time delay for the message transmission and processing times in db, respectively. The repair computation analyzes the TDT and produces several syntactic repairs to the NTA that avoid the property violation. In , the computed repairs aim at the modification of clock bounds in location invariants and transition guards. An example of such a repair is to reduce the bound in the time constraint $`w \leq 2`$ from $`2`$ to $`1`$. The modified bound constrains the maximal transmit time of the req message so that the resulting NTA receives all responses within the expected time. This repair eliminates the problematic executions of the TDT in the original NTA without changing the functional behavior of the system, which is confirmed by an admissibility test defined in . However, in general, it may not be possible to repair the model using only clock bound alterations.

Contributions.

We present  , which extends the initial prototype implementation of the clock bound repair analysis presented in  to a more comprehensive NTA repair tool. Specifically, the extended tool implements new analyses that can suggest a whole range of repairs in addition to clock bound variation, such as modifying comparison operators in constraints, clock references, clock resets, and location urgency. Examples of new repairs computed for the model in Figure 1 are:

• Exchanging the comparison operator in the constraint $`w \geq 1`$ to $`w<1`$ ensures that the time to send a request is below $`1`$ time unit.

• An exchange of clock $`z`$ in $`z \leq 2`$ with clock $`y`$ restricts the time of processing and receiving the response to at most 2 time units.

• To reset the clock $`y`$ on the previous transition instead ensures that the time for sending and processing the request is below $`1`$ time unit.

• Making the location serReceiving urgent reduces the time to receive a response to 0.

We call a repair admissible if the repaired system is functionally equivalent to the unrepaired system. The repair analysis implemented in returns the complete set of admissible repairs.

The repair analysis combines concepts and algorithms from model checking, constraint solving, and automata theory. A real-time model checker is used to generate TDTs for a given NTA that violate a given timed safety property. translates the TDT into a linear real arithmetic constraint system. An SMT solver is used to compute a repair for the generated constraint system by solving a MaxSMT problem. An automata-based language equivalence test checks whether the repair is admissible in the NTA model. The collaboration between these subcomponents yields a complex tool architecture. We provide insights into the design and implementation of this architecture and the underlying infrastructure of supporting tools. We evaluate the new repair analyses by applying to a number of NTA models. We systematically inject different modifications in these correct models and compute repairs for the obtained faulty models, which results in at least one admissible repair for $`69`$% of the TDTs.

Related Work.

Other tools exist that compute repairs. The tool BugAssist  analyzes C-code by solving a MaxSMT problem. The tool ReAssert  checks a set of possible modification to repair broken unit tests. Angelix , S3  and SemFix  compute repairs by symbolic execution and constraint solving. SketchFix  is based on lazy candidate generation. All tools are not repairing broken time constraints. We are not aware of related work on tools for the repair of timed automata models. A more comprehensive overview of related work on automated repair is given in . A discussion of work related to the foundations of our repair analysis can be found in .

Evaluation

Evaluation Strategy.

In order to evaluate the repair analyses both qualitatively and quantitatively, we need to synthesize a set of faulty timed automata. To the best of our knowledge, no benchmark suite for faulty timed automata exists. We therefore create faulty models by using the fault seeding strategy from  which is motivated by ideas from mutation testing . Mutation testing evaluates the quality of a test suite for a given program by systematically corrupting program code and determining the ratio of corruptions that the test suite is able to detect. We apply the same principle to evaluate the quality of our repair technique. As proposed in , fault seeding modifies a single clock constraint so that the result is a set of models that violate a given property. During the seeding, the bound of a single clock constraint is modified by an amount of $`\{-10, -1, +1, +0.1M, +M\}`$, where M is the maximal clock bound occurring in a given model. Our observation was that making either small modifications that are close to the bound value or modifications in the order of the maximal bound value M often introduce actual errors in the model. We have extended fault seeding to the new types of repairs. In particular, fault seeding additionally exchanges the comparison operator in a clock constraint by $`\{<, \leq, =, \geq, >\}`$, swap a referenced clock with all other clocks occurring in the given model, modify the reset clocks of any transition, and switch for any location whether it is urgent. checks automatically whether a modified TA violates a given property. If this is the case, it performs all of the above defined repair analyses.

Results.

We applied fault seeding to the models in  and analyzed the obtained TDTs using the above described repair analyses implemented in . All analyses were performed on a computer with an i7-6700K CPU (4.00GHz), 60GB of RAM and a 64 bit Linux operating system. We summarize the results of the experiment per considered model (Table 1) and per type of considered repair (Table 2).

Column Sd contains the count of seeded faults that result in a number #T of faulty models. $`T_{\textit{UP}}`$ is the maximal time that Uppaal needs to create a TDT for the faulty models, and the longest TDT has a length of Ln. computed for the TDTs overall a number #R repairs of which #A are admissible. An admissible repair is found for #S of the TDTs. The computation effort for a repair analysis is given by the time $`T_{\textit{QE}}`$ for successful quantifier elimination, the number of timeouts #O of quantifier eliminations after 10 minutes, the average time $`T_{\textit{R}}`$ to compute a repair and the memory consumption $`M_R`$. The constraint system that Z3 solves has the count #Vr of variables and #Cn of constraints. The effort for the admissibility check is given in time $`T_{\textit{Adm}}`$ and memory $`M_A`$. All times are given in seconds and memory consumption in MB. Notice that we omit the columns pertaining to the fault seeding and TDT computation in Table 2 as they are irrelevant here.

Overall, seeded 4.508 faults. This resulted in 175 TDTs in total ($`60`$ TDTs due to bound modification, $`72`$ due to operator variation, $`27`$ due to changing the clock reference, $`8`$ due to complementing the reset of clocks and $`8`$ due to the switching of urgent locations). found 5,355 repairs, out of which 1,169 were admissible. It found at least one admissible repair for 122 of the TDTs. The maximal number of modified constraints in the admissible repairs computed for a single TDT using all types of analysis was $`25`$.

Interpretation.

Few of the seeded faults resulted in a property violation. seeded 4.508 faults which led to $`175`$ TDTs, thus only 3.9% of these faults result in a TDT. This supports the hypothesis that, in practice, often times only few time constraints have an impact on a property violation. computes at least one admissible repair by bound modification for $`85`$ (48%) of the $`175`$ TDTs, by operator variation for $`51`$ (29%), by clock reference for $`35`$ (20%), by clock reset for $`13`$ (7%) and by urgent location for $`37`$ (21%). Every analysis on its own computes less admissible repairs than the combination of all repair analyses, which solves $`122`$ (69%) of the $`175`$ TDTs. The largest number of modified constraints in all the admissible repairs for a single TDT was $`25`$, which is less than anticipated. This low number of modified constraints infer that, for the examples that we considered, only a few constraints of each TDT combined to admissible repairs. The number of modified constraints determines the number of possible repairs that have an impact on whether a property is violated or not. Since it was observed in  that the computational effort for the repair computation is largely determined by the quantifier elimination step, we expect that in light of the observed $`226`$ timeouts a more efficient quantifier elimination would lead to a significantly higher number of repairs. Furthermore, the number of timeouts, and thus the computation time needed for the repair, rises with the length of the analyzed TDT. The model SBR has the most timeouts with $`86`$ and the third longest trace with a length of $`84`$ steps. The model bando has the third most timeouts with $`46`$ and the longest trace. Obviously, the longer the TDT, the larger the resulting constraint system, leading to increased computational effort. The bando model has the largest constraint system with $`1,156`$ variables and $`8,144`$ constraints. The SBR model has the second largest constraint system with $`765`$ variables and $`1,211`$ constraints. The model FDDI has a shorter trace of length of $`11`$ and a much smaller constraint system with $`116`$ variables and $`272`$ constraints. From this we conclude that the complexity of a repair depends not only on the trace length, but also on the intrinsic complexity of the model. Modifying states from urgent to non-urgent during fault seeding resulted in only $`8`$ TDTs. This low number is due to the observation that the considered models contain only few urgent states. Modifying non-urgent states to urgent ones, however, did not lead to a single property violation resulting in a TDT. The rationale is that urgency ensures to leave a state immediately without a delay which leads to a restriction rather than a relaxation regarding the time budget spent along an execution trace. As a consequence, making a state urgent does not cause a property violation in many models since the type of the checked properties is typically time bounded reachability, and a restricted time budget does not make it more likely that the property is violated. We finally observe that the admissibility check requires more computation resources than the repair computation. The maximal memory used for the admissibility test was $`1,251\text{MB}`$ in contrast to $`37.16\text{MB}`$ for the repair computation. This is in line with our expectation since the admissibility test searches the state space of the full NTA, while the repair analyses only considers a single TDT.