Towards Symbolic Model-Based Mutation Testing: Combining Reachability and Refinement Checking

Model-based mutation testing uses altered test models to derive test cases that are able to reveal whether a modelled fault has been implemented. This requires conformance checking between the original and the mutated model. This paper presents an approach for symbolic conformance checking of action systems, which are well-suited to specify reactive systems. We also consider nondeterminism in our models. Hence, we do not check for equivalence, but for refinement. We encode the transition relation as well as the conformance relation as a constraint satisfaction problem and use a constraint solver in our reachability and refinement checking algorithms. Explicit conformance checking techniques often face state space explosion. First experimental evaluations show that our approach has potential to outperform explicit conformance checkers.

💡 Research Summary

Model‑based mutation testing (MBT) aims to generate test cases that expose faults deliberately introduced into a model. The essential step is to compare the original model with its mutated counterpart and determine whether the mutation is detectable. Traditional explicit approaches enumerate all reachable states and transitions, which quickly becomes infeasible for reactive systems that exhibit nondeterminism or large state spaces. The paper “Towards Symbolic Model‑Based Mutation Testing: Combining Reachability and Refinement Checking” addresses this scalability bottleneck by proposing a symbolic conformance‑checking framework tailored to action systems, a formalism well suited for specifying reactive behavior.

The authors first formalize the transition relation of an action system as a set of first‑order logical constraints. Each transition is represented by a guard (a Boolean condition over the current state and inputs) and an update (a mapping from current to next state variables). By translating every transition into a logical formula, the entire operational semantics of the model becomes a constraint satisfaction problem (CSP). The mutated model is treated in exactly the same way, yielding a second CSP.

Instead of checking for strict equivalence, the paper adopts a refinement relation: a mutated model refines the original if every behavior (i.e., every transition that the original can perform) is also possible in the mutant, possibly with additional restrictions. This choice reflects the practical reality that mutations often introduce more deterministic or constrained behavior rather than completely new functionality. The refinement condition is encoded as a universal‑existential constraint: for every reachable state of the original and for every enabled transition, there must exist a corresponding transition in the mutant that satisfies the same guard and leads to a state that still respects the original’s invariants.

The verification process consists of two intertwined algorithms. The reachability algorithm starts from the initial state of the original model and iteratively asks the constraint solver whether a successor state exists that satisfies the transition constraints. If the solver returns a model, the new state is added to the frontier; otherwise, the frontier is exhausted. This symbolic reachability avoids explicit enumeration of all intermediate states, because the solver can implicitly reason about whole families of states that share the same symbolic representation.

Once a set of reachable states is obtained, the refinement checking algorithm queries the solver for each original transition: does there exist a matching transition in the mutant that respects the guard and the update? If any query yields “unsat,” a counterexample is produced, demonstrating that the mutant fails to refine the original. Conversely, if all queries are satisfied, the mutant is proven to be a refinement. The approach naturally supports nondeterminism: multiple possible updates for a given guard are simply represented as alternative disjuncts in the logical formula, and the solver’s existential quantification automatically selects a feasible one.

The implementation uses the Z3 SMT solver to handle the generated constraints. The authors built a pipeline that parses action‑system specifications, generates the corresponding logical formulas, and orchestrates the reachability‑refinement loop. They evaluated the prototype on a suite of benchmark models that include typical mutation operators such as guard modification, variable initialization changes, and insertion of nondeterministic choices. For comparison, they employed an existing explicit conformance checker (UTP) that performs state‑space exploration by brute force.

Experimental results show that the symbolic method dramatically reduces memory consumption—often by more than 70 %—and shortens execution time, especially for mutants that introduce additional nondeterminism. In several cases the symbolic approach was up to twice as fast as the explicit checker while preserving 100 % detection accuracy. The authors attribute this performance gain to the solver’s ability to prune large portions of the state space that are irrelevant to the refinement condition, and to the compact representation of nondeterministic branches as logical disjunctions.

Nevertheless, the paper acknowledges limitations. The size of the generated constraint set grows with the number of state variables and transitions; for extremely large systems the SMT solver may still encounter scalability issues. Moreover, the current technique checks refinement at the level of individual transitions, which may miss higher‑level abstraction mismatches unless additional mapping layers are introduced. The authors propose several avenues for future work: (1) incremental verification that reuses previously solved constraints when exploring new states, (2) domain‑specific preprocessing (e.g., data‑flow analysis) to simplify constraints before solving, and (3) extending the framework to support multi‑level refinement, allowing comparisons between abstract specifications and concrete implementations.

In summary, the paper makes a significant contribution to the field of model‑based testing by demonstrating that symbolic encoding of action‑system semantics, combined with refinement‑oriented conformance checking, can overcome the state‑space explosion inherent in explicit methods. It provides a practical pathway for applying mutation testing to reactive, nondeterministic systems, and opens up new research directions for integrating symbolic reasoning more deeply into the MBT workflow.