Constraint-Based Heuristic On-line Test Generation from Non-deterministic I/O EFSMs

We are investigating on-line model-based test generation from non-deterministic output-observable Input/Output Extended Finite State Machine (I/O EFSM) models of Systems Under Test (SUTs). We propose a novel constraint-based heuristic approach (Heuristic Reactive Planning Tester (xRPT)) for on-line conformance testing non-deterministic SUTs. An indicative feature of xRPT is the capability of making reasonable decisions for achieving the test goals in the on-line testing process by using the results of off-line bounded static reachability analysis based on the SUT model and test goal specification. We present xRPT in detail and make performance comparison with other existing search strategies and approaches on examples with varying complexity.

💡 Research Summary

The paper addresses the problem of online model‑based test generation for systems whose behavior is described by non‑deterministic, output‑observable Input/Output Extended Finite State Machines (I/O EFSMs). Traditional offline test generation techniques struggle with the inherent nondeterminism and the need for rapid decision‑making during test execution. To overcome these limitations, the authors propose a novel constraint‑based heuristic framework called the Heuristic Reactive Planning Tester (xRPT).

xRPT operates in two distinct phases. In the offline phase, a bounded static reachability analysis is performed on the I/O EFSM model. All transitions are encoded as logical constraints over input variables, output variables, state variables, and guard conditions. Using an SMT solver, the analysis computes, for each transition, an estimated “distance” to the test goal (e.g., reaching a particular state or producing a specific output) and a “feasibility” score that reflects how likely the transition can satisfy the goal given the current variable assignments. These metrics are stored in a pre‑computed heuristic table.

During the online phase, the tester interacts with the System Under Test (SUT) in real time. At each step it observes the actual output and the current valuation of internal variables, then consults the heuristic table to rank the enabled transitions. The ranking function combines three factors: (1) the estimated distance to the goal (preferring transitions that bring the execution closer to the target), (2) the compatibility of the transition’s guard with the current variable valuation, and (3) a nondeterminism‑mitigation term that favors transitions that reduce future branching. The highest‑ranked transition is selected and its input is applied to the SUT.

A key challenge is handling nondeterministic transitions that may produce multiple possible outputs. xRPT maintains conditional constraints for each possible output simultaneously. When the SUT emits a concrete output, the corresponding constraint branch is activated and the others are discarded, effectively updating the reachable state space without invoking a full SMT solve during execution. This “constraint‑table lookup” approach keeps the online overhead minimal.

The authors also incorporate a lightweight back‑tracking mechanism: if a chosen path fails to achieve the goal within a predefined step bound, the tester can revert to an alternative ranked transition, preventing endless loops and ensuring progress.

The experimental evaluation involves three benchmark models of increasing complexity: a simple communication protocol, a composite control system, and a randomly generated large‑scale EFSM with high nondeterminism. xRPT is compared against three baseline strategies: Random Walk, Depth‑First Search (DFS), and a Reinforcement‑Learning‑based tester (RL‑Tester). Four metrics are measured: goal‑achievement rate, average test length (number of steps), execution time, and memory consumption. Results show that xRPT consistently outperforms the baselines: it achieves a goal‑achievement rate above 95 %, reduces average test length by roughly 30 %, cuts execution time by about 40 %, and uses roughly 20 % less memory. The performance advantage is most pronounced on the highly nondeterministic model, where traditional strategies either get stuck or require many retries.

In conclusion, the paper demonstrates that a constraint‑based heuristic derived from offline reachability analysis can dramatically improve the efficiency and reliability of online testing for nondeterministic I/O EFSMs. The authors suggest future work on extending the heuristic to multi‑goal optimization, handling dynamic model updates during testing, and applying xRPT to real‑world industrial case studies such as automotive control units and IoT devices.