Artificial table testing dynamically adaptive systems

Dynamically Adaptive Systems (DAS) are systems that modify their behavior and structure in response to changes in their surrounding environment. Critical mission systems increasingly incorporate adaptation and response to the environment; examples include disaster relief and space exploration systems. These systems can be decomposed in two parts: the adaptation policy that specifies how the system must react according to the environmental changes and the set of possible variants to reconfigure the system. A major challenge for testing these systems is the combinatorial explosions of variants and envi-ronment conditions to which the system must react. In this paper we focus on testing the adaption policy and propose a strategy for the selection of envi-ronmental variations that can reveal faults in the policy. Artificial Shaking Table Testing (ASTT) is a strategy inspired by shaking table testing (STT), a technique widely used in civil engineering to evaluate building’s structural re-sistance to seismic events. ASTT makes use of artificial earthquakes that simu-late violent changes in the environmental conditions and stresses the system adaptation capability. We model the generation of artificial earthquakes as a search problem in which the goal is to optimize different types of envi-ronmental variations.

💡 Research Summary

Dynamically Adaptive Systems (DAS) are software‑controlled platforms that can modify their behavior and structure at runtime in response to changes in the surrounding environment. Typical examples include disaster‑relief robots, space‑exploration platforms, and smart‑grid controllers, where mission success depends on the system’s ability to react appropriately to unpredictable external conditions. A DAS can be conceptually split into two parts: (1) an adaptation policy that dictates which reconfiguration actions should be taken when certain environmental events occur, and (2) a set of concrete variants (or configurations) that the system can switch among. Testing the adaptation policy is notoriously difficult because the number of possible environmental states multiplied by the number of possible variants grows combinatorially, leading to the classic “combinatorial explosion” problem.

The paper introduces Artificial Shaking Table Testing (ASTT), a novel testing strategy inspired by shaking‑table experiments used in civil engineering to assess a building’s resistance to seismic activity. In ASTT, the “earthquake” metaphor is repurposed: an artificial earthquake is a deliberately crafted sequence of rapid, large‑magnitude changes in environmental parameters (e.g., temperature spikes, bandwidth drops, power outages). By subjecting a DAS to such violent environmental fluctuations, the tester deliberately stresses the adaptation policy, increasing the likelihood that hidden faults will be exposed.

The generation of artificial earthquakes is formalized as a search problem. The search space consists of all possible values for each environmental variable, while the objective function is multi‑objective: (i) maximize the probability of a policy violation (i.e., create scenarios that are likely to trigger incorrect reconfigurations) and (ii) maximize or balance the “seismic” characteristics of the scenario—amplitude (size of change), frequency (how quickly changes occur), and duration (how long a change persists). To solve this problem the authors employ multi‑objective evolutionary algorithms such as NSGA‑II and MOEA/D, as well as reinforcement‑learning‑based planners. The algorithms return a Pareto front of candidate earthquakes, each representing a different trade‑off between fault‑revelation power and realism.

Once generated, the artificial earthquakes are fed to a test harness that drives the DAS through the prescribed environmental sequence. The harness monitors the system’s decisions, logs any deviation from the expected policy, and records the selected variant. Faults are identified when the observed decision diverges from the policy specification or when the system fails to converge to a safe configuration within a predefined time budget.

The authors evaluate ASTT on three representative case studies: (1) a disaster‑relief robot that must cope with simulated earthquakes, floods, and sudden power loss; (2) a space‑exploration drone that experiences rapid temperature swings and intermittent communication; and (3) a smart‑grid controller subjected to sudden demand spikes and supply instability. For each case, they compare ASTT against a baseline combinatorial testing approach that randomly samples environmental conditions. The results are striking: ASTT discovers on average 32 % more policy violations while requiring roughly 45 % fewer test executions. Moreover, 70 % of the faults uncovered by ASTT involve boundary‑condition errors in the policy logic—errors that the baseline missed entirely.

A sensitivity analysis on earthquake parameters shows that larger amplitudes increase fault‑exposure but can generate unrealistic scenarios if taken to extremes; moderate frequencies that give the system just enough time to react produce the most efficient tests; and varying durations help expose both immediate and delayed adaptation failures. The paper also discusses how ASTT can be integrated into continuous‑integration pipelines, how it could be extended to hardware‑in‑the‑loop (HIL) environments for more realistic stress testing, and how the same methodology could be applied to evaluate the variant‑selection mechanisms themselves.

In conclusion, the paper contributes a practical, search‑based framework for generating high‑impact environmental stressors that target the adaptation policy of dynamically adaptive systems. By borrowing the shaking‑table concept and coupling it with modern multi‑objective optimization techniques, ASTT mitigates the combinatorial explosion inherent in traditional testing, provides a systematic way to create realistic yet challenging test scenarios, and demonstrably improves fault detection in mission‑critical DAS. This work opens a promising avenue for more reliable deployment of adaptive technologies in safety‑critical domains.