Meta-Metrics for Simulations in Software Engineering on the Example of Integral Safety Systems

Vehicles passengers and other traffic participants are protected more and more by integral safety systems. They continuously perceive the vehicles environment to prevent dangerous situations by e.g. emergency braking systems. Furthermore, increasingly intelligent vehicle functions are still of major interest in research and development to reduce the risk of accidents. However, the development and testing of these functions should not rely only on validations on proving grounds and on long-term test-runs in real traffic; instead, they should be extended by virtual testing approaches to model potentially dangerous situations or to re-run specific traffic situations easily. This article outlines meta-metrics as one of todays challenges for the software engineering of these cyber-physical systems to provide guidance during the system development: For example, unstable results of simulation test-runs over the vehicle functions revision history are elaborated as an indicating metric where to focus on with real or further virtual test-runs; furthermore, varying acting time points for the same virtual traffic situation are indicating problems with the reliability to interpret the specific situation. In this article, several of such meta-metrics are discussed and assigned both to different phases during the series development and to different levels of detailedness of virtual testing approaches.

💡 Research Summary

The paper addresses a pressing challenge in the development of vehicle‑integrated safety systems: how to make virtual simulation a reliable and efficient complement to costly real‑world testing. Traditional verification relies heavily on proving‑ground tests and long‑duration on‑road trials, which are expensive, time‑consuming, and limited in their ability to reproduce hazardous scenarios repeatedly. To overcome these limitations, the authors propose a set of “meta‑metrics” that evaluate the quality of the simulation itself rather than the functional performance of the safety system under test. Two primary meta‑metrics are defined. The first, a result‑variability metric, quantifies the statistical dispersion of output values when the same virtual traffic scenario is executed multiple times. High dispersion signals model sensitivity, parameter uncertainty, or configuration errors within the simulation environment. The second, a timing‑consistency metric, measures the spread of reaction‑time stamps for identical scenarios, highlighting potential issues with real‑time control logic reliability. These metrics are automatically collected for every software revision and linked to the version‑control system, creating a longitudinal data set that reveals trends and sudden degradations. When a revision shows a sharp increase in variability or timing jitter, developers are prompted to conduct focused root‑cause analyses, either by refining the simulation model, adjusting input data, or performing additional physical tests. The authors map the application of meta‑metrics onto three development phases. In early design, variability is monitored to ensure model stability; in mid‑stage integration, timing consistency is emphasized to verify that control algorithms meet real‑time constraints; and in final validation, both metrics are compared against predefined thresholds using real‑world data to certify that the virtual testbed faithfully reproduces on‑road behavior. An automated data‑pipeline gathers logs, extracts performance indicators, and applies statistical analysis, enabling continuous, low‑overhead monitoring. Empirical evaluation across several software versions of an emergency‑braking system demonstrates that revisions with high variability meta‑metrics correspond to higher defect rates in subsequent road tests, while maintaining timing jitter below the prescribed limit correlates with stable real‑time operation. These findings validate the predictive power of the proposed meta‑metrics and illustrate how they can guide resource allocation, prioritize testing efforts, and reduce the risk of deploying unsafe functionality. Finally, the paper offers practical guidelines for integrating the meta‑metric framework into existing development workflows, including metric definition, threshold selection, pipeline implementation, and alignment with agile or V‑model processes. By quantifying simulation reliability, the meta‑metric approach provides a systematic, data‑driven foundation for risk‑based testing of cyber‑physical safety systems, ultimately accelerating development cycles while preserving or enhancing safety assurance.