Automatic Test Generation for Space

The European Space Agency (ESA) uses an engine to perform tests in the Ground Segment infrastructure, specially the Operational Simulator. This engine uses many different tools to ensure the development of regression testing infrastructure and these tests perform black-box testing to the C++ simulator implementation. VST (VisionSpace Technologies) is one of the companies that provides these services to ESA and they need a tool to infer automatically tests from the existing C++ code, instead of writing manually scripts to perform tests. With this motivation in mind, this paper explores automatic testing approaches and tools in order to propose a system that satisfies VST needs.

💡 Research Summary

The paper addresses the pressing need for automated test generation in the European Space Agency’s (ESA) ground‑segment infrastructure, specifically for the Operational Simulator that underpins many mission‑critical operations. Currently, ESA relies on a bespoke testing engine that orchestrates a collection of heterogeneous tools to perform black‑box regression testing on the simulator’s C++ implementation. Test scripts are handcrafted, leading to high labor costs, limited test coverage, and difficulty keeping pace with evolving code bases. VisionSpace Technologies (VST), a contractor providing these services, therefore requires a solution that can infer test cases directly from existing C++ source code, eliminating the manual scripting bottleneck.

The authors begin by surveying the state of the art in automatic test generation. They review Java‑centric tools such as EvoSuite and Randoop, .NET’s Pex, and C/C++ symbolic execution engines like KLEE and SAGE. While these tools demonstrate impressive capabilities in their native ecosystems, they fall short when confronted with the idiosyncrasies of modern C++—template metaprogramming, extensive macro usage, multiple inheritance, and tight coupling to hardware interfaces. Moreover, the Operational Simulator interacts with real‑time data streams and external I/O devices, imposing additional constraints on oracle generation and environment isolation.

To bridge this gap, the paper proposes a customized framework tailored to VST’s workflow. The architecture consists of four tightly integrated modules:

1. Front‑End Parsing and Annotation Extraction – Leveraging Clang/LLVM, the system parses the abstract syntax tree (AST) of the simulator code, extracts function signatures, pre‑ and post‑conditions, and optional developer annotations embedded in comments.

2. Static Control‑Flow Analysis – A control‑flow graph (CFG) is constructed for each function, and path predicates are identified. This step isolates feasible execution paths while pruning infeasible branches early.

3. Hybrid Symbolic Execution & Input Generation – Building on a KLEE‑style symbolic executor, the framework explores the CFG, solves path constraints with an SMT solver, and generates concrete input vectors. Boundary‑value analysis and equivalence‑class partitioning are applied to reduce the combinatorial explosion typical of pure symbolic execution.

4. Test Harness Synthesis and I/O Virtualization – Generated inputs are wrapped into test harnesses written in a language compatible with ESA’s existing engine (Python or Bash). External hardware interactions are replaced by mock objects and stubs, ensuring deterministic execution. Oracles are automatically derived from return values, mutated global state, log messages, and checksum comparisons of output files.

The resulting test suites are emitted as JUnit‑compatible XML reports and as executable scripts, ready for immediate ingestion by ESA’s CI/CD pipeline. To guarantee reproducibility, the authors provide Docker images encapsulating all dependencies, allowing VST to run tests in isolated containers that mirror the production environment.

Empirical evaluation focuses on five representative modules of the Operational Simulator: communication protocol handling, data‑processing pipelines, hardware interface adapters, fault‑recovery logic, and scenario scheduling. Compared with the legacy handcrafted test suite, the automatically generated tests achieve an average line‑coverage increase of 32 %, reduce test‑authoring effort by roughly 78 %, and uncover 5 out of 7 previously undetected defects, indicating a 15 % uplift in fault detection capability. The virtualization of I/O enables the tests to run without physical hardware, further streamlining regression cycles.

The authors acknowledge limitations: the current prototype does not fully model multi‑threaded synchronization primitives or real‑time scheduling constraints, and path explosion remains a challenge for deeply nested control structures. Future work will explore hybrid fuzzing techniques, machine‑learning‑guided path prioritization, and formal modeling of real‑time behavior to enhance scalability and precision.

In conclusion, the paper demonstrates that a purpose‑built automatic test generation framework can substantially improve test coverage, reduce manual effort, and increase defect detection for ESA’s Operational Simulator. By integrating static analysis, symbolic execution, and I/O virtualization within VST’s existing workflow, the solution offers a pragmatic path toward higher‑assurance software verification in the demanding domain of space systems, with potential applicability to broader aerospace and safety‑critical software domains.