Before Autonomy Takes Control: Software Testing in Robotics

Robotic systems are complex and safety-critical software systems. As such, they need to be tested thoroughly. Unfortunately, robot software is intrinsically hard to test compared to traditional software, mainly since the software needs to closely interact with hardware, account for uncertainty in its operational environment, handle disturbances, and act highly autonomously. However, given the large space in which robots operate, anticipating possible failures when designing tests is challenging. This paper presents a mapping study by considering robotics testing papers and relating them to the software testing theory. We consider 247 robotics testing papers and map them to software testing, discussing the state-of-the-art software testing in robotics with an illustrated example, and discuss current challenges. Forming the basis to introduce both the robotics and software engineering communities to software testing challenges. Finally, we identify open questions and lessons learned.

💡 Research Summary

The paper “Before Autonomy Takes Control: Software Testing in Robotics” addresses the growing need for rigorous testing of robotic systems, which are increasingly deployed in safety‑critical domains such as healthcare, agriculture, and transportation. Unlike traditional software, robot software must tightly integrate with hardware, handle noisy sensors, variable actuators, and operate under uncertain, dynamic environments. These characteristics make testing far more complex and demand new approaches beyond the well‑established software‑testing standards used in conventional IT systems.

The authors conduct a systematic mapping study of the robotics‑testing literature. Starting from a broad Scopus query (“Robot AND Test”) they retrieve 21,068 records, then apply a two‑stage filtering process based on titles and abstracts, ultimately selecting 198 peer‑reviewed papers that genuinely discuss verification and validation of robotic systems. They exclude short papers, inaccessible works, and those that merely mention testing without substantive content. The selected corpus is then mapped onto the ISO 29119 testing standard family (parts 1‑4) and ISTQB testing knowledge, allowing the authors to assess which testing concepts have been adopted in robotics and where gaps remain.

The paper first clarifies the distinction between static and dynamic testing, focusing on the latter. A dynamic test case is defined by four elements: (1) pre‑condition (initial robot state, environment, configuration), (2) test input (commands, stimuli, sensor feeds), (3) expected outcome (desired behavior or state), and (4) post‑condition (observed results, logs, success/failure signals). This formalism is used throughout the mapping.

To illustrate the challenges, the authors present a detailed case study of a service robot operating in a restaurant. The robot is a PAL Robotics TIAGo platform equipped with ROS, a differential‑drive base, a 7‑DOF arm, RGB‑D camera, microphone array, speaker, and touchscreen. Three core missions are defined: (i) taking customer orders (via app, gesture, or speech), (ii) picking up and delivering meals, and (iii) cleaning tables. A set of concrete functional requirements (e.g., “add pick‑up task to queue”, “maintain safety velocity ≤ 0.8 m/s”, “recharge when battery ≤ 40 %”) is listed in Table 1. The robot’s software architecture follows the Self‑Adaptive Decentralized Robotic Architecture (SERA), a three‑layer hierarchy: mission management, change management, and component control. The authors map testing activities to each layer, showing how mission‑level tests verify task scheduling, change‑management tests validate replanning under blocked paths, and component‑control tests assess low‑level perception, localization, and motion‑control correctness.

The mapping results reveal that robotics testing literature heavily emphasizes simulation‑based testing (including Hardware‑in‑the‑Loop) and system‑integration experiments, while classic test‑design techniques such as boundary‑value analysis, equivalence partitioning, or formal coverage metrics are rarely applied. Moreover, only a handful of papers explicitly link testing to safety standards such as ISO 26262 or IEC 61508, indicating a substantial gap between safety certification requirements and current robotic testing practice.

Based on these observations, the authors identify several research directions:

1. Robotics‑specific test metrics and coverage criteria – defining measurable goals (e.g., collision‑avoidance success rate, latency under real‑time constraints) and mapping them to coverage models.
2. Digital‑twin and model‑based verification – reducing the simulation‑to‑reality gap by synchronizing high‑fidelity virtual twins with physical robots.
3. Test automation and continuous integration pipelines – integrating ROS‑based unit, integration, and system tests into CI/CD workflows to enable regression testing at scale.
4. Standardized safety‑aligned testing processes – extending ISO 29119 with robotics‑focused extensions that satisfy functional‑safety standards.
5. Human‑Robot Interaction (HRI) testing frameworks – systematic generation of interaction scenarios, user‑study protocols, and metrics for trust and usability.

The paper concludes that while robotics testing is a vibrant but fragmented field, a concerted effort to import mature software‑testing knowledge, adapt it to the cyber‑physical nature of robots, and align it with safety standards is essential for the reliable deployment of autonomous systems. By providing a comprehensive mapping, a concrete case study, and a clear agenda for future work, the authors aim to bridge the gap between the robotics and software‑engineering communities, fostering more systematic, repeatable, and safety‑aware testing practices for the next generation of autonomous robots.