Digital Twin Supervised Reinforcement Learning Framework for Autonomous Underwater Navigation

Autonomous navigation in underwater environments remains a major challenge due to the absence of GPS, degraded visibility, and the presence of submerged obstacles. This article investigates these issues through the case of the BlueROV2, an open platform widely used for scientific experimentation. We propose a deep reinforcement learning approach based on the Proximal Policy Optimization (PPO) algorithm, using an observation space that combines target-oriented navigation information, a virtual occupancy grid, and ray-casting along the boundaries of the operational area. The learned policy is compared against a reference deterministic kinematic planner, the Dynamic Window Approach (DWA), commonly employed as a robust baseline for obstacle avoidance. The evaluation is conducted in a realistic simulation environment and complemented by validation on a physical BlueROV2 supervised by a 3D digital twin of the test site, helping to reduce risks associated with real-world experimentation. The results show that the PPO policy consistently outperforms DWA in highly cluttered environments, notably thanks to better local adaptation and reduced collisions. Finally, the experiments demonstrate the transferability of the learned behavior from simulation to the real world, confirming the relevance of deep RL for autonomous navigation in underwater robotics.

💡 Research Summary

This paper presents a comprehensive study on applying deep reinforcement learning (RL) to the challenge of autonomous navigation for underwater vehicles, using the BlueROV2 as a practical experimental platform. The core problem addressed is navigating in GPS-denied, visually degraded environments cluttered with static obstacles.

The authors propose a learning-based approach centered on the Proximal Policy Optimization (PPO) algorithm. The RL agent’s observation space is strategically designed to integrate multiple information streams: target-relative navigation data (distance and heading), a virtual occupancy grid for obstacle detection (simulating a forward-looking sonar), and ray-casting information along the boundaries of the operational area. This rich state representation allows the agent to develop a nuanced understanding of its surroundings. The learned policy is systematically compared against a classical, deterministic reactive planner: the Dynamic Window Approach (DWA), which serves as a robust baseline in mobile robotics.

The evaluation framework is twofold. Primary training and comparative testing are conducted in a realistic simulation environment modeling the BlueROV2’s kinematics and a rectangular operational area populated with randomly placed obstacles. Crucially, the study then validates the transferability of the simulation-trained policy through real-world experiments. This is done by deploying the policy on a physical BlueROV2, whose operation is supervised and monitored via a 3D digital twin of the test site, thereby mitigating the high risks and costs associated with direct in-water testing.

The results demonstrate that the PPO-based policy consistently outperforms the DWA planner, particularly in highly cluttered environments. Key advantages include better local adaptation to complex obstacle arrangements and a significant reduction in collisions. The successful execution of the learned policy on the real vehicle, guided by the digital twin supervision, provides strong empirical evidence for the sim-to-real transferability of the deep RL approach. The paper concludes that deep reinforcement learning is a highly relevant and promising technique for advancing autonomy in underwater robotics, offering superior performance over traditional methods in unstructured settings. It also highlights the critical role of digital twins as a safe and effective bridge between simulation and reality for developing and validating such autonomous systems.