Learning Agile Striker Skills for Humanoid Soccer Robots from Noisy Sensory Input

Learning fast and robust ball-kicking skills is a critical capability for humanoid soccer robots, yet it remains a challenging problem due to the need for rapid leg swings, postural stability on a single support foot, and robustness under noisy sensory input and external perturbations (e.g., opponents). This paper presents a reinforcement learning (RL)-based system that enables humanoid robots to execute robust continual ball-kicking with adaptability to different ball-goal configurations. The system extends a typical teacher-student training framework – in which a “teacher” policy is trained with ground truth state information and the “student” learns to mimic it with noisy, imperfect sensing – by including four training stages: (1) long-distance ball chasing (teacher); (2) directional kicking (teacher); (3) teacher policy distillation (student); and (4) student adaptation and refinement (student). Key design elements – including tailored reward functions, realistic noise modeling, and online constrained RL for adaptation and refinement – are critical for closing the sim-to-real gap and sustaining performance under perceptual uncertainty. Extensive evaluations in both simulation and on a real robot demonstrate strong kicking accuracy and goal-scoring success across diverse ball-goal configurations. Ablation studies further highlight the necessity of the constrained RL, noise modeling, and the adaptation stage. This work presents a system for learning robust continual humanoid ball-kicking under imperfect perception, establishing a benchmark task for visuomotor skill learning in humanoid whole-body control.

💡 Research Summary

This paper presents a comprehensive reinforcement learning (RL) system designed to equip humanoid soccer robots with robust and adaptive ball-kicking skills, even under significant perceptual noise and uncertainty. The core challenge addressed is learning a whole-body control policy that integrates fast leg swings for powerful kicks, dynamic balance on a single support foot, and precise targeting—all while relying on noisy, delayed, and occasionally missing sensory estimates of the ball and goal positions.

The proposed solution is a sophisticated four-stage training framework that extends the classic teacher-student paradigm. In the first two stages, a “teacher” policy, which has access to privileged ground-truth state information (exact ball/goal positions, velocities, and physical parameters), is trained. Stage 1 focuses on “Long-Distance Chasing,” where the teacher learns a robust walking gait to approach a ball from various distances. Stage 2, “Directional Kicking,” teaches the teacher to execute precise kicks toward a target goal. Aggressive domain randomization, including random pushes, ball disturbances, and variations in robot/ball dynamics, is applied throughout these stages to foster robustness and recovery capabilities.

Stage 3, “Teacher Policy Distillation,” transfers the knowledge from the privileged teacher to a “student” policy that must operate with only imperfect perceptual inputs. This student policy receives a history of observations containing noisy estimates of ball and goal position, modeled with velocity-dependent noise, random delays, and frame drops to simulate real-world perception issues like occlusion. The distillation is performed using the DAgger algorithm.

However, the initial student policy distilled in Stage 3 often exhibits jittery, unsafe motions, such as sharp turns just before a kick. To rectify this, Stage 4, “Student Adaptation and Refinement,” employs an online constrained RL algorithm called N-P3O. This stage fine-tunes the student policy by allowing it to take slightly riskier actions when a high reward (like a successful kick) is imminent, while generally enforcing smoother and safer behavior. This “heterogeneous credit assignment” is key to closing the sim-to-real gap and improving motion quality.

The system was extensively evaluated in simulation, demonstrating high kicking accuracy, goal-scoring success rates, and adaptability across diverse ball-goal configurations. Ablation studies critically confirmed the necessity of the final adaptation stage (Stage 4), the constrained RL refinement, and the realistic noise modeling for achieving robust performance. Finally, the policy was deployed zero-shot on a real Booster T1 humanoid robot, achieving an average goal-scoring success rate of 66.7% across five different test scenarios, thereby validating the practical efficacy of the entire framework. This work establishes a notable benchmark for agile visuomotor skill learning in humanoid robots under imperfect perception.