Sim2Real Reinforcement Learning for Soccer skills

This thesis work presents a more efficient and effective approach to training control-related tasks for humanoid robots using Reinforcement Learning (RL). The traditional RL methods are limited in adapting to real-world environments, complexity, and natural motions, but the proposed approach overcomes these limitations by using curriculum training and Adversarial Motion Priors (AMP) technique. The results show that the developed RL policies for kicking, walking, and jumping are more dynamic, and adaptive, and outperformed previous methods. However, the transfer of the learned policy from simulation to the real world was unsuccessful, highlighting the limitations of current RL methods in fully adapting to real-world scenarios.

💡 Research Summary

This paper investigates an advanced Reinforcement Learning (RL) approach aimed at mastering complex, dynamic skills for humanoid robots, specifically focusing on soccer-related tasks such as walking, jumping, and kicking. The primary challenge addressed is the inherent difficulty in training humanoid controllers to exhibit both high-level task proficiency and natural, fluid motion, while simultaneously bridging the notorious “Sim2Real gap.”

To overcome the limitations of traditional RL—which often results in unnatural or unstable movements—the researcher proposes a dual-strategy framework. The first component is Curriculum Training, a method that incrementally increases the complexity of the learning tasks. By starting with fundamental balance and locomotion and progressively introducing more strenuous maneuvers like jumping and kicking, the agent is able to navigate the high-dimensional state space more effectively without encountering the instability common in high-difficulty task initialization. The second component is the implementation of Adversarial Motion Priors (AMP). By utilizing a discriminator-based approach inspired by Generative Adversarial Networks (GANs), the framework constrains the learned policy to stay within the distribution of a predefined dataset of natural motions. This ensures that the resulting behaviors are not merely goal-oriented but also biologically plausible and aesthetically smooth.

The experimental results within the simulated environment were highly successful. The proposed method demonstrated superior performance over baseline RL algorithms, producing policies that were significantly more dynamic, adaptive, and capable of executing complex soccer maneuvers with high fidelity. The robot’s ability to maintain stability during high-impact motions like kicking and jumping showed marked improvement.

However, the study reaches a critical and sobering conclusion regarding the Sim2Real transfer. Despite the impressive performance in simulation, the learned policies failed to transfer successfully to the physical humanoid robot. This failure highlights the persistent “reality gap”—the discrepancy between the idealized physics of a simulator and the unpredictable, noisy, and non-linear dynamics of the real world, such as sensor latency, friction variations, and mechanical backlash. Ultimately, while the research provides a significant leap forward in achieving naturalistic motion through AMP and curriculum learning, it underscores that the fundamental challenge of making RL-trained policies robust enough for real-world deployment remains one of the most significant hurdles in modern robotics.