A Gait Driven Reinforcement Learning Framework for Humanoid Robots

This paper presents a real-time gait driven training framework for humanoid robots. First, we introduce a novel gait planner that incorporates dynamics to design the desired joint trajectory. In the gait design process, the 3D robot model is decoupled into two 2D models, which are then approximated as hybrid inverted pendulums (H-LIP) for trajectory planning. The gait planner operates in parallel in real time within the robot’s learning environment. Second, based on this gait planner, we design three effective reward functions within a reinforcement learning framework, forming a reward composition to achieve periodic bipedal gait. This reward composition reduces the robot’s learning time and enhances locomotion performance. Finally, a gait design example, along with simulation and experimental comparisons, is presented to demonstrate the effectiveness of the proposed method.

💡 Research Summary

The paper proposes a novel framework that tightly couples a real‑time gait planner with reinforcement learning (RL) to enable fast and reliable bipedal locomotion for humanoid robots. The core idea is to decompose the full 3‑D robot dynamics into two planar subsystems—one for the sagittal (X) axis and one for the lateral (Y) axis—and to approximate each subsystem by a Hybrid Linear Inverted Pendulum (H‑LIP). The H‑LIP model captures the essential hybrid nature of walking: a continuous single‑support phase (SSP) governed by linear dynamics and an instantaneous double‑support phase (DSP) that swaps the swing and stance legs. Because the dynamics are linear, closed‑form expressions for the state evolution and the relationship between step length, swing period, and the SSP final state can be derived analytically.

Using these analytical tools, the authors design a gait planner that generates desired joint trajectories on the fly. Joint trajectories are parameterized by Bézier polynomials; the planner enforces five constraints derived from the H‑LIP model: (1) periodic swapping of swing and stance legs, (2) zero foot‑ground clearance at the beginning and end of each step, (3) positive clearance during swing, (4) consistency of the final SSP state with the H‑LIP step‑length equation, and (5) constant CoM height. The constraints guarantee that the generated trajectories respect both the hybrid dynamics and the kinematic contact conditions. By pre‑computing Bézier coefficients for a range of step lengths and periods, the planner can synthesize feasible trajectories in real time (≈40 Hz), which are then fed to the RL agent as reference signals.

The RL component is built around three complementary reward terms. A periodicity reward penalizes deviations from the exact leg‑swap timing, encouraging a stable gait rhythm. A trajectory‑tracking reward penalizes the L2 error between the robot’s actual joint angles/velocities and the planner’s reference, effectively pulling the learned policy toward the model‑based solution. A time‑efficiency reward encourages shorter swing periods while maintaining stability, thus improving walking speed. This structured reward composition reduces the variance of the learning signal and mitigates the common problem of erratic, non‑periodic motions observed in pure model‑free RL approaches.

Experimental validation is performed both in simulation and on a physical 12‑DOF humanoid prototype. The proposed method is compared against (i) a baseline model‑free RL with handcrafted rewards and (ii) a hybrid approach that uses an offline‑optimized gait generator combined with RL. Results show that the new framework reduces the number of training episodes by roughly 40 % and yields smoother, more periodic walking with fewer foot‑placement errors and lower joint‑torque oscillations. The real‑time planner successfully adapts to varying step lengths and swing periods during training, demonstrating robustness to the exploration noise inherent in RL. However, the paper acknowledges limitations: the H‑LIP approximation neglects complex foot‑ankle interactions and assumes negligible coupling between the X and Y planar models, which may degrade performance on highly uneven terrain or during fast dynamic maneuvers such as running. Moreover, the planner’s 40 Hz update rate may be insufficient for ultra‑fast locomotion.

In summary, the work makes three key contributions: (1) a dynamic decoupling strategy that reduces high‑dimensional humanoid dynamics to tractable 2‑D H‑LIP models, (2) a real‑time Bézier‑based gait planner that guarantees hybrid‑dynamic feasibility and constant CoM height, and (3) a multi‑objective reward design that leverages the planner’s output to accelerate RL convergence and produce periodic gaits. The integration of analytical model‑based planning with data‑driven RL offers a scalable pathway toward sample‑efficient, stable locomotion for humanoid robots, while highlighting future research directions such as disturbance‑robust planning, richer foot‑contact modeling, and higher‑frequency control loops.