ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithm

Deep reinforcement learning has been able to solve various tasks successfully, however, due to the construction of policy gradient and training dynamics, tuning deep reinforcement learning models remains challenging. As one of the most successful deep reinforcement-learning algorithm, the Proximal Policy Optimization algorithm (PPO) clips the policy gradient within a conservative on-policy updates, which ensures reliable and stable policy improvement. However, this training pattern may sacrifice sample efficiency. On the other hand, off-policy methods make more adequate use of data through sample reuse, though at the cost of increased the estimation variance and bias. To leverage the advantages of both, in this paper, we propose a new PPO variant based on the stability guarantee from conservative on-policy iteration with a more efficient off-policy data utilization. Specifically, we first derive an extended off-policy improvement from an expectation form of generalized policy improvement lower bound. Then, we extend the clipping mechanism with segmented exponential functions for a suitable surrogate objective function. Third, the trajectories generated by the past $M$ policies are organized in the replay buffer for off-policy training. We refer to this method as Extended Off-policy Proximal Policy Optimization (ExO-PPO). Compared with PPO and some other state-of-the-art variants, we demonstrate an improved performance of ExO-PPO with balanced sample efficiency and stability on varied tasks in the empirical experiments.

💡 Research Summary

ExO‑PPO (Extended Off‑policy Proximal Policy Optimization) is a novel variant of the widely used PPO algorithm that seeks to combine the stability guarantees of on‑policy learning with the sample‑efficiency benefits of off‑policy data reuse. The authors first derive an “Extended Off‑policy Improvement Lower Bound” that generalizes the classic policy‑improvement lower bound to any reference policy, allowing the expected advantage of a target policy to be expressed in terms of past policies stored in a replay buffer. By treating the last M policies as reference policies, the bound shows that the performance gap between the current policy and the mixture of past policies can be bounded by a term involving the importance‑sampling ratio and a penalty proportional to the total‑variation distance between policies. This theoretical result justifies reusing trajectories generated by earlier policies without sacrificing the monotonic improvement guarantee.

To handle the large distribution shift that inevitably arises when off‑policy samples are used, the authors replace PPO’s simple clipping function with a “segmented exponential clipping” mechanism. Inside the usual trust‑region interval (