HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments

As large language models (LLMs) continue to scale and new GPUs are released even more frequently, there is an increasing demand for LLM post-training in heterogeneous environments to fully leverage underutilized mid-range or previous-generation GPUs across regions and alleviate the shortage of homogeneous high-end GPUs within a single region. However, achieving high-performance reinforcement learning (RL) training for LLMs on such computing resources remains challenging because the workflow involves multiple models and tasks with complex computation and data dependencies. In this paper, we present HetRL, a distributed system for efficient RL training in infrastructures with heterogeneous GPUs and networks. HetRL formulates the scheduling of RL training in heterogeneous environments as a constrained joint optimization problem and introduces a novel scheduling algorithm that (1) decomposes the complex search space with a multi-level search framework; and (2) allocates the search budget via successive halving. Our extensive evaluation, consuming 20,000 GPU-hours, shows that HetRL delivers up to 9.17x the throughput of state-of-the-art systems, and 3.17x on average, under various workloads and settings.

💡 Research Summary

The paper “HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments” presents a novel distributed system designed to address the significant challenge of performing Reinforcement Learning (RL) fine-tuning for Large Language Models (LLMs) across infrastructures composed of heterogeneous GPUs and networks. The core problem stems from the explosive computational demand of RL for LLMs (like PPO), which traditionally relies on expensive, homogeneous clusters of high-end GPUs. Meanwhile, a vast pool of underutilized, mid-range or previous-generation GPUs exists across different geographical regions. Leveraging these heterogeneous resources is highly desirable but difficult because the RL workflow involves multiple models (actor, critic, reward, reference) and tasks (generation, inference, training) with complex dependencies, making efficient scheduling non-trivial.

HetRL formulates the scheduling problem in heterogeneous environments as a constrained joint optimization of partitioning strategies (how to parallelize computations within and across models/tasks) and assignment strategies (how to map computational units to physical devices). To tackle the immense search space of this NP-hard problem, HetRL introduces a scheduling algorithm built on two key ideas: a multi-level search framework and successive halving for budget allocation. The multi-level framework decomposes the search into manageable levels: (1) grouping RL tasks and assigning them to coarse-grained GPU groups, (2) determining intra-model parallelization strategies (Tensor, Pipeline, Data Parallelism) for each group, and (3) performing fine-grained assignment of tasklets to individual GPUs. This hierarchical approach reduces complexity. The successive halving technique then efficiently allocates evaluation time among candidate scheduling plans, quickly pruning poor performers.

The authors implemented HetRL by extending the existing verl RL training system with approximately 3,000 lines of code, incorporating a new scheduler, profiler, and an execution engine with support for fine-grained resource assignment and load balancing. Evaluation was extensive, consuming 20,000 GPU-hours across diverse workloads (varying model sizes: 7B, 13B, 70B) and heterogeneous settings (mixes of A100, L4, L40S, H100 GPUs across simulated geo-distributed data centers with varying network bandwidths and latencies). The results demonstrate that HetRL significantly outperforms state-of-the-art systems like verl and OpenRLHF, achieving up to 9.17x and an average of 3.17x higher training throughput. The performance gains are most substantial in highly heterogeneous environments. HetRL thus provides a practical and high-performance solution for utilizing geographically dispersed, heterogeneous GPU pools for large-scale LLM RL training, potentially alleviating resource bottlenecks and improving overall hardware utilization in the face of rapidly evolving and diverse computing landscapes.