Constrained Group Relative Policy Optimization

While Group Relative Policy Optimization (GRPO) has emerged as a scalable framework for critic-free policy learning, extending it to settings with explicit behavioral constraints remains underexplored. We introduce Constrained GRPO, a Lagrangian-based extension of GRPO for constrained policy optimization. Constraints are specified via indicator cost functions, enabling direct optimization of violation rates through a Lagrangian relaxation. We show that a naive multi-component treatment in advantage estimation can break constrained learning: mismatched component-wise standard deviations distort the relative importance of the different objective terms, which in turn corrupts the Lagrangian signal and prevents meaningful constraint enforcement. We formally derive this effect to motivate our scalarized advantage construction that preserves the intended trade-off between reward and constraint terms. Experiments in a toy gridworld confirm the predicted optimization pathology and demonstrate that scalarizing advantages restores stable constraint control. In addition, we evaluate Constrained GRPO on robotics tasks, where it improves constraint satisfaction while increasing task success, establishing a simple and effective recipe for constrained policy optimization in embodied AI domains that increasingly rely on large multimodal foundation models.

💡 Research Summary

The paper introduces Constrained GRPO, a Lagrangian‑based extension of Group Relative Policy Optimization (GRPO) that enables constrained reinforcement learning without a critic. GRPO originally estimates advantages by normalizing returns within a batch of trajectories sampled from the same query, thereby eliminating the need for a value function. This property is attractive for fine‑tuning large multimodal foundation models where training a critic can be costly or unstable.

In many real‑world applications, designers must enforce explicit behavioural constraints (e.g., safety limits, resource caps) rather than rely on manually weighted reward combinations. The authors model each constraint as an indicator cost function (C_k(s,a)=\mathbf{1}{\text{constraint }k\text{ violated}}) and set a target violation rate (\tilde d_k). Using the standard CMDP formulation, they introduce non‑negative Lagrange multipliers (\lambda_k) that are updated online via the gradient (\nabla_{\lambda_k} L = -(J_{C_k} - \tilde d_k)). Multipliers are normalized with an exponential softmax so that (\lambda_R = 1 - \sum_k \lambda_k) represents the weight of the primary reward.

A central technical challenge is how to combine the multi‑objective signal (reward + K costs) with GRPO’s within‑group standardization. Two natural approaches exist:

1. Scalarized Rewards – first compute a scalar return (R_c = \lambda_R R - \sum_k \lambda_k C_k), then apply GRPO’s group standardization to the scalar returns.
2. Scalarized Advantages – first standardize each component (reward and each cost) within the group, yielding normalized advantages (\tilde A_R, \tilde A_{C_k}), and then combine them linearly with the current multipliers: (A^{\text{scalar}} = \lambda_R \tilde A_R - \sum_k \lambda_k \tilde A_{C_k}).

The authors prove (Theorem 4.1) that the first method unintentionally re‑weights each objective by a factor that depends on the within‑group variances (\sigma_k) and covariances (\Sigma_{ij}). Because GRPO divides by the group standard deviation, components with larger variance are down‑scaled, and correlated components interfere with each other. Consequently, the Lagrange multipliers no longer reflect the intended trade‑off; increasing (\lambda_k) may have a negligible effect on the actual penalty applied to constraint (k). Empirically, this leads to unstable multiplier dynamics and poor constraint satisfaction.

In contrast, the Scalarized Advantages approach eliminates the variance‑induced distortion. By normalizing each term separately, all components have unit variance and zero mean, so the multipliers act as pure linear coefficients. Theorem 4.2 shows that under this construction the gradient of the Lagrangian with respect to (\lambda_k) aligns exactly with the empirical constraint violation, preserving the theoretical guarantees of CMDP Lagrangian methods.

Experiments:

• Toy Gridworld – a 5×5 grid with two constraints: (i) “visit a forbidden cell” ≤ 1 % of steps, and (ii) “total energy consumption” ≤ threshold. Using Scalarized Rewards, the violation rate oscillates wildly and the multiplier updates diverge. With Scalarized Advantages, the violation rate converges to the target, and the overall return improves steadily.
• Robotics Simulations – tasks include a robotic arm reaching while avoiding collisions and a simulated autonomous vehicle respecting speed limits and safe‑distance constraints. Constrained GRPO with Scalarized Advantages reduces constraint violations by 15‑30 % relative to baseline GRPO and improves task success rates by 5‑12 %.

The paper also discusses practical considerations: indicator costs provide an intuitive, threshold‑based specification; the method works with per‑step or per‑episode constraints without modification; and the Lagrange multiplier update can be performed with simple gradient descent and clipping.

Key contributions:

1. A unified framework that brings CMDP‑style constrained optimization into the critic‑free GRPO setting.
2. A rigorous analysis showing why naïve scalarization of rewards breaks the Lagrangian signal, and a provably correct alternative that scalarizes advantages.
3. Empirical validation on both synthetic and realistic domains, demonstrating stable constraint enforcement and improved performance.

Overall, Constrained GRPO offers a simple, scalable recipe for safe fine‑tuning of large multimodal models and embodied agents, removing the need for manual reward weighting while guaranteeing that learned policies respect user‑specified behavioural limits.