Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewards

Group Relative Policy Optimization (GRPO) assigns a single scalar advantage to all tokens in a completion. For structured generations with explicit segments and objectives, this couples unrelated reward signals across segments, leading to objective interference and misattributed credit. We propose Blockwise Advantage Estimation, a family of GRPO-compatible methods that assigns each objective its own advantage and applies it only to the tokens in the corresponding text block, reducing reliance on hand-designed scalar rewards and scaling naturally to additional objectives. A key challenge is estimating advantages for later blocks whose rewards are conditioned on sampled prefixes; standard unbiased approaches require expensive nested rollouts from intermediate states. Concretely, we introduce an Outcome-Conditioned Baseline that approximates intermediate state values using only within-group statistics by stratifying samples according to a prefix-derived intermediate outcome. On math tasks with uncertainty estimation, our method mitigates reward interference, is competitive with a state-of-the-art reward-designed approach, and preserves test-time gains from confidence-weighted ensembling. More broadly, it provides a modular recipe for optimizing sequential objectives in structured generations without additional rollouts.

💡 Research Summary

The paper tackles a fundamental limitation of Group Relative Policy Optimization (GRPO), a popular critic‑free reinforcement‑learning method for fine‑tuning large language models (LLMs). GRPO samples a batch of completions for each prompt, normalizes the scalar rewards across the batch, and then applies the same advantage to every token in a completion. While this works for single‑objective tasks, it becomes problematic when a generation is naturally segmented into multiple, independent objectives—e.g., a math solution followed by a self‑confidence report. In such cases, a single scalar advantage mixes unrelated reward signals, causing “objective interference” and mis‑attributed credit.

The authors propose Blockwise Advantage Estimation (BAE), a GRPO‑compatible framework that treats a completion as a sequence of K contiguous blocks (