Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards has driven recent advances in LLM post-training, in particular for reasoning. Policy optimization algorithms generate a number of responses for a given prompt and then effectively weight the corresponding gradients depending on the rewards. The most popular algorithms including GRPO, DAPO, and RLOO focus on ambiguous prompts, i.e., prompts with intermediate success probability, while downgrading gradients with very easy and very hard prompts. In this paper, we consider asymmetric prompt weightings that assign higher weights to prompts with low, or even zero, empirical success probability. We find that asymmetric weighting particularly benefits from-scratch RL (as in R1-Zero), where training traverses a wide accuracy range, and less so in post-SFT RL where the model already starts at high accuracy. We also provide theory that characterizes prompt weights which minimize the time needed to raise success probability from an initial level to a target accuracy under a fixed update budget. In low-success regimes, where informative responses are rare and response cost dominates, these optimal weights become asymmetric, upweighting low success probabilities and thereby accelerating effective-time convergence.

💡 Research Summary

This paper investigates how to improve reinforcement learning with verifiable rewards (RL‑VR) for large language models (LLMs) by redesigning the way prompts are weighted during policy updates. In standard RL‑VR pipelines such as GRPO, DAPO, and RLOO, each training step samples a batch of prompts, generates multiple completions per prompt, computes a binary reward (e.g., correct/incorrect), and updates the policy using a weighted sum of per‑prompt gradients. The weighting schemes employed by these methods are symmetric: they assign the highest importance to “ambiguous” prompts whose empirical success probability ρ̂ lies near 0.5, while down‑weighting both very easy (ρ̂≈1) and very hard (ρ̂≈0) prompts. This design stabilizes training but discards potentially useful learning signals from the hardest prompts, especially when the model is still naïve and most prompts have near‑zero success.

The authors propose asymmetric prompt weighting: a family of weighting functions ω(ρ̂) that deliberately up‑weight prompts with low success probability, even when ρ̂=0. They define five concrete schemes:

1. Linear‑R: w(ρ)=1/ρ → effective weight 1‑ρ. For wrong completions (r=0) the advantage becomes –1, guaranteeing a non‑zero gradient even when ρ̂=0.
2. Sqrt‑R: w(ρ)=1/(ρ√(1‑ρ)) → behaves like Linear‑R for small ρ and like GRPO for ρ close to 1. Effective weight √(1‑ρ).
3. Plateau‑R: piecewise; constant weight for ρ<0.5 and GRPO‑like scaling for ρ≥0.5. Provides a plateau of importance for very hard prompts.
4. Uniform‑R: w(ρ)=1/(ρ(1‑ρ)) → cancels the usual ρ(1‑ρ) factor, giving every prompt equal effective weight.
5. Baseline (GRPO/RLOO): symmetric schemes that down‑weight both extremes.

All schemes share the same direction estimator used by RLOO/GRPO: \