Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning

Recent advances in Large Language Models (LLMs) have underscored the potential of Reinforcement Learning (RL) to facilitate the emergence of reasoning capabilities. Despite the encouraging results, a fundamental dilemma persists as RL improvement relies on learning from high-quality samples, yet the exploration for such samples remains bounded by the inherent limitations of LLMs. This, in effect, creates an undesirable cycle in which what cannot be explored cannot be learned. In this work, we propose Rubric-Scaffolded Reinforcement Learning (RuscaRL), a novel instructional scaffolding framework designed to break the exploration bottleneck for general LLM reasoning. Specifically, RuscaRL introduces checklist-style rubrics as (1) explicit scaffolding for exploration during rollout generation, where different rubrics are provided as external guidance within task instructions to steer diverse high-quality responses. This guidance is gradually decayed over time, encouraging the model to internalize the underlying reasoning patterns; (2) verifiable rewards for exploitation during model training, where we can obtain robust LLM-as-a-Judge scores using rubrics as references, enabling effective RL on general reasoning tasks. Extensive experiments demonstrate the superiority of the proposed RuscaRL across various benchmarks, effectively expanding reasoning boundaries under the Best-of-N evaluation. Our code is available at https://github.com/IANNXANG/RuscaRL.

💡 Research Summary

The paper addresses a fundamental “exploration bottleneck” in reinforcement learning with verifiable rewards (RL‑VR) for large language models (LLMs). While RL‑VR can improve reasoning by learning from high‑quality samples, the generation of such samples is limited by the LLM’s own capabilities, especially for open‑ended tasks where no single ground‑truth answer exists (e.g., medical consultation, creative writing). To break this cycle, the authors propose Rubric‑Scaffolded Reinforcement Learning (RuscaRL), a framework that leverages checklist‑style rubrics in two complementary ways: (1) as explicit scaffolding during rollout generation to guide exploration, and (2) as a source of verifiable rewards during policy optimization.

Scaffolding for Exploration
For each instruction, a rubric R = {c₁,…,c_N} containing criteria and associated point values is injected into the prompt. The model generates a group of G candidate responses, each receiving a different proportion of the rubric (λ_i = (G‑i)/(G‑1)), creating intra‑group diversity: some samples are heavily scaffolded, others receive minimal guidance. Over the course of training, the overall scaffolding ratio λ_step(t) follows a sigmoid decay (λ_step(t) = 1/(1+e^{α(t‑t₀)})), gradually withdrawing external support so the model internalizes the reasoning patterns. This design mirrors Vygotsky’s scaffolding theory, providing structured assistance early on and fading it as competence grows.

Verifiable Rewards for Exploitation
Each criterion c_i is evaluated by a separate “LLM‑as‑a‑Judge” model, which outputs a binary decision b_i ∈ {0,1}. The final rubric score is s = Σ_i b_i·p_i, where p_i are the rubric points (positive or negative). The score is normalized by the total possible points to produce a scalar reward r. Rewards for the G responses in a group are then transformed into group‑relative advantages: ˆA_i = (r_i – mean(r))/std(r). These advantages feed directly into Group Relative Policy Optimization (GRPO), a policy‑gradient algorithm that eliminates the need for a value network by using group‑based advantage estimation. The GRPO objective incorporates token‑level importance ratios ρ_i,t and clipping, similar to PPO but simpler.

Experiments
RuscaRL is evaluated on several open‑ended benchmarks: HealthBench‑500 (medical advice), CodeEval (code generation), MathProof (formal proofs), and additional reasoning datasets. Under a “Best‑of‑N” evaluation, RuscaRL consistently outperforms strong baselines such as DeepSeek‑R1, GPT‑4, Gemini‑2.5, and specialized RL‑VR methods. Notably, a mid‑size model (Qwen3‑30B‑A3B‑Instruct) reaches performance comparable to OpenAI‑o3 on HealthBench‑500, demonstrating that rubric‑guided exploration can elevate smaller models to SOTA levels. Ablation studies show that (i) removing intra‑group scaffolding differentiation reduces response diversity and harms performance, and (ii) fixing the scaffolding ratio (no decay) leads to over‑reliance on rubrics and poorer generalization. Further analysis reveals that the number of rubric criteria and their weighting significantly affect reward sensitivity and exploration efficiency.

Limitations and Future Work
The approach requires human‑crafted rubrics, which incurs annotation cost and may introduce bias. The LLM‑as‑Judge itself can propagate systematic errors into the reward signal. Current experiments focus on token‑level policies; extending RuscaRL to tree‑structured reasoning or multimodal tasks remains open. Future directions include automated rubric generation, incorporation of multimodal evaluation criteria, and integration with hierarchical or tree‑based policy architectures to handle more complex reasoning trajectories.

Conclusion
RuscaRL demonstrates that checklist‑style rubrics can simultaneously enrich exploration and provide reliable, verifiable rewards, effectively breaking the exploration bottleneck that has limited RL‑VR for LLMs. By coupling instructional scaffolding with a simple yet powerful group‑relative policy optimization, the framework expands the reasoning capabilities of LLMs across diverse, open‑ended domains.