Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning

Reinforcement Learning with Verifiable Rewards (RLVR) can elicit strong multi-step reasoning, yet it often encourages overly verbose traces. Moreover, naive length penalties in group-relative optimization can severely hurt accuracy. We attribute this failure to two structural issues: (i) Dilution of Length Baseline, where incorrect responses (with zero length reward) depress the group baseline and over-penalize correct solutions; and (ii) Difficulty-Penalty Mismatch, where a static penalty cannot adapt to problem difficulty, suppressing necessary reasoning on hard instances while leaving redundancy on easy ones. We propose Dynamic Decoupled Conditional Advantage (DDCA) to decouple efficiency optimization from correctness. DDCA computes length advantages conditionally within the correct-response cluster to eliminate baseline dilution, and dynamically scales the penalty strength using the group pass rate as a proxy for difficulty. Experiments on GSM8K, MATH500, AMC23, and AIME25 show that DDCA consistently improves the efficiency–accuracy trade-off relative to adaptive baselines, reducing generated tokens by approximately 60% on simpler tasks (e.g., GSM8K) versus over 20% on harder benchmarks (e.g., AIME25), thereby maintaining or improving accuracy. Code is available at https://github.com/alphadl/DDCA.

💡 Research Summary

The paper tackles a pervasive inefficiency in large reasoning models (LRMs) trained with Reinforcement Learning with Verifiable Rewards (RLVR). While RLVR, especially Group‑Relative Policy Optimization (GRPO), can elicit strong multi‑step reasoning, it also encourages “overthinking”: models generate excessively verbose chain‑of‑thought (CoT) traces that inflate inference cost without improving answer correctness. A naïve remedy—adding a length penalty to the reward—often degrades performance. The authors identify two structural failures that explain this trade‑off.

1. Dilution of Length Baseline: In GRPO the baseline is the mean reward across all samples in a group. Incorrect samples receive zero reward (and no length signal), pulling the baseline down. Consequently, even correct responses are penalized relative to an artificially low baseline, leading to over‑penalization of longer yet valid reasoning.

2. Difficulty‑Penalty Mismatch: A static length‑penalty coefficient γ cannot adapt to problem difficulty. Hard problems require longer reasoning chains; a fixed γ may dominate the correctness signal and force premature truncation, harming accuracy. Easy problems, on the other hand, need stronger pressure to eliminate redundancy, which a γ tuned for hard problems fails to provide.

To resolve both issues, the authors propose Dynamic Decoupled Conditional Advantage (DDCA), a reward‑shaping framework compatible with GRPO and Reinforce‑Leave‑One‑Out (RLOO). DDCA decouples the total advantage into two independent components: an accuracy advantage (A_{\text{acc}}) and a length advantage (A_{\text{len}}). The key innovations are:

• Conditional Length Normalization – Length statistics (mean µ_C and standard deviation σ_C) are computed only within the correct‑response cluster C. For each correct sample, a Z‑score (z_i = (|y_i| - µ_C)/σ_C) is calculated and passed through a sigmoid, yielding a bounded length reward (r_{\text{len},i} = 1/(1+e^{-z_i})). This eliminates baseline dilution and caps extreme outliers.

• Dynamic Difficulty‑Aware Scaling – The group pass rate (\rho = n/G) (fraction of correct samples) serves as a proxy for difficulty. The length advantage is multiplied by (\rho), so that on hard problems ((\rho) small) the length term is suppressed and the model focuses on correctness, while on easy problems ((\rho) near 1) the length term is fully applied, encouraging concise solutions.

• RLOO Estimation for Both Advantages – Both (A_{\text{acc}}) and (A_{\text{len}}) are estimated with a leave‑one‑out baseline to reduce gradient variance. The final objective is (A = A_{\text{acc}} - \beta \cdot A_{\text{len}}), where (\beta) controls overall penalty strength.

Experiments were conducted on two backbone models: DeepSeek‑R1‑Distill‑1.5B (a supervised‑fine‑tuned model) and DeepScaleR‑1.5B‑Preview (further RL‑refined). Training used a compact dataset of 2,470 historical AIME/AMC problems. Evaluation covered four benchmarks of increasing difficulty: GSM8K (grade‑school math), MATH500 (harder math), AMC23, and AIME25 (competition‑level).

Key results:

• Token Efficiency – On the easy GSM8K benchmark, DDCA reduced average token consumption by roughly 60% compared to the vanilla RLOO baseline. On the hardest AIME25 benchmark, token usage dropped by over 20%, demonstrating that the dynamic scaling preserves necessary reasoning length for difficult problems.

• Accuracy Preservation / Improvement – For DeepSeek‑R1‑Distill, DDCA raised pass@1 from 68.1 % to 68.5 % while cutting tokens from 6,061 to 4,908 (≈20 % reduction). For DeepScaleR‑Preview, accuracy remained essentially unchanged (72.5 % → 72.2 %) while tokens fell from 4,182 to 3,095 (≈26 % reduction).

• Superior AES Scores – The Accuracy‑Efficiency Score (AES) was highest for DDCA across both models (0.42 vs. 0.30 for the baseline), indicating a better balance of cost and performance than competing methods such as GRPO+LP, ThinkPrune‑4K, and TLMRE.

The analysis confirms that the two identified structural flaws are the root cause of the inefficiency‑accuracy trade‑off in RLVR. By conditioning length rewards on correct responses and scaling them with a difficulty‑aware factor, DDCA enables LRMs to “think dense” – delivering concise, high‑information reasoning when possible, while retaining the capacity for longer, more elaborate chains when the problem demands it. This approach offers a practical pathway to deploy large reasoning models in latency‑sensitive or compute‑constrained environments without sacrificing problem‑solving ability.