Continuous-Utility Direct Preference Optimization

Large language model reasoning is often treated as a monolithic capability, relying on binary preference supervision that fails to capture partial progress or fine-grained reasoning quality. We introduce Continuous Utility Direct Preference Optimization (CU-DPO), a framework that aligns models to a portfolio of prompt-based cognitive strategies by replacing binary labels with continuous scores that capture fine-grained reasoning quality. We prove that learning with K strategies yields a Theta(K log K) improvement in sample complexity over binary preferences, and that DPO converges to the entropy-regularized utility-maximizing policy. To exploit this signal, we propose a two-stage training pipeline: (i) strategy selection, which optimizes the model to choose the best strategy for a given problem via best-vs-all comparisons, and (ii) execution refinement, which trains the model to correctly execute the selected strategy using margin-stratified pairs. On mathematical reasoning benchmarks, CU-DPO improves strategy selection accuracy from 35-46 percent to 68-78 percent across seven base models, yielding consistent downstream reasoning gains of up to 6.6 points on in-distribution datasets with effective transfer to out-of-distribution tasks.

💡 Research Summary

The paper “Continuous‑Utility Direct Preference Optimization” (CU‑DPO) addresses a fundamental limitation of current alignment methods for large language models (LLMs) that rely on binary preference labels. Binary supervision collapses the nuanced progress that occurs during multi‑step reasoning—e.g., a chain that is correct up to the final arithmetic step receives the same label as a chain that fails at the first logical inference. To capture these fine‑grained differences, the authors replace binary labels with continuous utility scores that aggregate three dimensions: correctness, step‑efficiency, and reasoning coherence. A lightweight LLM judge (Qwen 2.5 7B) produces these scores, achieving a high correlation (r = 0.85) with ground‑truth correctness while remaining largely independent of chain length.

The theoretical contribution is twofold. First, the authors prove that learning from K distinct reasoning strategies using continuous utilities reduces the sample complexity from Ω(N K² log K) (required for binary preferences under uniform passive sampling) to O(N K). This yields a Θ(K log K) improvement, meaning that for K = 8 strategies the method is theoretically about 24 times more sample‑efficient. Second, they show that when the Bradley‑Terry model is used to convert utilities into pairwise preference probabilities, the Direct Preference Optimization (DPO) loss implicitly learns a reward function rθ that satisfies rθ(x, y) = U(x, y) + c(x). Consequently, the optimal policy converges to the entropy‑regularized utility‑maximizing policy.

Practically, the paper introduces a two‑stage training pipeline designed to exploit the continuous signal without causing supervision conflicts. In Phase 1 (strategy selection), each problem is solved with K = 8 pre‑defined prompts that induce distinct cognitive strategies (step‑by‑step, backward reasoning, verification, etc.). The LLM judge scores each resulting chain, and the best‑vs‑all pairs are constructed, guaranteeing a large utility margin (average 0.35) for every comparison. This phase teaches the model to pick the most suitable strategy for a given problem. In Phase 2 (execution refinement), low‑utility chains (U < 0.4) are iteratively refined using a deterministic operator R, producing higher‑utility versions. Margin‑stratified sampling then creates intra‑strategy pairs that prioritize small utility gaps (< 0.15), forcing the model to learn subtle step‑level improvements rather than coarse strategy switches.

Empirical evaluation spans 450 problems from DeepMath, HARDMath2, and ProofNet. CU‑DPO raises strategy‑selection accuracy from 35‑46 % (binary baselines) to 68‑78 % across seven base models. Downstream reasoning performance improves by up to +6.6 points on in‑distribution benchmarks and shows consistent gains on out‑of‑distribution tasks such as GSM8K, Math‑500, and U‑Math. The margin‑stratified approach yields a mean utility margin of 0.244 versus 0.15‑0.20 for uniform sampling, confirming that high‑signal pairs accelerate learning.

In summary, CU‑DPO demonstrates that (1) continuous utility supervision can faithfully represent partial reasoning progress, (2) separating strategy selection from execution refinement eliminates conflicting gradient signals, (3) the method enjoys provable sample‑efficiency gains, and (4) it delivers tangible performance improvements on a suite of mathematical reasoning tasks. The framework is readily extensible to other domains requiring diverse problem‑solving approaches, such as code generation, scientific reasoning, and multi‑step planning.