Small-Margin Preferences Still Matter-If You Train Them Right

Preference optimization methods such as DPO align large language models (LLMs) using paired comparisons, but their effectiveness can be highly sensitive to the quality and difficulty of preference pairs. A common heuristic treats small-margin (ambiguous) pairs as noisy and filters them out. In this paper, we revisit this assumption and show that pair difficulty interacts strongly with the optimization objective: when trained with preference-based losses, difficult pairs can destabilize training and harm alignment, yet these same pairs still contain useful supervision signals when optimized with supervised fine-tuning (SFT). Motivated by this observation, we propose MixDPO, a simple yet effective difficulty-aware training strategy that (i) orders preference data from easy to hard (a curriculum over margin-defined difficulty), and (ii) routes difficult pairs to an SFT objective while applying a preference loss to easy pairs. This hybrid design provides a practical mechanism to leverage ambiguous pairs without incurring the optimization failures often associated with preference losses on low-margin data. Across three LLM-judge benchmarks, MixDPO consistently improves alignment over DPO and a range of widely-used variants, with particularly strong gains on AlpacaEval~2 length-controlled (LC) win rate.

💡 Research Summary

This paper revisits the role of small‑margin (ambiguous) preference pairs in aligning large language models (LLMs) with human feedback. While most recent preference‑optimization methods such as Direct Preference Optimization (DPO) treat low‑margin pairs as noisy and either filter them out or apply the same loss to all pairs, the authors demonstrate that these pairs can still provide valuable supervision if handled correctly. They first define pairwise difficulty as the rating‑score margin M = s_w − s_l, where a larger M indicates an easy pair and a smaller M an difficult one. Empirical analysis on the UltraFeedback and Argilla datasets shows that training exclusively on easy pairs yields higher AlpacaEval 2 win rates, faster convergence, higher reward accuracy, and larger reward margins, whereas difficult pairs cause slower optimization and a “likelihood displacement” phenomenon (both preferred and rejected responses receive lower log‑probabilities) when optimized with the standard DPO loss.

Motivated by these observations, the authors propose MixDPO, a difficulty‑aware curriculum strategy. The training data are sorted from easy to hard based on M, and a dynamic loss switch is applied: easy pairs are optimized with the usual DPO loss, while difficult pairs are trained with a supervised fine‑tuning (SFT) loss that simply maximizes the likelihood of the chosen response without penalizing the rejected one. This hybrid approach preserves the strong preference signal from easy pairs and avoids the instability caused by small‑margin pairs.

Experiments compare MixDPO against vanilla DPO and several popular variants (CPO, IPO, KTO, SimPO, SelectiveDPO) across three LLM‑judge benchmarks, most notably AlpacaEval 2 (including a length‑controlled split). MixDPO consistently improves alignment, achieving up to a 3.2 percentage‑point increase in LC win rate and 2.7 pp in overall win rate over the baseline DPO. Ablation studies confirm that the curriculum ordering alone helps, and that the SFT component is crucial for extracting signal from difficult pairs. Further evaluations on a different base model and on the Argilla‑7k dataset demonstrate that the method generalizes beyond the initial setup.

In summary, the paper shows that small‑margin preference pairs are not merely noise; when paired with a curriculum‑based ordering and an appropriate loss function, they contribute positively to LLM alignment. MixDPO offers a simple, computationally cheap modification to existing DPO pipelines and sets a new direction for data‑centric, difficulty‑aware preference learning.