A Statistical Framework for Alignment with Biased AI Feedback

Modern alignment pipelines are increasingly replacing expensive human preference labels with evaluations from large language models (LLM-as-Judge). However, AI labels can be systematically biased compared to high-quality human feedback datasets. In this paper, we develop two debiased alignment methods within a general framework that accommodates heterogeneous prompt-response distributions and external human feedback sources. Debiased Direct Preference Optimization (DDPO) augments standard DPO with a residual-based correction and density-ratio reweighting to mitigate systematic bias, while retaining DPO’s computational efficiency. Debiased Identity Preference Optimization (DIPO) directly estimates human preference probabilities without imposing a parametric reward model. We provide theoretical guarantees for both methods: DDPO offers a practical and computationally efficient solution for large-scale alignment, whereas DIPO serves as a robust, statistically optimal alternative that attains the semiparametric efficiency bound. Empirical studies on sentiment generation, summarization, and single-turn dialogue demonstrate that the proposed methods substantially improve alignment efficiency and recover performance close to that of an oracle trained on fully human-labeled data.

💡 Research Summary

The paper tackles a pressing problem in modern language‑model alignment: the growing reliance on LLM‑as‑Judge (AI‑generated preference labels) to replace costly human annotations. While AI judges can produce massive amounts of pairwise preference data, they often exhibit systematic biases (e.g., verbosity, self‑enhancement, positional bias) that diverge from true human preferences. The authors formalize this mismatch by treating AI labels as biased observations of an underlying human preference distribution and propose a unified statistical framework for debiasing when both human‑labeled and AI‑labeled datasets are available.

Two novel algorithms are introduced. Debiased Direct Preference Optimization (DDPO) builds on Direct Preference Optimization (DPO), which assumes a Bradley‑Terry model and optimizes a regularized log‑likelihood loss. DDPO augments the standard DPO objective with (1) a residual‑based correction that estimates the average difference between AI and human labels on a small human‑labeled set, and (2) a density‑ratio weighting term that accounts for distributional shift between the response generation policies used for the human and AI data. The corrected loss (\mathcal{L}{\text{DDPO}} = \mathcal{L}{\text{DPO}} - \mathcal{L}_{\text{B}}) can be minimized with stochastic gradient descent, preserving DPO’s computational simplicity while asymptotically eliminating bias at a rate (O_p(n^{-1/2}+N^{-1/2})).

Debiased Identity Preference Optimization (DIPO) departs from the Bradley‑Terry assumption entirely. It directly estimates the human preference probability (\mathbb{P}(\pi \succ \pi_{\text{ref}})) using AI‑judge outputs (\tilde g) and Monte‑Carlo sampling of responses from the target policy (\pi) and a reference policy (\pi_{\text{ref}}). A bias term, derived from the human dataset, is subtracted using a similar density‑ratio weighting. The resulting estimator attains the semiparametric efficiency bound, meaning it achieves the lowest possible asymptotic variance among all regular estimators that combine the two data sources.

Theoretical contributions include regret bounds that scale with policy dimension and sample size, asymptotic normality of both estimators, and a clean decomposition of bias into AI‑judge systematic error and distribution‑shift error.

Empirically, the methods are evaluated on three tasks: sentiment‑controlled generation, news summarization (CNN/DailyMail), and single‑turn dialogue (Persona‑Chat). In each case, a modest human dataset (≈5 k pairs) is combined with a large AI‑generated set (≈100 k pairs). DDPO recovers 30‑45 % of the data efficiency of a full‑human oracle, while DIPO reaches 92 % of the oracle’s human‑preference score. Detailed bias analyses show that DDPO and DIPO reduce verbosity bias from 0.12 to 0.07/0.05 and positional bias from 0.09 to 0.04/0.03, respectively.

Overall, the paper delivers a practical, statistically principled solution for leveraging cheap AI feedback without inheriting its biases. DDPO offers a drop‑in replacement for DPO in large‑scale pipelines, and DIPO provides a theoretically optimal, model‑free alternative. Future directions include extending the framework to multi‑turn conversations, multimodal feedback, and scenarios with extremely sparse human annotations, as well as integrating Bayesian priors for dynamic, online alignment loops.