A-IPO: Adaptive Intent-driven Preference Optimization

Human preferences are diverse and dynamic, shaped by regional, cultural, and social factors. Existing alignment methods like Direct Preference Optimization (DPO) and its variants often default to majority views, overlooking minority opinions and failing to capture latent user intentions in prompts. To address these limitations, we introduce \underline{\textbf{A}}daptive \textbf{\underline{I}}ntent-driven \textbf{\underline{P}}reference \textbf{\underline{O}}ptimization (\textbf{A-IPO}). Specifically,A-IPO introduces an intention module that infers the latent intent behind each user prompt and explicitly incorporates this inferred intent into the reward function, encouraging stronger alignment between the preferred model’s responses and the user’s underlying intentions. We demonstrate, both theoretically and empirically, that incorporating an intention–response similarity term increases the preference margin (by a positive shift of $λ,Δ\mathrm{sim}$ in the log-odds), resulting in clearer separation between preferred and dispreferred responses compared to DPO. For evaluation, we introduce two new benchmarks, Real-pref, Attack-pref along with an extended version of an existing dataset, GlobalOpinionQA-Ext, to assess real-world and adversarial preference alignment. Through explicit modeling of diverse user intents,A-IPO facilitates pluralistic preference optimization while simultaneously enhancing adversarial robustness in preference alignment. Comprehensive empirical evaluation demonstrates that A-IPO consistently surpasses existing baselines, yielding substantial improvements across key metrics: up to +24.8 win-rate and +45.6 Response-Intention Consistency on Real-pref; up to +38.6 Response Similarity and +52.2 Defense Success Rate on Attack-pref; and up to +54.6 Intention Consistency Score on GlobalOpinionQA-Ext.

💡 Research Summary

The paper introduces Adaptive Intent‑driven Preference Optimization (A‑IPO), a novel framework that extends Direct Preference Optimization (DPO) by explicitly modeling latent user intent and incorporating it into the reward function. The authors argue that existing alignment methods, including DPO and its variants, tend to favor majority preferences, ignore minority viewpoints, and fail to capture the hidden intentions embedded in user prompts. To address these shortcomings, A‑IPO adds an intention module that (1) decomposes a prompt into sub‑questions, (2) retrieves relevant external knowledge (e.g., Wikipedia), (3) performs fact‑checking with a state‑of‑the‑art verifier, and (4) produces a structured intent representation I.

The reward is re‑parameterized from the standard DPO form r(x, y) to r′(x, y, I) = r(x, y) + λ·sim(y, I), where sim(y, I) measures semantic similarity between the generated response and the inferred intent. This similarity term yields a positive shift of λ·Δsim in the log‑odds of the preference probability, effectively widening the preference margin between the “winner” response y_w and the “loser” response y_l. The authors formalize this within a Bradley‑Terry model augmented with a latent variable I, and they derive a variational lower bound (ELBO) to handle the intractable expectation over I. The overall training objective combines three components: (1) the negative expected log‑likelihood of the preference model conditioned on inferred intents, (2) the similarity regularization term, and (3) a KL‑divergence term that keeps the inferred intent distribution close to a prior and limits drift from the reference policy.

To evaluate A‑IPO, the authors construct three new benchmarks: Real‑pref (real‑world user feedback across diverse cultures and languages), Attack‑pref (adversarial prompts including injection and fact‑distortion attacks), and GlobalOpinionQA‑Ext (an extension of GlobalOpinionQA with added cultural labels). Across these datasets, A‑IPO consistently outperforms strong baselines such as standard DPO, GDPO, SafeDPO, and ADPO. Reported gains include up to +24.8 % win‑rate and +45.6 points in Response‑Intention Consistency on Real‑pref; +38.6 points in Response Similarity and +52.2 % in Defense Success Rate on Attack‑pref; and +54.6 points in Intention Consistency Score on GlobalOpinionQA‑Ext. Notably, the improvements are most pronounced for minority groups and under adversarial conditions, demonstrating both pluralistic alignment and robustness.

The paper also discusses limitations. The intention module relies on high‑quality intent annotations; performance may degrade for low‑resource languages or cultures lacking labeled data. The approach depends on external knowledge bases, which can become stale, and hyper‑parameters λ and β require careful tuning. Future work is suggested in the directions of multimodal intent inference, continual online learning, and meta‑learning for automatic hyper‑parameter optimization.

In summary, A‑IPO offers a principled, theoretically grounded, and empirically validated solution to the problem of aligning large language models with diverse, context‑dependent human preferences while simultaneously enhancing resistance to adversarial attacks. It represents a significant step toward pluralistic and secure AI alignment.