Multi-Objective Reward and Preference Optimization: Theory and Algorithms

This thesis develops theoretical frameworks and algorithms that advance constrained reinforcement learning (RL) across control, preference learning, and alignment of large language models. The first contribution addresses constrained Markov Decision Processes (CMDPs) under the average-cost criterion through the Average-Constrained Policy Optimization (ACPO) algorithm. ACPO integrates sensitivity analysis with trust-region updates to ensure stable constraint handling, achieving state-of-the-art empirical performance with theoretical guarantees. Constrained RL is then extended to finite-horizon settings via e-COP, the first policy optimization method for episodic CMDPs. Built on an episodic policy difference lemma, e-COP offers provable performance, simplicity, and scalability in safety-critical environments. The thesis then investigates reinforcement learning from human preferences. warmPref-PS introduces a posterior sampling strategy for linear bandits that integrates offline preference data from heterogeneous raters into online learning. Explicit modeling of rater competence yields substantial regret reduction and more efficient data collection for RLHF. The PSPL algorithm further advances preference-based RL by jointly sampling reward models and transition dynamics from pairwise trajectory comparisons, providing Bayesian simple-regret guarantees and robust empirical identification of optimal policies. The final contribution applies these methods to large-scale model alignment. A multi-objective constrained optimization view yields MOPO, an iterative algorithm with closed-form updates that scales to multi-billion-parameter language models and remains robust across alignment settings. Collectively, the thesis unifies constrained RL across average-cost, episodic, and preference-driven paradigms, delivering theoretical advances and practical tools for safe and aligned decision-making.

💡 Research Summary

This thesis presents a comprehensive and unified theoretical and algorithmic framework for advancing Constrained Reinforcement Learning (CRL) across three critical domains: control, preference learning, and the alignment of Large Language Models (LLMs). The core objective is to bridge the gap between theoretical stability in constrained environments and the practical necessity of aligning AI behavior with human values and safety constraints.

The research is structured into three primary pillars. The first pillar focuses on the fundamental optimization of Constrained Markov Decision Processes (CMDPs). The author introduces ACPO (Average-Constrained Policy Optimization) to address the average-cost criterion. By integrating sensitivity analysis with trust-region updates, ACPO ensures stable constraint handling and achieves state-of-the-art empirical results with rigorous theoretical guarantees. Furthermore, the thesis extends these advancements to finite-horizon settings through e-COP, the first policy optimization method specifically designed for episodic CMDPs. Utilizing an episodic policy difference lemma, e-COP provides a scalable and provable approach for safety-critical applications where episodic transitions are the norm.

The second pillar investigates Reinforcement Learning from Human Preferences (RLHF). To tackle the challenges of noisy and heterogeneous human feedback, the author proposes warmPref-PS, a posterior sampling strategy for linear bandits. This algorithm effectively integrates offline preference data from diverse raters into online learning by explicitly modeling rater competence, thereby significantly reducing regret and improving data collection efficiency. Building upon this, the PSPL algorithm advances preference-based RL by jointly sampling reward models and transition dynamics from pairwise trajectory comparisons. This Bayesian approach provides robust simple-regret guarantees, enabling the efficient identification of optimal policies from complex trajectory data.

The third and final pillar applies these theoretical breakthroughs to the large-scale alignment of LLMs. By framing LLM alignment as a multi-objective constrained optimization problem, the author develops MOPO, an iterative algorithm featuring closed-form updates. MOPO is specifically engineered to scale to multi-billion-parameter models, maintaining robustness across various alignment settings and computational constraints.

In conclusion, this thesis achieves a significant unification of constrained RL paradigms—covering average-cost, episodic, and preference-driven frameworks. By delivering both theoretical advancements and practical, scalable algorithms, the work provides essential tools for the development of safe, reliable, and human-aligned autonomous decision-making systems in the era of foundation models.