Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference

📝 Original Info

• Title: Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference
• ArXiv ID: 2511.04286
• Date: 2025-11-06
• Authors: ** 정보가 제공되지 않음 (논문에 명시된 저자 정보를 확인해 주세요.) **

📝 Abstract

Learning from human preferences is a cornerstone of aligning machine learning models with subjective human judgments. Yet, collecting such preference data is often costly and time-consuming, motivating the need for more efficient learning paradigms. Two established approaches offer complementary advantages: RLHF scales effectively to high-dimensional tasks such as LLM fine-tuning, while PBO achieves greater sample efficiency through active querying. We propose a hybrid framework that unifies RLHF's scalability with PBO's query efficiency by integrating an acquisition-driven module into the RLHF pipeline, thereby enabling active and sample-efficient preference gathering. We validate the proposed approach on two representative domains: (i) high-dimensional preference optimization and (ii) LLM fine-tuning. Experimental results demonstrate consistent improvements in both sample efficiency and overall performance across these tasks.

💡 Deep Analysis

📄 Full Content

Reference