Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling

Reward models learned from human preferences are central to aligning large language models (LLMs) via reinforcement learning from human feedback, yet they are often vulnerable to reward hacking due to noisy annotations and systematic biases such as response length or style. We propose Bayesian Non-Negative Reward Model (BNRM), a principled reward modeling framework that integrates non-negative factor analysis into Bradley-Terry (BT) preference model. BNRM represents rewards through a sparse, non-negative latent factor generative process that operates at two complementary levels: instance-specific latent variables induce disentangled reward representations, while sparsity over global latent factors acts as an implicit debiasing mechanism that suppresses spurious correlations. Together, this disentanglement-then-debiasing structure enables robust uncertainty-aware reward learning. To scale BNRM to modern LLMs, we develop an amortized variational inference network conditioned on deep model representations, allowing efficient end-to-end training. Extensive empirical results demonstrate that BNRM substantially mitigates reward over-optimization, improves robustness under distribution shifts, and yields more interpretable reward decompositions than strong baselines.

💡 Research Summary

The paper addresses a critical weakness in current reinforcement learning from human feedback (RLHF) pipelines: reward models (RMs) learned from noisy human preference data are prone to reward hacking, where policies exploit spurious correlations such as response length or stylistic cues to achieve high proxy scores while diverging from true human intent. Traditional reward modeling relies on a deterministic scalar function built on top of a pretrained language model backbone and a linear head, trained with the Bradley‑Terry (BT) ranking loss. This approach suffers from two fundamental issues: (1) it provides no uncertainty quantification, leading to over‑confident scoring, and (2) its dense latent representation makes it easy for the model to latch onto shortcut features, resulting in systematic bias.

To overcome these problems, the authors propose the Bayesian Non‑Negative Reward Model (BNRM). BNRM integrates non‑negative factor analysis (NFA) into a Bayesian extension of the BT model, introducing two complementary sparsity mechanisms. At the instance level, each prompt‑response pair (x, y) is associated with a non‑negative latent vector θ ∈ ℝ⁺ᴷ drawn from a Gamma prior. Sparsity in θ forces each example to activate only a small subset of latent factors, encouraging disentangled, interpretable representations of semantic preference dimensions (e.g., accuracy, creativity, conciseness). At the global level, a shared dictionary Φ ∈ ℝ⁺ᴷ×¹, also Gamma‑distributed, defines a universal set of reward factors. Global sparsity on Φ acts as a regularizer that suppresses population‑wide spurious correlations, effectively debiasing the reward function.

The generative process proceeds as follows: for a given (x, y), sample θ, compute a non‑negative reward r = θᵀΦ, and for a pair of candidate responses (y₁, y₂) generate the observed human preference via the BT likelihood σ(r₁ − r₂). This construction captures both aleatoric uncertainty (through stochastic θ) and epistemic uncertainty (through stochastic Φ). Because exact posterior inference is intractable in large‑scale settings, the authors employ variational inference. They treat the pretrained LLM backbone f as an amortized encoder: the deterministic high‑dimensional feature z = f(x, y) parameterizes a Weibull variational distribution q(θ|x, y). The Weibull choice enables re‑parameterization while respecting the non‑negative constraint. A separate Weibull variational distribution q(Φ) approximates the global posterior. The training objective is the evidence lower bound (ELBO), which combines the BT log‑likelihood with KL divergences that enforce the Gamma priors and encourage sparsity.

Extensive experiments compare BNRM against strong baselines: a standard BT reward model, ensemble‑based uncertainty methods, and information‑bottleneck approaches. In reward‑over‑optimization tests, policies trained with BNRM exhibit markedly reduced exploitation of length or stylistic shortcuts (over 30 % reduction in hackability). Under distribution shift (new topics, varied response lengths), BNRM maintains higher preference prediction accuracy (5–7 % gain) and provides calibrated uncertainty estimates that can be used to down‑weight unreliable reward signals during policy updates. Moreover, visualizing the learned θ and Φ reveals coherent, disentangled factors corresponding to human‑interpretable dimensions, confirming the model’s interpretability advantage.

The paper acknowledges limitations: the strength of the sparsity hyperparameters (α₀, γ₀) must be carefully tuned to avoid under‑expressivity, and the Weibull variational approximation may not capture highly complex preference structures. Future work includes exploring hybrid non‑negative/real‑valued factor models, richer priors, and integrating Bayesian reward uncertainty directly into the RL objective for safer policy optimization.

In summary, BNRM offers a principled, scalable, and uncertainty‑aware reward modeling framework that fundamentally reshapes the reward representation to be sparse, non‑negative, and interpretable. By doing so, it mitigates reward hacking, improves robustness to out‑of‑distribution data, and enhances the safety and transparency of RLHF pipelines.