Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarios

Reliable reward models (RMs) are critical for ensuring the safe alignment of large language models (LLMs). However, current RM evaluation methods focus solely on preference perception accuracies in given specific scenarios, obscuring the critical vulnerabilities of RMs in real-world scenarios. We identify the true challenge lies in assessing a novel dimension: Suitability, defined as conditional reliability under specific real-world perturbations. To this end, we introduce Reward Auditor, a hypothesis-testing framework specifically designed for RM suitability inference. Rather than answering “How accurate is the RM’s preference perception for given samples?”, it employs scientific auditing to answer: “Can we infer RMs exhibit systematic vulnerabilities in specific real-world scenarios?”. Under real-world perturbed scenarios, Reward Auditor quantifies statistical significance and effect size by auditing distribution degradation of RM preference perception confidence. This enables inference of both the certainty and severity of RM vulnerabilities across diverse real-world scenarios. This lays a solid foundation for building next-generation LLM alignment systems that are verifiably safe, more robust, and trustworthy.

💡 Research Summary

The paper addresses a critical gap in the evaluation of reward models (RMs) that serve as the backbone of reinforcement learning from human feedback (RLHF) for large language models (LLMs). While existing benchmarks focus on static accuracy—measuring how often an RM correctly predicts the preferred response on a fixed test set—they fail to capture systematic vulnerabilities that emerge under realistic, noisy conditions such as user typos, linguistic variations, format changes, or multilingual inputs. To fill this gap, the authors introduce a new evaluation dimension called “suitability,” defined as the conditional reliability of an RM when faced with specific real‑world perturbations.

The core contribution is Reward Auditor, a hypothesis‑testing framework that infers suitability by comparing the distribution of preference‑perception confidence scores before and after applying a perturbation function P. For each preference pair (x, y_w, y_l), the RM produces a confidence Pθ(y_w ≻ y_l | x). The original dataset D yields a set M of these confidences, while the perturbed dataset D′ = P(D) yields M′. Suitability is formalized as a stochastic dominance test: under the null hypothesis H₀, M and M′ are identically distributed; under the alternative H₁, M first‑order stochastically dominates M′, indicating systematic confidence degradation.

To operationalize this test, the authors adopt a paired‑sample design: each original confidence Mi is paired with its perturbed counterpart M′i, and the difference ΔMi = Mi − M′i is computed. They then calculate two key metrics: (1) a paired‑sample t‑statistic ˆt for statistical significance, and (2) a paired‑sample Cohen’s d ˆe for practical effect size. The p‑value is obtained via a non‑parametric paired permutation test, which constructs the null distribution by randomly swapping the labels of each pair, thereby avoiding assumptions about underlying data distributions. The effect size and p‑value are combined into a “Suitability Risk Report” rS, which includes tiered significance markers (*, **, ***).

Because real‑world robustness must be assessed across many perturbation types, the authors evaluate ten systematic scenarios ranging from simple typographical noise to complex multilingual and stylistic transformations. Conducting multiple hypothesis tests raises the risk of false discoveries; to control the false discovery rate (FDR), they devise a group‑aware Benjamini‑Hochberg procedure that adjusts p‑values within each perturbation group while preserving overall FDR control.

Empirical studies cover several state‑of‑the‑art RM families: discriminative (LM backbone + linear head), generative (autoregressive preference scoring), and Direct Preference Optimization (DPO) models. Across all models, many perturbations trigger statistically significant confidence drops (p < 0.05) and effect sizes exceeding a pre‑specified tolerance margin m, leading to a rejection of suitability. Notably, perturbations that mimic user typos or language‑specific idioms often produce the largest risk scores, suggesting that even high‑performing RMs can be brittle in everyday usage.

The paper’s contributions are threefold: (1) introducing suitability as a rigorous, statistically grounded metric for RM robustness; (2) providing a complete auditing pipeline—paired‑sample effect‑size estimation, exact permutation testing, and FDR‑controlled multi‑scenario analysis; (3) demonstrating that quantified suitability risks correlate with downstream alignment performance, thereby validating the practical relevance of the audit.

Limitations are acknowledged. The perturbation functions are handcrafted approximations of real user behavior; their fidelity to actual deployment conditions may vary. The tolerance margin m is a domain‑specific hyperparameter that requires careful calibration. Moreover, while the permutation test offers exact p‑values, it can be computationally intensive for very large test sets.

In summary, Reward Auditor transforms RM evaluation from a static accuracy checklist into a scientific auditing process capable of detecting and quantifying systematic vulnerabilities under realistic conditions. By marrying statistical significance with effect‑size relevance and controlling for multiple testing, the framework offers a robust, reproducible method for certifying the safety and reliability of reward models before they are deployed in high‑stakes LLM applications. This work paves the way for more trustworthy alignment pipelines and sets a new standard for RM robustness assessment.