Reward Model Interpretability via Optimal and Pessimal Tokens

Reward modeling has emerged as a crucial component in aligning large language models with human values. Significant attention has focused on using reward models as a means for fine-tuning generative models. However, the reward models themselves – which directly encode human value judgments by turning prompt-response pairs into scalar rewards – remain relatively understudied. We present a novel approach to reward model interpretability through exhaustive analysis of their responses across their entire vocabulary space. By examining how different reward models score every possible single-token response to value-laden prompts, we uncover several striking findings: (i) substantial heterogeneity between models trained on similar objectives, (ii) systematic asymmetries in how models encode high- vs low-scoring tokens, (iii) significant sensitivity to prompt framing that mirrors human cognitive biases, and (iv) overvaluation of more frequent tokens. We demonstrate these effects across ten recent open-source reward models of varying parameter counts and architectures. Our results challenge assumptions about the interchangeability of reward models, as well as their suitability as proxies of complex and context-dependent human values. We find that these models can encode concerning biases toward certain identity groups, which may emerge as unintended consequences of harmlessness training – distortions that risk propagating through the downstream large language models now deployed to millions.

💡 Research Summary

**
This paper tackles a largely overlooked component of modern language‑model alignment pipelines: the reward model (RM) itself. While much recent work has focused on using RMs to fine‑tune large language models (LLMs) or on improving the quality of human feedback, the internal behavior of the RMs—how they actually encode human value judgments—has received far less systematic scrutiny.

The authors introduce a novel interpretability technique they call “optimal and pessimal token analysis.” For a given value‑laden prompt (e.g., “Is discrimination based on race ever justified?”), they exhaustively evaluate the RM’s scalar score for every possible single‑token response in the model’s vocabulary (roughly 50 k–100 k tokens depending on the model). By mapping the full distribution of token‑level scores, they can identify which tokens are deemed “optimal” (high‑scoring) and which are “pessimal” (low‑scoring), and they can examine how these patterns differ across models, prompt framings, and token frequencies.

The study evaluates ten recent open‑source reward models spanning a range of architectures (LLaMA‑2, GPT‑Neo, Falcon, etc.) and parameter counts (from 2 B to 70 B). All models were trained on the same human‑feedback datasets (including the “harmlessness” fine‑tuning stage) to keep the training objective constant while allowing architectural and scaling differences to surface.

Four major findings emerge:

1. Substantial heterogeneity between models. Even when trained on identical objectives, the token‑score landscapes differ dramatically. Smaller models tend to over‑reward high‑frequency tokens (e.g., common function words) and under‑reward rare or domain‑specific tokens, whereas larger models exhibit a more balanced distribution but still display systematic biases.

2. Asymmetric encoding of high‑ vs. low‑scoring tokens. The set of optimal tokens clusters around positive, socially acceptable, or fact‑based expressions, while the pessimal set is broader and includes negative, offensive, or culturally sensitive words. The low‑score region is noticeably larger, suggesting that the RM’s safety‑oriented training pushes the model to be overly conservative in penalizing a wide swath of language.

3. Prompt‑framing sensitivity mirroring human cognitive biases. Minor rephrasings of the same question (“Why do you think…?” vs. “What is your opinion on…?”) cause substantial shifts in which tokens receive high scores. This indicates that the RM inherits framing effects from the human feedback data, and that the model’s value judgments can be steered unintentionally by subtle prompt changes.

4. Identity‑group bias. Tokens associated with particular demographic groups (gender, race, religion, etc.) are sometimes systematically under‑ or over‑valued. For example, tokens linked to “women” may receive unusually low scores, while certain ethnicity‑related tokens appear disproportionately in the high‑score region. The authors trace these patterns back to the harmlessness fine‑tuning stage, where attempts to suppress toxic language inadvertently introduced collateral bias.

Collectively, these results challenge the assumption that reward models are interchangeable proxies for human values. Because downstream LLMs directly inherit the RM’s scalar signal, any token‑level bias or asymmetry can propagate into the generated text, potentially amplifying unfair or harmful behavior at scale.

The exhaustive token‑scoring methodology itself constitutes a powerful diagnostic tool. It enables (a) quantitative comparison of reward models for model‑selection purposes, (b) fine‑grained debugging of specific token biases, and (c) the creation of benchmark suites that explicitly test for framing sensitivity and identity bias. The authors suggest several avenues for future work: extending the analysis to multi‑token sequences, diversifying the human‑feedback data to cover a broader set of cultural contexts, and developing prompt‑design guidelines that minimize framing effects.

In conclusion, while reward models are indispensable for aligning LLMs with human preferences, their internal representations are far from neutral. The paper’s token‑level interpretability framework reveals systematic heterogeneity, safety‑driven over‑penalization, framing‑driven volatility, and demographic bias across state‑of‑the‑art open‑source RMs. Understanding and mitigating these issues is essential before deploying aligned models at the scale of millions of users.