Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations

Recent research has shown that large language models (LLMs) favor their own outputs when acting as judges, undermining the integrity of automated post-training and evaluation workflows. However, it is difficult to disentangle which evaluation biases are explained by narcissism versus general experimental confounds, distorting measurements of self-preference bias. We discover a core methodological confound which could reduce measurement error by 89.6%. Specifically, LLM evaluators may deliver self-preferring verdicts when the judge responds to queries which they completed incorrectly themselves; this would be true regardless of whether one of their responses is their own. To decouple self-preference signals from noisy outputs on hard problems, we introduce an Evaluator Quality Baseline, which compares the probability that a judge incorrectly votes for itself against the probability that it votes for an incorrect response from another model. Evaluating this simple baseline on 37,448 queries, only 51% of initial findings retain statistical significance. Finally, we turn towards characterizing the entropy of “easy” versus “hard” evaluation votes from LLM judges. Our corrective baseline enables future research on self-preference by eliminating noisy data from potential solutions. More widely, this work contributes to the growing body of work on cataloging and isolating judge-bias effects.

💡 Research Summary

The paper tackles the recently reported phenomenon that large language models (LLMs) tend to favor their own generated outputs when acting as judges—a behavior that has been labeled “self‑preference bias” or “narcissism.” The authors argue that prior findings conflate genuine self‑bias with a more mundane confound: judges are more likely to exhibit self‑preference on queries where they themselves produced an incorrect answer. To disentangle these effects, they first decompose the traditional bias metric (Bias = Self‑Preference – Accuracy) into two components: legitimate self‑preference (LSP) on correctly answered items and illegitimate self‑preference (ILSP) on incorrectly answered items. Since harmful bias resides entirely in ILSP, the focus shifts to measuring this term accurately.
The core methodological contribution is the “Evaluator Quality Baseline.” For each query where the judge model J gives an incorrect (or inferior) response, the authors retrieve a proxy model K that produces an answer of equivalent quality according to an oracle ground‑truth label G. By prompting J to compare its own answer o_J with the reference o_R and, separately, to compare K’s answer o_K with the same reference, they compute a per‑example preference differential Δs_J = s_J(o_J, o_R) – s_J(o_K, o_R). Averaging Δs_J over all matched examples yields a test statistic T_quality. The null hypothesis H₀: T_quality ≤ 0 asserts that J does not preferentially select its own output over an equally capable peer; rejecting H₀ indicates genuine self‑bias beyond what can be explained by task difficulty or uncertainty.
The authors apply this baseline to 9 publicly available, verifiable datasets (including MATH‑500, MBPP‑Plus, and MMLU) and 16 LLMs spanning open‑source (Llama‑3 variants, Qwen‑2.5, Gemma‑2, DeepSeek‑V3) and closed‑source (GPT‑4o, GPT‑3.5‑Turbo) families. They construct example‑level, outcome‑matched proxies rather than coarse model‑level matches, ensuring that each comparison controls for the exact difficulty of the query. Across a total of 37 448 evaluation instances, only 51 % of the originally reported self‑preference effects survive statistical significance when the baseline is applied. Moreover, the authors estimate that on average 89.6 % of the previously measured bias can be attributed to evaluator uncertainty on hard items (ILSP), dramatically reducing but not entirely eliminating evidence for a self‑bias.
Additional analyses explore whether prompting the judge with chain‑of‑thought (CoT) reasoning mitigates bias. The results show only modest reductions, contradicting earlier claims that CoT substantially curbs self‑preference. An entropy‑based investigation reveals that “hard” queries produce higher uncertainty in the judge’s probability distribution, leading to near‑random self‑selection when the judge is unsure. This supports the hypothesis that self‑preference is largely an artifact of uncertainty rather than a sophisticated form of situational awareness.
In summary, the paper demonstrates that much of the reported LLM narcissism can be explained by a simple confound: judges are more likely to favor their own answers when they are wrong. By introducing the Evaluator Quality Baseline, the authors provide a rigorous, statistically sound tool for future work to isolate true self‑bias from evaluation noise. Their findings call for a re‑evaluation of prior claims about LLM self‑awareness and suggest that future alignment and safety pipelines should incorporate such baselines to ensure unbiased meta‑evaluation.