Sparse Reward Subsystem in Large Language Models

In this paper, we identify a sparse reward subsystem within the hidden states of Large Language Models (LLMs), drawing an analogy to the biological reward subsystem in the human brain. We demonstrate that this subsystem contains value neurons that represent the model’s internal expectation of state value, and through intervention experiments, we establish the importance of these neurons for reasoning. Our experiments reveal that these value neurons are robust across diverse datasets, model scales, and architectures; furthermore, they exhibit significant transferability across different datasets and models fine-tuned from the same base model. By examining cases where value predictions and actual rewards diverge, we identify dopamine neurons within the reward subsystem which encode reward prediction errors (RPE). These neurons exhibit high activation when the reward is higher than expected and low activation when the reward is lower than expected.

💡 Research Summary

The paper “Sparse Reward Subsystem in Large Language Models” proposes that large language models (LLMs) contain a compact, brain‑inspired reward subsystem embedded in their hidden states. Within this subsystem two types of neurons are identified: (1) value neurons that encode the model’s internal estimate of the expected value of the current state, and (2) dopamine‑like neurons that signal reward prediction error (RPE) by activating strongly when the actual reward exceeds the expectation and suppressing when it falls short.

To uncover these neurons the authors train a lightweight two‑layer MLP “value probe” on each transformer layer using temporal‑difference (TD) learning. The probe takes the full hidden vector as input and outputs a scalar value prediction. After training, they prune input dimensions based on the L1 norm of the probe’s first‑layer weights, progressively removing up to 80 % of the dimensions while monitoring the area under the ROC curve (AU‑ROC) for predicting final binary reward (correct/incorrect answer). Remarkably, AU‑ROC remains stable even when more than 99 % of the dimensions are removed, indicating that a tiny fraction (often <1 %) of hidden units—designated as value neurons—carry most of the reward‑related information.

The functional importance of these neurons is demonstrated through intervention experiments. In the Qwen‑2.5‑7B model, zero‑out the top 1 % of value neurons in a given layer leads to a dramatic drop in accuracy (average loss > 55 percentage points) on the MATH500 benchmark, whereas randomly zero‑ing the same proportion of neurons produces negligible change. This suggests that value neurons are causally involved in the model’s reasoning process.

Robustness and transferability are examined across five benchmark datasets (GSM8K, MATH500, Minerva Math, ARC, MMLU‑STEM), three model scales (0.5 B, 7 B, 14 B), and four architectures (Qwen, Llama, Phi, Gemma). In every setting the AU‑ROC curves stay flat or even improve as pruning increases, confirming that the reward subsystem is a universal feature of LLMs rather than an artifact of a particular model or task. Moreover, models fine‑tuned from the same base exhibit nearly identical locations of value neurons, indicating that the subsystem is largely preserved during downstream adaptation.

Dopamine‑like neurons are identified by analyzing cases where the value probe’s prediction diverges from the actual reward. Their activation patterns align with classic RPE signatures: heightened firing for positive prediction errors and suppressed firing for negative errors. This mirrors the role of dopaminergic cells in the ventral tegmental area and substantia nigra of the brain.

The authors acknowledge several limitations. The reward signal is binary, which may oversimplify nuanced value judgments. The value probe, though simple, could still overfit to the training data, and the experiments focus mainly on mathematical reasoning tasks, leaving open the question of generalization to dialogue or commonsense domains. Finally, while the analogy to biological value and dopamine neurons is compelling, the underlying mechanisms differ substantially, and the terminology should be used cautiously.

In sum, the work provides the first systematic evidence for a sparse, brain‑like reward subsystem inside LLMs, demonstrates its causal relevance for reasoning, and shows its robustness across data, scale, and architecture. These findings open new avenues for model interpretability, targeted interventions, and cross‑disciplinary research linking neuroscience and artificial intelligence.