Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers

Large language models (LLMs) can acquire new knowledge through fine-tuning, but this process exhibits a puzzling duality: models can generalize remarkably from new facts, yet are also prone to hallucinating incorrect information. However, the reasons for this phenomenon remain poorly understood. In this work, we argue that both behaviors stem from a single mechanism known as out-of-context reasoning (OCR): the ability to deduce implications by associating concepts, even those without a causal link. Our experiments across five prominent LLMs confirm that OCR indeed drives both generalization and hallucination, depending on whether the associated concepts are causally related. To build a rigorous theoretical understanding of this phenomenon, we then formalize OCR as a synthetic factual recall task. We empirically show that a one-layer single-head attention-only transformer with factorized output and value matrices can learn to solve this task, while a model with combined weights cannot, highlighting the crucial role of matrix factorization. Our theoretical analysis shows that the OCR capability can be attributed to the implicit bias of gradient descent, which favors solutions that minimize the nuclear norm of the combined output-value matrix. This mathematical structure explains why the model learns to associate facts and implications with high sample efficiency, regardless of whether the correlation is causal or merely spurious. Ultimately, our work provides a theoretical foundation for understanding the OCR phenomenon, offering a new lens for analyzing and mitigating undesirable behaviors from knowledge injection.

💡 Research Summary

The paper investigates why large language models (LLMs) exhibit both strong generalization and harmful hallucination after fine‑tuning with new factual knowledge. The authors propose a unified mechanism called out‑of‑context reasoning (OCR), defined as the model’s ability to infer implications by linking concepts that may not share a causal relationship. When the linked concepts are causally related (e.g., “people living in Paris speak French”), OCR yields correct generalization; when they are unrelated (e.g., “people living in Paris code in Java”), OCR produces hallucinations.

Empirical validation is performed on five state‑of‑the‑art LLMs (Gemma‑2‑9B, OLMo‑7B, Qwen‑2‑7B, Mistral‑7B‑v0.3, Llama‑3‑8B) using a synthetic dataset. The dataset consists of subject tokens, two relation tokens (r₁, r₂), and two answer sets: factual answers A₁ and implied answers A₂. Five association types are tested: a real “City‑Language” pair (causal) and four fabricated pairs (“City‑Language (counter‑factual)”, “Country‑Code”, “Profession‑Color”, “Sport‑Music”) that lack causal grounding. Training provides facts for all subjects and implications for only a small subset (20 % of subjects). Evaluation uses mean‑rank of the correct implication among all candidates. Results show near‑perfect generalization for the causal pair (mean‑rank ≈ 0) and substantially higher ranks for the non‑causal pairs, confirming that OCR can drive both desirable and undesirable behavior. Notably, only a handful of examples are needed for the models to learn these associations, indicating high sample efficiency.

To understand OCR theoretically, the authors formalize it as a symbolic factual recall task. Input sequences encode a subject‑relation pair, and the model must output the corresponding answer token. They study a one‑layer, single‑head attention‑only transformer under two parameterizations: (1) a factorized model with separate output matrix (W_O) and value matrix (W_V); (2) a non‑factorized model that combines them into a single matrix (W_{OV}=W_O W_V^\top). Experiments reveal that the factorized model learns OCR successfully, while the combined‑weight model fails.

The key insight comes from the implicit bias of gradient descent. For the cross‑entropy loss used, gradient flow implicitly minimizes the nuclear norm (trace norm) of the combined matrix (W_{OV}). This bias favors low‑rank solutions that capture shared structure across many subject‑relation pairs. When the training data contains repeated fact‑implication pairs ((b_i, c_i)) across different subjects, the low‑rank bias enables the factorized model to quickly discover a shared subspace linking (r_1) facts to (r_2) implications, thus achieving OCR with very few examples. In contrast, the non‑factorized model cannot separate the roles of output and value matrices, so the nuclear‑norm minimization does not produce the necessary factorization, and OCR does not emerge.

The theoretical analysis yields concrete conditions for OCR: the proportion of observed fact‑implication pairs must be sufficiently high for the low‑rank solution to dominate. This explains why LLMs, which already encode many real-world causal relations from pre‑training, can generalize from a tiny amount of fine‑tuning data. The same mechanism, however, also explains hallucination: when the injected association is spurious, the model still forms the low‑rank link and erroneously projects it onto unseen subjects.

Finally, the authors discuss mitigation strategies. Enforcing a factorized architecture or explicitly regularizing the nuclear norm can control OCR’s strength, potentially preserving beneficial generalization while reducing hallucination. Adding meta‑information about causal validity during fine‑tuning could also guide the model to weight causal links more heavily. The work thus provides a rigorous theoretical foundation for OCR, linking empirical observations of LLM behavior to the optimization dynamics of attention‑only transformers, and suggests concrete avenues for safer knowledge injection.