Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure

Despite their impressive capabilities, LLMs exhibit a basic generalization failure known as the Reversal Curse, where they struggle to learn reversible factual associations. Understanding why this occurs could help identify weaknesses in current models and advance their generalization and robustness. In this paper, we conjecture that the Reversal Curse in LLMs is a manifestation of the long-standing binding problem in cognitive science, neuroscience and AI. Specifically, we hypothesize two primary causes of the Reversal Curse stemming from transformers’ limitations in conceptual binding: the inconsistency and entanglements of concept representations. We perform a series of experiments that support these conjectures. Our exploration leads to a model design based on JEPA (Joint-Embedding Predictive Architecture) that for the first time breaks the Reversal Curse without side-stepping it with specialized data augmentation or non-causal masking, and moreover, generalization could be further improved by incorporating special memory layers that support disentangled concept representations. Our research opens up the broader fundamental challenge of designing models capable of learning systematic conceptual binding with less human scaffolding.

💡 Research Summary

The paper investigates a basic generalization failure of large language models (LLMs) known as the Reversal Curse: after learning a fact in one direction (e.g., “Tom Smith’s wife is Mary Stone”), the model often cannot correctly answer the reverse query (“Mary Stone’s husband is …”). The authors argue that this phenomenon is not merely a data‑scarcity or objective‑function issue but a concrete manifestation of the long‑standing binding problem from cognitive science and neuroscience. They focus on conceptual binding—the ability of a system to integrate distributed representations into coherent, reusable concepts—and identify two structural shortcomings of standard transformer architectures that they claim give rise to the Reversal Curse.

1. Inconsistency of concept representations. When an entity switches roles between subject and object, the transformer creates distinct internal embeddings for the same underlying concept in different layers (lower layers for perception, upper layers for generation). Because there is no mechanism to enforce that these embeddings refer to the same abstract entity, the model treats the forward and reverse facts as unrelated, preventing the formation of a unified reversible rule.

2. Entanglement of concept representations. The mapping from surface‑form tokens to concept embeddings is performed by a shared MLP and linear projection. During gradient descent, the update of a given concept’s embedding is contaminated by gradients from other simultaneously active concepts. The degree of contamination is proportional to the inner product of their hidden activations (αᵀβ). As depth increases, these cross‑concept interferences accumulate, degrading the model’s ability to keep distinct concepts separate and thus harming generalization on reversible relations.

The authors validate these hypotheses through two complementary experimental regimes.

Concept‑level experiments replace textual inputs with learned embeddings for entities and relations, allowing the model to operate directly on abstract concepts. Standard decoder‑only transformers (1–18 layers) trained on millions of steps achieve mean reciprocal rank (MRR) scores above 0.90, demonstrating that transformers are capable of learning perfect reversal when the binding problem is removed.

Surface‑form experiments revert to ordinary token‑level inputs. Here, performance drops sharply with depth; 12‑ and 18‑layer models attain MRR below 0.80. Analyses of internal representations reveal low cosine similarity between the same entity’s subject‑role and object‑role embeddings, and high αᵀβ values indicating strong entanglement during training.

To overcome these limitations, the paper proposes a Joint‑Embedding Predictive Architecture (JEPA)‑based model. The architecture first maps input tokens to a recognition module that produces stable concept embeddings. These embeddings are then trained with an in‑batch contrastive loss, encouraging consistency across contexts. The decoder predicts the next concept embedding rather than raw tokens, thereby bypassing the noisy surface‑to‑concept mapping. This design achieves MRR ≈ 0.92 on the surface‑form reversal task without any data augmentation or non‑causal masking, effectively breaking the Reversal Curse.

Further, the authors augment the recognition module with a dedicated memory layer (key‑value slots per concept). The memory provides a fixed locus for each concept, preventing gradient spill‑over between concepts and eliminating entanglement. Experiments with memory‑enhanced JEPA models show depth‑invariant performance (MRR ≈ 0.94 even at 24 layers).

An additional observation is that mastering reversal enables parametric forward‑chaining: the model can internally store intermediate reasoning steps in its parameters and retrieve them later, achieving strong arithmetic and logical reasoning without external retrieval mechanisms.

In summary, the paper makes three major contributions: (1) it reframes the Reversal Curse as a concrete instance of the binding problem, (2) it empirically isolates two failure modes—representation inconsistency and entanglement—underlying the curse, and (3) it introduces a JEPA‑plus‑memory architecture that resolves the issue in a principled way. The work highlights the importance of systematic conceptual binding for robust generalization and opens a research agenda toward LLMs that can learn and manipulate abstract concepts with human‑like systematicity.