Attention as Binding: A Vector-Symbolic Perspective on Transformer Reasoning

A central question therefore arises: what representational operation is attention actually performing? Attention is commonly viewed as content-addressable lookup (Olsson et al. 2022), but such interpretations say little about how roles, fillers, and structured relations are internally encoded. Classical work in cognitive science and neurosymbolic AI emphasizes that robust reasoning requires mechanisms for binding and unbinding symbolic constituents (Smolensky 1990;d’Avila Garcez, Lamb, and Gabbay 2008;Garcez and Lamb 2023). This paper adopts that perspective.

We argue that attention can be understood as a soft binding/unbinding operator. Queries and keys define role-like subspaces; values supply fillers; and attention weights implement a differentiable unbinding step that retrieves fillers according to role similarity. Residual connections superpose many such bindings. When these operations behave coherently, transformers can approximate structured symbolic manipulation; when they do not, we obtain the logical brittleness now extensively documented.

The central thesis of this review is that transformer attention can be interpreted as a soft, approximate instance of this VSA algebra. Queries and keys define role-like vectors, values supply fillers, and attention weights implement a differentiable unbinding operation. Residual connections act as superposition, and multi-head structure enables layered compositions. Logical brittleness arises when these approximations fail-e.g., embeddings are insufficiently decorrelated or roles and fillers interfere-mirroring known VSA failure modes.

Rather than a conventional survey, this paper offers a conceptual synthesis connecting three strands of research:

1. Transformer/attention mechanisms and reasoning techniques such as chain-of-thought and tool-augmented models (Vaswani et al. 2017;Wei et al. 2022;Wang et al. 2023;Mialon et al. 2023). 2. Vector Symbolic Architectures as a principled algebra for compositional vector representations (Plate 1995;Kanerva 2009;Kleyko et al. 2022). 3. Neurosymbolic reasoning frameworks integrating statistical and symbolic computation (d’Avila Garcez, Lamb, and Gabbay 2008;Garcez and Lamb 2023;Creswell et al. 2023;Mialon et al. 2023).

Our contributions are:

• A unified interpretation of attention and residual streams as performing approximate VSA-style binding, unbinding, and superposition. • A taxonomy distinguishing “VSA-like” from “non-VSA-like” transformer mechanisms. • A conceptual framework linking VSA structure to chainof-thought behavior, tool use, and logical consistency. • A research agenda-including evaluation protocols and architectural proposals such as explicit binding heads and hyperdimensional memories-for building VSA-inspired reasoning architectures.

Grounding transformer reasoning in vector-symbolic computation enables a more principled account of when LLMs can support reliable symbolic manipulation, moving beyond descriptive benchmarking toward a structured algebraic understanding of their reasoning capabilities.

Transformers and scaled dot-product attention

The transformer architecture (Vaswani et al. 2017) processes a sequence x 1:n by embedding tokens, adding positional encodings, and applying layers of multi-head self-attention and feed-forward blocks, each wrapped in residual connections and layer normalization (Ba, Kiros, and Hinton 2016). For a single head with input X ∈ R n×d ,

and scaled dot-product attention is

(2)

Multi-head attention concatenates several such heads and projects them back into the residual stream, which accumulates information across layers and effectively superposes representations generated by different tokens and heads (Elhage et al. 2021).

Operationally, attention is a form of content-addressed read/write. The QK ⊤ term dynamically determines which tokens influence each other, producing an input-dependent computation graph rather than a fixed routing structure (Elhage et al. 2021). This perspective already suggests that attention enforces a consistent algebra for combining and propagating representations, motivating the binding-unbinding interpretation developed in later sections.

A complementary view treats attention as a differentiable key-value store (Graves, Wayne, and Danihelka 2014;Weston, Chopra, and Bordes 2015). Each row of K acts as an address and the corresponding row of V as content; a query q i retrieves a similarity-weighted combination of values. In decoder-only models, KV caching extends this mechanism across timesteps, turning the transformer into a recurrent memory system that appends new key-value pairs during generation (Brown et al. 2020;Liu et al. 2024).

Each attention head forms a distinct memory channel, projecting the residual stream into different key and value subspaces. Interpreted symbolically, keys can encode roles or variables (e.g., subject vs. object, premise vs. hypothesis), while values encode the corresponding fillers. The learned matrices W Q , W K , W V decide which components of the residual stream participate in addressing and which become memory contents. This memory-centric interpretation provides an initial bridge from attention to symbolic variable binding and memory management (Smolensky 1990;Graves et al. 2016;Elhage et al. 2021).

Pretraining and instruction-tuning produce surprisingly strong reasoning-like behaviors (Brown et al. 2020;Chowdhery et al. 2023;OpenAI et al. 2024). Chain-of-thought prompting (Wei et al. 2022) improves arithmetic, commonsense, and symbolic tasks (Cobbe et al. 2021;Kojima et al. 2023;Nye et al. 2022), and self-consistency (Wang et al. 2023) further boosts performance. Separately, LLMs augmented with external tools-code interpreters (Chen et al. 2021;Gao et al. 2023;Schick et al. 2023), theorem provers (Polu and Sutskever 2020;Wu et al. 2022), and logical/probabilistic solvers (Pan et al. 2023;Creswell et al. 2023;Mialon et al. 2023)-can solve more complex problems by delegating computation.

Yet persistent limitations remain. LLMs frequently exhibit logical inconsistency across related prompts (Razeghi et al. 2022;Jin et al. 2025), struggle with systematic generalization and variable substitution (Keysers et al. 2020;Hup-kes et al. 2020), rely on superficial cues rather than underlying structure (Min et al. 2022), and are sensitive to small perturbations in problem phrasing (Perez, Kiela, and Cho 2021). These issues indicate that transformers can approximate symbolic reasoning but do not inherently guarantee reliable symbolic computation.

Logical reasoning requires consistent manipulation of structured representations: variables must be bound and renamed without collision, predicates must compose systematically, and inference rules must preserve truth across transformations (Enderton 2001;Mitchell 1997;Garcez and Lamb 2023). Neural architectures aiming to support such reasoning must therefore implement disciplined operations analogous to binding, unbinding, and superposition.

Standard interpretations of attention-as token-weight distributions or dynamic feature selectors-do not specify how roles and fillers are represented or combined, making it difficult to articulate or constrain the invariances logic demands. This motivates an explicitly algebraic framework for understanding attention.

Vector Symbolic Architectures offer such a framework: a small set of operations (binding, superposition, permutation) with well-defined algebraic properties for symbolic computation in high-dimensional spaces (Plate 1995;Kanerva 2009;Kleyko et al. 2022). Recasting attention through this lens clarifies when its behavior approximates principled binding/unbinding and when it departs from it, illuminating both strengths and limitations of LLMs as logical reasoners.

Vector Symbolic Architectures: Algebra for Compositional Vectors

Vector Symbolic Architectures (VSAs), or hyperdimensional computing, represent discrete symbols by dense, high-dimensional vectors that are approximately orthogonal (Plate 1995;Kanerva 2009;Gayler 2004;Kleyko et al. 2022). Let H ⊂ R D denote this space. Symbolic structure is encoded using three algebraic operations:

• Binding (a⊗b) forms role-filler pairs using elementwise multiplication, circular convolution, or XOR (Plate 1995;Rachkovskij and Kussul 2001). • Superposition sums vectors to represent sets or multisets (Kanerva 2009). • Permutation applies fixed invertible transformations to encode order or structure (Plate 1995). These operations remain within H and support approximate unbinding: given a bound vector and one constituent, the other can be recovered via similarity search. This closed algebra enables a form of symbolic computation directly in vector space (Gayler 2004;Kleyko et al. 2022).

Complex structures arise from repeated binding, superposition, and permutation. A sequence (x 1 , . . . , x n ) may be en-coded as

retrieved by unbinding with π i (pos) (Plate 1995;Kanerva 2009). Sets correspond to pure superpositions, while keyvalue stores take the form j k j ⊗ v j . Trees and graphs follow similar principles: subtrees or edges are bound to role vectors such as LEFT, RIGHT, or edge labels and then superposed (Frady, Kleyko, and Sommer 2023;Kleyko et al. 2022). All representations remain in the same vector space, with decoding performed via similarity, giving VSAs a uniform representational substrate for structured data.

Since VSA operations support similarity-based retrieval and unbinding, they naturally implement associative memories, symbolic rule application, and multi-step inference (Kanerva 2009;Frady, Kleyko, and Sommer 2023;Kleyko et al. 2022). Predicates, for instance, can be represented as vectors p such that p ⊗ x encodes “P (x),” with logical combinations handled via superposition. These properties make VSAs attractive for neurosymbolic models of working memory, compositional semantics, and cognitive reasoning (Plate 1995;Gayler 2004;Kanerva 2009). Moreover, binding and superposition are differentiable or readily approximated, enabling neural architectures that manipulate VSA representations while preserving a symbolic interpretation.

VSAs sit among several frameworks for distributed compositional representations. Tensor Product Representations (TPRs) (Smolensky 1990) bind roles and fillers via higherorder tensors, while Holographic Reduced Representations (HRRs) (Plate 1995)

Scaled dot-product attention for a single head computes the attention as mentioned in equation ( 2), where Q, K, V ∈ R n×d k are derived from the residual stream (Vaswani et al. 2017). For position i,

The Q, K, V ∈ R n×d k are derived from the residual stream (Vaswani et al. 2017) when computing the scaled dot-product attention for a single head. Each query induces a similarity distribution over keys, producing a convex combination of values-a classic content-addressable memory operation (Graves, Wayne, and Danihelka 2014;Weston, Chopra, and Bordes 2015;Elhage et al. 2021).

In VSA terms, unbinding retrieves a filler by comparing a role cue with stored bindings (Plate 1995;Kanerva 2009;Kleyko et al. 2022). Dot-product attention is a differentiable analogue: queries act as roles, keys as stored roles, and values as fillers, with the softmax functioning as a smooth unbinding operator.

This suggests interpreting key and value projections as implementing a role-filler decomposition (Smolensky 1990;Elhage et al. 2021). The key projection W K extracts the role a token plays (e.g., subject/object, premise/conclusion), while W V encodes the corresponding filler. W Q determines which roles each position seeks to access. Multi-head attention provides multiple parallel binding channels, each with its own (W

Empirical work shows head specialization: some track syntax, others lexical or positional features (Clark et al. 2020;Elhage et al. 2021;Rogers, Kovaleva, and Rumshisky 2020). In VSA language, each head implements a distinct binding scheme in a shared highdimensional substrate.

Classical VSAs perform hard binding, e.g. z = r ⊗ f , with unbinding recovering f from z using r (Plate 1995). Transformers instead realize a soft version: keys serve as roles, values as fillers, and α ij measures the binding strength between a query role and stored roles. The resulting output is a weighted superposition of fillers.

Attention best approximates VSA binding when (i) role vectors (keys) are near-orthogonal to reduce interference (Kanerva 2009;Frady, Kleyko, and Sommer 2023), (ii) attention is relatively sparse, yielding crisp role-filler matches (Michel, Levy, and Neubig 2019;Voita et al. 2019), and (iii) normalization (like layer norm) ensures consistent geometric structure. Under these conditions, attention functions as an approximate VSA binding/unbinding operator.

Residual connections naturally implement VSA-style superposition. With layer update

the residual stream accumulates contributions from many heads and transformations. Each attention output corresponds to one or more bound pairs, and the residual stores their superposition over depth. Layer norm maintains useful geometry, while MLPs act as learned, nonlinear reencodings (Ba, Kiros, and Hinton 2016;Elhage et al. 2021). Together, these operations approximate a pipeline of soft binding, superposition, and rewriting within a fixeddimensional space.

Approximation gap: when attention fails to be VSA-like (Plate 1995). RoPE’s rotations can be viewed as differentiable permutation operations that compose naturally with binding and superposition.

Recurrent and memory-augmented models, including Transformer-XL (Dai et al. 2019), Compressive Transformers (Rae et al. 2020), and long-range KV caching, append historical key-value pairs over time. This creates a growing superposed store of bound pairs, whose quality depends on decisions about retention, compression, and decay, directly affecting how VSA-like the memory dynamics become.

Several architectures implement explicit binding and slotting operations. Memory Networks (Weston, Chopra, and Bordes 2015), Neural Turing Machines, and the Differentiable Neural Computer (Graves, Wayne, and Danihelka 2014;Graves et al. 2016) maintain addressable memory with differentiable read/write operations. Viewed through a VSA lens, memory cells hold bound role-filler vectors, reads perform unbinding, and writes superpose or overwrite bindings.

Slot Attention (Locatello et al. 2020) learns a small set of “slot” vectors that act as roles binding to object features via attention. Fast-weights models (Schmidhuber 1992; Ba et al. 2016) and multiplicative RNNs (Sutskever, Martens, and Hinton 2011) implement dynamic parameter matrices functioning as implicit high-dimensional binding stores.

Together, these architectures already instantiate VSA-like primitives: similarity-based addressing, additive superposition, and multiplicative binding. Their primary divergence from classical VSAs lies in how explicitly the algebra of roles and fillers is enforced. These architectures vary in how VSA-like their internal operations are. Some treat transformers as black-box encoders, offloading structure to external modules. Others align attention heads or memory representations with logical roles (predicate, argument, quantifier), resulting in internal states resembling superposed sets of symbolic facts. The closer the architecture encodes roles and fillers directly in vector operations, the more naturally it fits a VSA algebraic interpretation.

A growing line of work incorporates hyperdimensional computing directly into neural models. VSA-based representations have been used as classifiers or associative memories (Kanerva 2009), and more recent architectures embed binding and superposition into recurrent or convolutional networks (Frady, Kleyko, and Sommer 2023;Kleyko et al. 2022). Some approaches replace or augment embeddings with hyperdimensional encodings, enforcing role-filler separation and robust superposition by design.

These systems demonstrate that high-dimensional binding operations are compatible with gradient-based learning and scalable models. They validate the feasibility of incorporating VSA principles into modern deep architecturesevidence we build on in later sections when proposing VSAinspired transformer designs aimed at improving logical and symbolic reasoning capabilities.

Chain-of-thought (CoT) prompting elicits intermediate reasoning steps, effectively externalizing an execution trace in natural language (Wei et al. 2022;Kojima et al. 2023). Such traces correlate strongly with improved arithmetic, commonsense, and symbolic reasoning performance (Wei et al. 2022;Wang et al. 2023), and their benefits scale with model size. Conceptually, a CoT response reflects a sequence of symbolic state updates. Evidence suggests that internal representations partially track these decompositions even without supervision: different subtasks appear across different layers (Wei et al. 2022), and some hidden states encode information predictive of CoT success before reasoning tokens are produced (Afzal et al. 2025). However, mismatches between generated CoT and actual computation also arise, yielding fluent but unfaithful reasoning (Turpin et al. 2023). This motivates a representational framework capable of describing both faithful and spurious traces in a unified algebraic language.

From a VSA perspective, each reasoning step corresponds to adding or modifying bound role-filler pairs in the residual stream. Let s (t) denote the internal state after t steps. A VSA-like update can be written as

where r

k encode roles (e.g., subgoal, partial result) and f (t) k encode fillers. Later steps may update or permute roles to reflect changing structure.

Reasoning sequences thus map to nested bindings and permutations: multi-step algebraic derivations correspond to progressively binding terms, while permutations encode ordering of intermediate quantities. CoT text can therefore be viewed as an externalization of a trajectory through a VSA-structured internal space. This view suggests concrete probes: decoding approximate role/filler vectors from hidden states and checking whether adjacent layers follow VSA-predicted structured updates.

Program-based reasoning extends CoT by expressing steps as executable code. Program-of-Thoughts (PoT) prompting delegates computation to external executors (Chen et al. 2023), and program-based CoT uses code-plus-tests as verifiable reasoning traces (Yang et al. 2025;Gao et al. 2023). These traces correspond to explicit program states (variables, memory, control flow).

VSAs naturally encode such states. A program environment can be written as

with additional bindings and permutations encoding control flow, call stacks, or proof-tree positions. Attention performs unbinding (reading a subgoal or variable), while MLP and subsequent layers rewrite fillers and rebind them. Under this view, both program synthesis and execution correspond to sequences of VSA operations guided by transformer parameters, regardless of whether explicit code is shown to the model.

If

Inductive biases from VSA algebra

The VSA perspective highlights representational properties essential for logical reasoning: (i) role-filler separation, keeping variables, argument positions, and contexts distinct from content; (ii) robust superposition, enabling multiple bindings to coexist with controlled interference; and (iii) near-orthogonality among role vectors to support reliable unbinding (Plate 1995;Kanerva 2009;Kleyko et al. 2022;Frady, Kleyko, and Sommer 2023). These can be encouraged through architectural and training biases. Certain heads or subspaces can be designated as role or value channels, with constraints such as coherence or orthogonality. Optimization-time regularizers can further maintain orthogonal key/query subspaces and well-conditioned value projections, nudging transformers toward VSA-like binding and unbinding by design rather than emergence.

A concrete proposal is to introduce specialized binding/unbinding heads whose parameters approximate VSA operators. Some heads may perform multiplicative binding via elementwise interactions between learned role vectors and filler vectors from the residual stream, replacing standard linear value projections. Positional structure can be handled by tying projections to rotary or permutation-like matrices, mirroring VSA positional permutations (Plate 1995;Su et al. 2021). Unbinding heads may approximate inverse operators (such as correlation for convolution-based binding), with queries constructed as role cues. These modules coexist with standard attention heads and feed-forward blocks, giving the model both unconstrained expressivity and disciplined VSA-style computation. Training dynamics then determine the degree of reliance on these specialized heads while structural constraints keep them close to interpretable algebraic forms.

Another direction is to incorporate hyperdimensional memory layers implementing explicit VSA-style associative stores. Such a module maintains a memory vector m ∈ R D that accumulates formulas, partial proofs, or tool states via superposed bindings:

Training objectives can explicitly promote VSA-like behavior. Orthogonality constraints on role vectors and key/query projections, via spectral penalties, Bjorck normalization, or coherence regularization, reduce interference and preserve clean role-filler structure. Auxiliary reconstruction tasks can train the model to unbind fillers from synthetic bound or superposed vectors, or to recover constituents after injected superposition noise. Logic-oriented auxiliary tasks further encourage systematic generalization, such as permuted-variable reasoning, symbolic substitution, or synthetic theorem-proving tasks where identical proof patterns must generalize across renamings (Keysers et al. 2020;Hupkes et al. 2020). Collectively, these objectives align the geometry of attention and residual streams with VSA algebra, increasing the likelihood that LLMs learn stable, compositional reasoning mechanisms.

Evaluation: measuring VSA-likeness and logical compositionality

Defining “VSA-likeness” metrics

The VSA view suggests that some attention heads approximate principled binding/unbinding while others behave as generic feature mixers. To quantify this, we define empirical VSA-likeness metrics. First, role-filler recoverability tests whether putative role and filler subspaces are separable. Using synthetic role vectors {r k } and fillers {f k } injected via prompts or residual edits, we measure how well a probe recovers f k when cued with r k , analogous to VSA unbinding (Plate 1995;Kanerva 2009;Kleyko et al. 2022). High recoverability indicates a clean role-filler factorization.

Second, interference under superposition measures how well unbinding works when multiple bindings are combined, e.g. z = r 1 ⊗ f 1 ⊕ r 2 ⊗ f 2 . Varying the number and similarity of bindings yields capacity and interference curves, paralleling VSA associative memory analyses (Kanerva 2009;Frady, Kleyko, and Sommer 2023).

Third, alignment with VSA operators assesses whether learned transformations approximate known binding/permutation operations. By fitting simple VSA models (e.g., convolution, fixed permutations) to the action of a head, we can quantify how closely its behavior adheres to a VSA algebra.

External benchmarks should stress abilities expected from VSA-like representations. Tasks testing variable binding and systematic generalization, such as SCAN (Lake and Baroni 2018), compositional benchmarks (Keysers et al. 2020;Hupkes et al. 2020), and algebraic manipulation under variable renaming, evaluate whether models manipulate abstract role-filler structure rather than memorizing patterns.

A second class focuses on structured symbolic manipulation: algebraic simplification, equation solving, and rewriting tasks (Saxton et al. 2019;Polu and Sutskever 2020), where variation in depth and breadth tests compositional generalization. Classical logic puzzles and paraphrased inference problems assess logical invariance across syntactically diverse but equivalent formulations (Razeghi et al. 2022).

To distinguish genuine compositionality from pattern matching, benchmarks should hold out combinations of roles/fillers during training, test permutation of symbol identities, and enforce generalization to unseen rule instantiations. VSA-like models are predicted to perform more robustly under such shifts.

These metrics motivate a probing agenda for pretrained LLMs. Representational similarity analysis (RSA) (Kriegeskorte, Mur, and Bandettini 2008) can compare hidden-state geometry to synthetic VSA encodings of symbolic structures, revealing whether layers approximate VSA embeddings. Linear and nonlinear probes (Alain and Bengio 2017;Belinkov et al. 2017;Hewitt and Manning 2019) can decode roles, fillers, and bindings from hidden states, estimating recoverability.

Interventional studies edit embeddings or residual streams to inject synthetic bindings (e.g., swapping variable roles) and check whether model predictions reflect consistent unbinding and rebinding. Combined with causal attention analyses, such interventions identify heads or layers exhibiting latent VSA-like structure.

A VSA-centric view helps systematize reasoning failures as breakdowns of binding/unbinding. Variable confusion (duplicated or swapped roles) indicates weak role-filler separation or overwritten bindings. Role swaps between premises and conclusions point to deficiencies in positional/permutation encoding. More subtle inconsistency across related queries can reflect interference between superposed bindings representing different contexts. Mapping such failures to specific aspects of VSA algebra-separation, capacity, permutation stability-yields diagnostic patterns that guide architectural and training interventions. Thus, VSA-likeness metrics and benchmarks not only evaluate models but also illuminate why logical and compositional reasoning succeeds or fails in attention-based LLMs.

Our VSA-based interpretation of attention raises foundational questions about the algebraic structure of transformer computation. One line of inquiry concerns conditions for equivalence: under what assumptions on initialization, training dynamics, and embedding geometry do attention layers implement an algebra closely matching a VSA binding/unbinding system? For instance, can approximate orthogonality of key/query spaces and sparsity of attention weights guarantee a well-formed binding operator with a similaritybased inverse?

A second line examines transformer expressivity in terms of VSA algebraic capacity. While recent work connects transformers to classes of formal languages (Strobl et al. 2024), these results rarely appeal to explicit role-filler algebras. A VSA-centric theory would ask what classes of logical transformations or proof procedures can be simulated with finite role vectors, binding operators, and superposition capacity, and how these capabilities scale with model depth, width, and number of heads. Understanding this capacity may clarify generalization patterns in reasoning tasks.

Architecturally, a key question is the granularity at which VSA structure should be imposed. Should binding and superposition be enforced at the token level (via specialized embeddings or heads), at the layer level (dedicated binding layers), or via a separate hyperdimensional memory module accessed through cross-attention (Graves et al. 2016;Frady, Kleyko, and Sommer 2023)? Each option trades off interpretability, capacity, and computational overhead.

Another question concerns balancing VSA-like rigidity with neural flexibility. Strict VSA constraints (fixed binding operators, hard orthogonality) may improve symbolic behavior but limit exploitation of statistical patterns in natural data. Fully unconstrained architectures, however, risk losing necessary binding discipline. Hybrid designs-with some heads or modules explicitly VSA-like and others freeform-may reveal how training dynamics distribute symbolic versus statistical responsibilities.

A further agenda concerns integration of VSA-style internal states with external symbolic systems. If transformers maintain hyperdimensional representations of formulas, partial proofs, or knowledge-graph fragments, how should these interface with logic and theorem provers, SMT solvers, or probabilistic programs (Serafini and d’Avila Garcez 2016;Manhaeve et al. 2018;Creswell et al. 2023;Mialon et al. 2023)? One possibility is to design encoders mapping symbolic structures into VSA representations compatible with the model’s internal algebra, and decoders transforming hyperdimensional states back into human-readable proofs or explanations. Similar considerations apply to ontologies and knowledge graphs (Nickel et al. 2016). VSA encodings may bridge graph-based reasoning and text-based LLM reasoning, while multi-agent or tool-augmented systems could use VSA memory as a shared workspace for exchanging structured information. Determining how to design, train, and coordinate such shared hyperdimensional stores remains an open challenge.

Finally, the VSA interpretation invites interdisciplinary work on the cognitive plausibility of binding in LLMs. VSAs have roots in theories of human working memory and variable binding (Smolensky 1990;Plate 1995;Kanerva 2009). If attention approximates a vectorial binding/unbinding algebra, how closely does this mirror hypothesized neural mechanisms for symbolic reasoning and language (Marcus 2001)? Are there biological correlates of role-filler separation and structured superposition, and can LLM error patterns (e.g., variable confusion, scope errors) inform cognitive models?

Concrete directions include comparing human-like reasoning errors to VSA failure modes, testing whether increasing VSA-likeness produces more human-like reasoning trajectories, and designing neurocognitive experiments probing for VSA-like representations. Progress here could enrich both AI and cognitive science, and inspire architectures grounded in principles of human symbolic computation.

Transformer attention can be interpreted as a soft vectorsymbolic algebra in which queries and keys define role subspaces, values encode fillers, and attention weights implement a differentiable unbinding operator that retrieves and recombines fillers according to role similarity. Residual connections act as superposition, enabling layered compositions of bound structures (Vaswani et al. 2017;Plate 1995;Kanerva 2009;Kleyko et al. 2022). This viewpoint unifies many empirical findings on LLM behavior, providing an algebraic explanation for how transformers sometimes succeed at symbolic manipulation and why they often fail when interference, entanglement, or dense attention patterns disrupt clean role-filler structure.

Interpreting attention through the VSA lens yields concrete implications for logical reasoning: transformer performance improves when role-filler separation is stable, superposition is controlled, and binding/unbinding behavior approximates principled algebraic operations; it degrades when these properties break down, producing inconsistency, brittleness, and poor systematic generalization (Keysers et al. 2020;Creswell et al. 2023). This perspective motivates new directions for theory, architecture, and evaluation, ranging from explicit binding heads and hyperdimensional memory layers to VSA-likeness metrics and logic-sensitive benchmarks. By framing reasoning in LLMs as approximate computation in a high-dimensional symbolic algebra, this review aims to provide a foundation for designing more interpretable, reliable, and compositionally robust models.

This work does not raise ethical or societal concerns. It advances theoretical understanding of transformer-based language models by interpreting attention and residual streams through the algebraic framework of Vector Symbolic Architectures. These insights may help guide the development of more reliable and interpretable reasoning systems, potentially improving the robustness of future machine learning models without altering their societal or ethical risk profile.