Deconstructing Positional Information: From Attention Logits to Training Biases

Positional encodings enable Transformers to incorporate sequential information, yet their theoretical understanding remains limited to two properties: distance attenuation and translation invariance. Because natural language lacks purely positional data, the interplay between positional and semantic information is still underexplored. We address this gap by deconstructing the attention-logit computation and providing a structured analysis of positional encodings, categorizing them into additive and multiplicative forms. The differing properties of these forms lead to distinct mechanisms for capturing positional information. To probe this difference, we design a synthetic task that explicitly requires strong integration of positional and semantic cues. As predicted, multiplicative encodings achieve a clear performance advantage on this task. Moreover, our evaluation reveals a hidden training bias: an information aggregation effect in shallow layers that we term the single-head deposit pattern. Through ablation studies and theoretical analysis, we proved that this phenomenon is inherent in multiplicative encodings. These findings deepen the understanding of positional encodings and call for further study of their training dynamics.

💡 Research Summary

This paper provides a systematic dissection of positional encodings (PE) in Transformers, distinguishing them into additive and multiplicative families and analyzing their impact on the attention‑logit computation through a Toeplitz‑matrix framework. The authors begin by decomposing each token representation into a content component c_i and a position component p_i, allowing the attention score matrix to be expressed as four inner‑product terms: content‑content, content‑position, position‑content, and position‑position. Additive PEs (absolute embeddings, T5 relative bias, ALiBi, etc.) contribute a Toeplitz‑structured bias either via an explicit bias matrix B or through the position‑position Gram matrix G_{q_p,k_p}. This yields a Toeplitz component that is independent of the content, while cross‑terms remain the only channel for content‑position interaction. In contrast, multiplicative PEs such as Rotary Positional Encoding (RoPE) apply a Toeplitz kernel G_e to all inner‑product terms via a Hadamard (element‑wise) product. Consequently, RoPE modulates every content interaction with a shared, distance‑dependent factor, creating a direct coupling between content and relative position that additive methods cannot achieve.

The theoretical analysis shows that additive methods preserve translation invariance but provide limited expressive power for tasks where the relevance of a token’s content depends on its relative distance. RoPE’s multiplicative coupling enables faster learning of such content‑position dependencies, as demonstrated by a derived upper bound on distance decay using Toeplitz spectral theory. However, this powerful coupling also introduces a strong inductive bias: positional reasoning tends to concentrate in a single attention head in the shallow layers, a phenomenon the authors name the “single‑head deposit pattern.”

To validate these hypotheses, the authors design two synthetic benchmarks. The first requires the model to reason about the relative positions of specially marked “anchor” tokens (a position‑sensitive task). The second depends only on token counts and is agnostic to order (a position‑agnostic task). Using a six‑layer Transformer decoder, they evaluate six PE configurations (absolute, T5, ALiBi, RoPE, and hybrids). Results show that RoPE dramatically outperforms additive encodings on the position‑sensitive task while lagging on the position‑agnostic one. Moreover, analysis of attention maps reveals that in shallow layers almost all positional information is processed by a single head when RoPE is used, confirming the deposit pattern. Ablation studies—such as applying RoPE to only a subset of heads or injecting absolute positional embeddings—demonstrate that the pattern disappears, proving it is intrinsic to the multiplicative structure rather than a training artifact.

Further theoretical work connects the Toeplitz spectrum to optimization dynamics, explaining why RoPE can achieve rapid early‑stage convergence yet sometimes exhibits larger generalization gaps in deeper layers. The paper concludes with three main contributions: (1) a unified Toeplitz‑based analytical framework that clearly separates additive and multiplicative PEs; (2) empirical identification and mechanistic explanation of RoPE’s “single‑head deposit pattern,” linking it to the observed performance paradox; and (3) a causal analysis establishing that this pattern is an inherent property of multiplicative encodings. The findings suggest that future PE design should balance the expressive benefits of content‑position coupling against the risk of overly specialized attention heads, and that training dynamics deserve as much attention as static architectural choices.