Parity, Sensitivity, and Transformers

The transformer architecture is almost a decade old. Despite that, we still have a limited understanding of what this architecture can or cannot compute. For instance, can a 1-layer transformer solve PARITY – or more generally – which kinds of transformers can do it? Known constructions for PARITY have at least 2 layers and employ impractical features: either a length-dependent positional encoding, or hardmax, or layernorm without the regularization parameter, or they are not implementable with causal masking. We give a new construction of a transformer for PARITY with softmax, length-independent and polynomially bounded positional encoding, no layernorm, working both with and without causal masking. We also give the first lower bound for transformers solving PARITY – by showing that it cannot be done with only one layer and one head.

💡 Research Summary

The paper investigates the expressive power of transformer models with respect to the parity function, a classic Boolean task that flips its output whenever any single input bit is flipped. The authors make two principal contributions. First, they prove a lower bound for one‑layer, one‑head transformers by analysing the average sensitivity of the functions they can compute. Average sensitivity is defined as the expected number of input bits whose flip changes the output, taken over all inputs of length n. For parity, this quantity grows linearly (Θ(n)). By exploiting quantifier elimination in the theory of real numbers with addition and order, the authors show that any function realized by a single‑layer, single‑head transformer can be expressed as a Boolean combination of a polynomial number of linear inequalities. Consequently, the number of input bits that can affect the output on average is bounded by O(√n). This establishes that a 1‑layer, 1‑head transformer cannot compute parity, providing the first quantitative lower bound of this type for soft‑max attention models.

The second contribution is a constructive upper bound: a concrete 4‑layer transformer architecture that computes parity exactly, using only standard components. The design employs soft‑max attention (no hard‑max), a length‑independent positional encoding that grows polynomially (e.g., linear in the position index), and does not rely on layer normalization or any length‑dependent tricks. The architecture works both in the full‑attention setting and under causal masking, because the final token always aggregates information from the entire sequence. The first two layers split the input bits into two groups and compute their sums; the third layer forms the difference of these sums, effectively extracting the parity information; the fourth layer maps this scalar to a binary decision via a final linear projection and softmax. All layers use a constant number of parameters per head, so the total parameter count remains polynomial in the input dimension.

The paper situates these results within a broad literature on transformer expressivity. Prior works showed that hard‑attention variants (UHA‑T, AHA‑T) are limited to AC⁰ and cannot compute parity, while constructions that do compute parity required at least two layers together with length‑dependent positional encodings, hard‑max, or layer‑norm with zero epsilon—features that are not used in standard training pipelines. By demonstrating that soft‑max attention alone, combined with a simple length‑independent encoding, suffices, the authors bridge the gap between theoretical hardness results and practical transformer designs. Moreover, the average‑sensitivity technique introduces a new tool for proving lower bounds on shallow transformers, potentially applicable to other highly sensitive Boolean functions such as majority or modular counting.

In summary, the work advances our understanding of how depth and head multiplicity affect a transformer’s ability to perform global Boolean computations. It shows that a single layer with a single head is provably insufficient for parity, while a modest four‑layer, multi‑head construction can solve the task using only standard, trainable components. This contributes both a theoretical framework for analyzing transformer limitations and a practical blueprint for building transformers that can handle globally coordinated tasks without resorting to non‑standard architectural tricks.