Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View

Scalable sequence models, such as Transformer variants and structured state-space models, often trade expressivity power for sequence-level parallelism, which enables efficient training. Here we examine the bounds on error and how error scales when models operate outside of their expressivity regimes using a Lie-algebraic control perspective. Our theory formulates a correspondence between the depth of a sequence model and the tower of Lie algebra extensions. Echoing recent theoretical studies, we characterize the Lie-algebraic class of constant-depth sequence models and their corresponding expressivity bounds. Furthermore, we analytically derive an approximation error bound and show that error diminishes exponentially as the depth increases, consistent with the strong empirical performance of these models. We validate our theoretical predictions using experiments on symbolic word and continuous-valued state-tracking problems.

💡 Research Summary

The paper investigates why depth is crucial for parallelizable sequence models such as Transformer variants and structured state‑space models (SSMs). While these architectures achieve high training efficiency by being order‑symmetric (i.e., invariant to permutations of their inputs), this symmetry limits their ability to represent order‑sensitive dynamics that arise in many real‑world tasks (natural language, symbolic reasoning, physical dynamics). The authors address the quantitative question: how large is the approximation error when a model operates outside its expressivity regime, and how does depth affect this error?

The authors first recast sequence models as controlled dynamical systems on Euclidean space. The state‑transition matrix Φ(t,0) of an SSM is expressed as a time‑ordered exponential, which is precisely a solution of a linear controlled Lie equation. Consequently, the flow lives on a Lie group G whose tangent space at the identity is a Lie algebra g generated by the set of instantaneous generators A(x(t)). The algebraic structure of g (abelian, nilpotent, solvable) directly determines the model’s expressive power: abelian algebras correspond to order‑symmetric models that cannot capture non‑commuting operations.

To quantify the error introduced by order‑symmetry, the authors employ the Magnus expansion of Φ. The second Magnus term Ω₂, called the “commutator mass,” integrates the commutator