Error-Free Linear Attention is a Free Lunch: Exact Solution from Continuous-Time Dynamics

Linear-time attention and State Space Models (SSMs) promise to solve the quadratic cost bottleneck in long-context language models employing softmax attention. We introduce Error-Free Linear Attention (EFLA), a numerically stable, full parallelism and generalized formulation of the delta rule. Specifically, we formulate the online learning update as a continuous-time dynamical system and prove that its exact solution is not only attainable but also computable in linear time with full parallelism. By leveraging the rank-1 structure of the dynamics matrix, we directly derive the exact closed-form solution effectively. This attention mechanism is theoretically free from error accumulation, perfectly capturing the continuous dynamics while preserving the linear-time complexity. Through an extensive suite of experiments, we show that EFLA enables robust performance in noisy environments, achieving lower language modeling perplexity and superior downstream benchmark performance than DeltaNet without introducing additional parameters. Our work provides a new theoretical foundation for building high-fidelity, scalable linear-time attention models.

💡 Research Summary

The paper tackles the quadratic cost of soft‑max attention in large language models by focusing on linear‑time attention and state‑space models (SSMs). While prior works such as DeltaNet and Mamba‑2 have linked attention to continuous‑time dynamics, they still rely on a first‑order Euler discretization, which introduces truncation error and numerical instability, especially for long sequences or large decay rates.

The authors propose Error‑Free Linear Attention (EFLA), a principled method that eliminates discretization error entirely. They first reformulate the online learning update of linear attention as a first‑order ordinary differential equation (ODE):

\