Low Rank Transformer for Multivariate Time Series Anomaly Detection and Localization

Multivariate time series (MTS) anomaly diagnosis, which encompasses both anomaly detection and localization, is critical for the safety and reliability of complex, large-scale real-world systems. The vast majority of existing anomaly diagnosis methods offer limited theoretical insights, especially for anomaly localization, which is a vital but largely unexplored area. The aim of this contribution is to study the learning process of a Transformer when applied to MTS by revealing connections to statistical time series methods. Based on these theoretical insights, we propose the Attention Low-Rank Transformer (ALoRa-T) model, which applies low-rank regularization to self-attention, and we introduce the Attention Low-Rank score, effectively capturing the temporal characteristics of anomalies. Finally, to enable anomaly localization, we propose the ALoRa-Loc method, a novel approach that associates anomalies to specific variables by quantifying interrelationships among time series. Extensive experiments and real data analysis, show that the proposed methodology significantly outperforms state-of-the-art methods in both detection and localization tasks.

💡 Research Summary

The paper tackles the dual problem of anomaly detection and localization in multivariate time series (MTS), a critical task for safety‑critical IoT‑enabled systems. The authors first provide a rigorous theoretical analysis of the Transformer encoder when applied to MTS. By unrolling the embedding, self‑attention, and residual connections, they show that the 1‑D convolutional embedding is mathematically equivalent to a learnable vector moving‑average (VMA) filter, and that the self‑attention mechanism implements a dynamic, data‑dependent set of weights that exactly matches the structure of a Space‑Time Autoregressive (ST‑AR) model. Without skip connections each latent series follows a single ST‑AR process; with skip connections the final representation becomes a linear combination of several ST‑AR processes. Feed‑forward layers do not alter this linear ST‑AR nature. Consequently, the reconstruction step is a linear projection of these ST‑AR components, revealing that a Transformer essentially learns a collection of linear time‑space models whose coefficients are recomputed at every forward pass.

Motivated by this insight, the authors introduce the Attention Low‑Rank Transformer (ALoRa‑T). They impose a low‑rank constraint on each self‑attention matrix via a nuclear‑norm (or truncated‑SVD) regularizer. The key observation is that normal MTS data tend to produce low‑rank attention patterns, while anomalous windows cause a sudden increase in rank. The rank (or the magnitude of the low‑rank penalty) is therefore used as an anomaly score, called the ALoRa‑Score. This score is computed per sliding window and directly reflects the temporal extent of an anomaly.

For localization, the paper defines two contribution matrices: E_{ij}, measuring how input series i influences latent feature j (derived from the Q‑K interaction), and C_{ij}, measuring how latent feature j contributes to the reconstructed output series i (derived from the V‑W_out interaction). The product E_{ij}·C_{ij} quantifies the total propagation of an anomaly from input to output. The proposed ALoRa‑Loc aggregates these products across all heads and layers, yielding a per‑variable anomaly contribution score. Variables with the highest contributions are identified as the sources of the anomaly, providing a transparent and computationally cheap localization mechanism that does not require gradient‑based attribution.

Extensive experiments are conducted on six public benchmark datasets (SMD, MSL, SMAP, etc.) and on real‑world industrial datasets (power grid, water distribution, healthcare monitoring). Evaluation metrics include the traditional point‑adjusted F1, segment‑level PR‑AUC, and a new variable‑level localization F1 (F1_loc). ALoRa‑T consistently outperforms state‑of‑the‑art Transformer‑based detectors (Anomaly‑Transformer, MEMTO, SARAD) by 8–12% in detection F1 and achieves superior PR‑AUC. ALoRa‑Loc surpasses reconstruction‑error‑based localization methods (OmniAnomaly, SARAD) and gradient‑based approaches (DAEMON) by more than 15% in variable‑level accuracy. The low‑rank regularization also reduces memory consumption by ~40% and speeds up inference by a factor of 2, making the approach suitable for real‑time monitoring.

In summary, the paper makes three major contributions: (1) a novel theoretical bridge linking Transformer encoders to classical ST‑AR models, providing interpretability; (2) the ALoRa‑T architecture with a low‑rank attention regularizer and the ALoRa‑Score for reliable anomaly detection; (3) the ALoRa‑Loc framework that leverages contribution matrices to pinpoint anomalous variables without expensive post‑hoc analysis. The work advances both the theoretical understanding and practical performance of deep learning models for multivariate time‑series anomaly diagnosis, and opens avenues for future research on integrating low‑rank constraints with more expressive feed‑forward modules and extending the methodology to multimodal time‑series data.