Universal Redundancies in Time Series Foundation Models

Time Series Foundation Models (TSFMs) leverage extensive pretraining to accurately predict unseen time series during inference, without the need for task-specific fine-tuning. Through large-scale evaluations on standard benchmarks, we find that leading transformer-based TSFMs exhibit redundant components in their intermediate layers. We introduce a set of tools for mechanistic interpretability of TSFMs, including ablations of specific components and direct logit attribution on the residual stream. Our findings are consistent across several leading TSFMs with diverse architectures, and across a diverse set of real-world and synthetic time-series datasets. We discover that all models in our study are robust to ablations of entire layers. Furthermore, we develop a theoretical framework framing transformers as kernel regressors, motivating a purely intrinsic strategy for ablating heads based on the stable rank of the per-head projection matrices. Using this approach, we uncover the specific heads responsible for degenerate phenomena widely observed in TSFMs, such as parroting of motifs from the context and seasonality bias. Our study sheds light on the universal properties of this emerging class of architectures for continuous-time sequence modeling.

💡 Research Summary

This paper investigates the internal structure of modern transformer‑based Time‑Series Foundation Models (TSFMs) and uncovers a striking pattern of redundancy that is shared across a diverse set of architectures. By focusing on five state‑of‑the‑art models—Chronos, Chronos‑Bolt, TimesFM 2.5, Toto, and Moirai—the authors conduct large‑scale zero‑shot forecasting experiments on the GIFT‑Eval benchmark and on synthetic chaotic ODE datasets. Their central methodological contribution is a toolbox for mechanistic interpretability that combines (1) Direct Logit Attribution (DLA) on the residual stream, (2) layer‑wise and head‑wise ablation studies, and (3) a theoretically grounded head‑selection metric based on the stable rank of the query‑key projection matrices.

The DLA analysis reveals that early layers often generate “spurious lines of thought,” i.e., implausible intermediate predictions that are later suppressed. Middle layers, however, produce highly similar updates across many tasks, increasing the entropy of the token distribution without substantially altering the final forecast. The final layer consolidates the representation and makes the decisive prediction. Quantitatively, the authors measure entropy, entropic rank, and Spearman distance to show that middle layers are the most ablateable, while the first and last layers are critical.

To explain why certain heads matter more than others, the paper frames self‑attention as a Nadaraya‑Watson kernel regression estimator. By examining the singular value decomposition of the combined query‑key matrix M for each head, they derive a notion of “sharpness”: heads with a few large singular values act as narrow‑bandwidth Gaussian kernels (sharp heads), whereas heads with a flat singular spectrum behave diffusely and contribute little to the output. This spectral view motivates the use of stable rank as an intrinsic head‑selection criterion. Ablating heads with low stable rank (or high entropy) removes up to 28 % of all heads while incurring less than a 6 % increase in Mean Absolute Scaled Error (MASE). Crucially, removing a single sharp head can eliminate the well‑known “context parroting” failure mode, where the model repeats exact subsequences from its context, and can also reduce the pervasive seasonality bias observed in many TSFMs.

The empirical results are consistent across all five models. Chronos‑Bolt (an encoder‑decoder) shows remarkable resilience to MLP ablations, whereas decoder‑only models such as Toto are far more sensitive, suggesting that architectural choices dictate how information is allocated between attention and feed‑forward pathways. Layer‑wise ablations demonstrate that the middle layers can be pruned without substantial loss, opening the door to model compression and faster inference. Head‑wise ablations guided by stable rank preserve performance while exposing the specific components responsible for degenerate behaviors.

In addition to the interpretability insights, the authors discuss broader implications for model compression, efficient fine‑tuning, and the design of future TSFMs that deliberately avoid redundant components. They outline three avenues for future work: (1) testing the identified redundancy patterns on a wider variety of domains, (2) evaluating how compressed, redundancy‑reduced models behave under downstream fine‑tuning, and (3) extending the kernel‑regression framework to more complex attention variants (e.g., QK‑norm, rotary embeddings). Overall, the paper provides a rigorous, theory‑backed, and experimentally validated picture of universal redundancies in time‑series foundation models, offering practical tools for both researchers and practitioners aiming to build smaller, more interpretable, and more reliable forecasting systems.