Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecasting

In the era of increasingly complex AI models for time series forecasting, progress is often measured by marginal improvements on benchmark leaderboards. However, this approach suffers from a fundamental flaw: standard evaluation metrics conflate a model’s performance with the data’s intrinsic unpredictability. To address this pressing challenge, we introduce a novel, predictability-aligned diagnostic framework grounded in spectral coherence. Our framework makes two primary contributions: the Spectral Coherence Predictability (SCP), a computationally efficient ($O(N\log N)$) and task-aligned score that quantifies the inherent difficulty of a given forecasting instance, and the Linear Utilization Ratio (LUR), a frequency-resolved diagnostic tool that precisely measures how effectively a model exploits the linearly predictable information within the data. We validate our framework’s effectiveness and leverage it to reveal two core insights. First, we provide the first systematic evidence of “predictability drift”, demonstrating that a task’s forecasting difficulty varies sharply over time. Second, our evaluation reveals a key architectural trade-off: complex models are superior for low-predictability data, whereas linear models are highly effective on more predictable tasks. We advocate for a paradigm shift, moving beyond simplistic aggregate scores toward a more insightful, predictability-aware evaluation that fosters fairer model comparisons and a deeper understanding of model behavior.

💡 Research Summary

The paper tackles a fundamental flaw in contemporary time‑series forecasting evaluation: standard aggregate metrics such as mean‑squared error (MSE) or mean absolute error (MAE) conflate a model’s predictive power with the intrinsic unpredictability of the data. As a result, a sophisticated deep‑learning model can appear inferior to a simple linear baseline simply because the test segment is highly regular and therefore easy to predict. To disentangle model capability from data difficulty, the authors introduce a predictability‑aligned diagnostic framework built on spectral coherence. The framework consists of two complementary tools.

1. Spectral Coherence Predictability (SCP). Using Welch’s method, the algorithm computes the power spectral densities (PSDs) of the past (history) x and the future target y, together with their cross‑power spectral density (CPSD). For each frequency f, the squared coherence
   \