Online Conformal Model Selection for Nonstationary Time Series

This paper introduces the MPS (Model Prediction Set), a novel framework for online model selection for nonstationary time series. Classical model selection methods, such as information criteria and cross-validation, rely heavily on the stationarity assumption and often fail in dynamic environments which undergo gradual or abrupt changes over time. Yet real-world data are rarely stationary, and model selection under nonstationarity remains a largely open problem. To tackle this challenge, we combine conformal inference with model confidence sets to develop a procedure that adaptively selects models best suited to the evolving dynamics at any given time. Concretely, the MPS updates in real time a confidence set of candidate models that covers the best model for the next time period with a specified long-run probability, while adapting to nonstationarity of unknown forms. Through simulations and real-world data analysis, we demonstrate that MPS reliably and efficiently identifies optimal models under nonstationarity, an essential capability lacking in offline methods. Moreover, MPS frequently produces high-quality sets with small cardinality, whose evolution offers deeper insights into changing dynamics. As a generic framework, MPS accommodates any data-generating process, data structure, model class, training method, and evaluation metric, making it broadly applicable across diverse problem settings.

💡 Research Summary

The paper tackles the problem of online model selection for non‑stationary time series, a setting where traditional offline techniques such as AIC, BIC, cross‑validation, or even offline Model Confidence Sets (MCS) often fail because they assume a fixed data‑generating process. The authors introduce the Model Prediction Set (MPS), a novel framework that continuously updates a confidence set of candidate models and guarantees—over the long run—that the set contains the best model for the next time point with a pre‑specified mis‑coverage rate (\bar\alpha).

MPS combines two existing ideas. First, it uses the offline MCS machinery to construct, at each time (t), a provisional set (C_t(1-\beta)) based on the loss matrix accumulated up to (t). Second, it adopts the adaptive calibration principle from recent Bellman Conformal Inference (BCI) to adjust the nominal mis‑coverage level (\alpha_t) online. By monitoring whether the true best model (M_{t+1}) was included in the previous set, the algorithm updates (\alpha_t) so that the empirical average of the indicator (\mathbf{1}{M_{t+1}\notin C_t}) stays below (\bar\alpha). Theoretical analysis shows that, under mild conditions (essentially a martingale‑type bound on the calibration error), the long‑run mis‑coverage bound (1) holds without requiring stationarity or a correctly specified model.

A key strength of MPS is its flexibility. The “best model” can be defined by any user‑chosen evaluation metric—information criteria, likelihood‑based scores, out‑of‑sample forecast errors, or even custom loss functions—allowing the framework to accommodate pure statistical models, black‑box machine‑learning algorithms, or policy rules. The candidate set (\mathcal M) can be arbitrarily large, though computational cost scales linearly with its cardinality because a loss must be computed for each model at every time step.

Empirical validation proceeds in two parts. In synthetic experiments the authors simulate abrupt regime shifts and gradual seasonal drift, demonstrating that MPS maintains the target mis‑coverage (e.g., (\bar\alpha=0.05)) while keeping the average set size small (often 2–3 models). In real‑world tests on the ETTh1 electricity‑demand series, a moving‑window evaluation (window size 100) shows that offline single‑model selectors suffer high mis‑coverage (>20 %) and that offline MCS either under‑covers or produces trivial sets containing all models. By contrast, MPS achieves mis‑coverage close to the nominal level and produces compact sets whose cardinality expands during periods of high uncertainty, providing interpretable signals about the underlying dynamics.

The paper also discusses limitations. Computing the loss matrix for many models can be costly; the adaptive calibration requires choosing a window or decay factor for updating (\alpha_t); and rapid structural breaks may cause a short lag before the set adapts. Future work is suggested on efficient sub‑sampling, multi‑stage adaptation (e.g., coupling variance estimation), extensions to multivariate/high‑dimensional series, and deployment in operational domains such as finance or smart grids.

In summary, MPS is the first online, distribution‑free method that delivers provable long‑run coverage for model selection under unknown, possibly severe non‑stationarity. Its blend of conformal inference and model confidence sets yields a practical tool that can be applied across a wide range of time‑series forecasting and decision‑making problems.