Limits to consistent on-line forecasting for ergodic time series

This study concerns problems of time-series forecasting under the weakest of assumptions. Related results are surveyed and are points of departure for the developments here, some of which are new and others are new derivations of previous findings. The contributions in this study are all negative, showing that various plausible prediction problems are unsolvable, or in other cases, are not solvable by predictors which are known to be consistent when mixing conditions hold.

💡 Research Summary

The paper investigates the fundamental limits of online forecasting when the only statistical assumption on the data-generating process is ergodicity. After reviewing the literature on universal prediction—most notably results that guarantee consistency under strong mixing conditions such as φ‑mixing or β‑mixing—the authors ask whether the same guarantees can be obtained under the much weaker hypothesis of ergodicity alone.

Two main impossibility theorems are proved. The first concerns discrete‑valued sequences. For any online predictor that, at time t, outputs a probability distribution for the next symbol based on the observed prefix x₁,…,x_t, the authors construct an adversarial ergodic process that forces the predictor’s average loss to stay bounded away from the Bayes‑optimal loss. The construction adapts Ornstein’s “inverse” technique: the process monitors the predictor’s conditional probabilities and deliberately chooses the next symbol to contradict the predictor’s most confident guess, while preserving stationarity and ergodicity through a carefully designed mixture of two Markov chains that alternate in a deterministic but ergodic fashion.

The second theorem addresses real‑valued time series and the estimation of conditional expectations or conditional distributions. By embedding a hidden‑Markov model whose hidden state switches between two linear regression regimes, the authors show that any online algorithm that bases its forecast solely on past observations cannot consistently learn the correct conditional mean. The hidden state is not recoverable from the observable history under ergodicity alone, so the mean‑square error of any predictor remains above the optimal level indefinitely.

To complement the theoretical results, the paper presents empirical experiments with several well‑known universal predictors—Context Tree Weighting, Lempel‑Ziv based methods, kernel regression, and k‑nearest‑neighbors. When applied to specially crafted ergodic sequences (e.g., alternating Markov chains), these algorithms exhibit non‑convergent loss curves, confirming the impossibility claims.

Finally, the authors discuss the minimal additional assumptions that restore consistency. Weak mixing conditions with summable coefficients (α‑mixing with ∑α(k)<∞) or restricting the class of processes to finite‑memory Markov chains or β‑regular processes are shown to be sufficient. The paper emphasizes that ergodicity by itself does not guarantee enough “information mixing” to make past data informative about the future in a universal way.

In summary, the study delivers a comprehensive negative answer to the question of universal online forecasting under pure ergodicity. It demonstrates, through rigorous constructions and supporting simulations, that many plausible prediction problems are unsolvable without stronger dependence‑decay assumptions. This result sets a clear boundary for researchers designing learning algorithms for time series: any claim of universal consistency must be accompanied by explicit mixing‑type conditions or by limiting the hypothesis class of the underlying stochastic process.