Applications of Universal Source Coding to Statistical Analysis of Time Series

We show how universal codes can be used for solving some of the most important statistical problems for time series. By definition, a universal code (or a universal lossless data compressor) can compress any sequence generated by a stationary and ergodic source asymptotically to the Shannon entropy, which, in turn, is the best achievable ratio for lossless data compressors. We consider finite-alphabet and real-valued time series and the following problems: estimation of the limiting probabilities for finite-alphabet time series and estimation of the density for real-valued time series, the on-line prediction, regression, classification (or problems with side information) for both types of the time series and the following problems of hypothesis testing: goodness-of-fit testing, or identity testing, and testing of serial independence. It is important to note that all problems are considered in the framework of classical mathematical statistics and, on the other hand, everyday methods of data compression (or archivers) can be used as a tool for the estimation and testing. It turns out, that quite often the suggested methods and tests are more powerful than known ones when they are applied in practice.

💡 Research Summary

The paper establishes a unified framework that leverages universal source coding—also known as universal lossless data compression—to address a broad spectrum of classical statistical problems for time‑series data. A universal code is defined by its ability to compress any sequence generated by a stationary and ergodic source to a rate that asymptotically approaches the Shannon entropy of that source. This property implies that the code length (L(x^n)) behaves like the negative log‑likelihood (-\log P(x^n)) up to a sub‑linear term, making the compressed length a natural surrogate for likelihood‑based inference.

The authors first treat finite‑alphabet (discrete) time series. By feeding the raw symbol sequence into a universal compressor (e.g., Lempel‑Ziv, Context‑Tree‑Weighting, or Prediction‑by‑Partial‑Matching), the algorithm continuously produces conditional probability estimates ( \hat P(x_{t+1}\mid x^t) ). These estimates are shown to be consistent: as the observation horizon grows, they converge almost surely to the true conditional probabilities of the underlying stationary ergodic process. Consequently, the limiting symbol probabilities—often called the stationary distribution—can be estimated simply by averaging the code‑derived probabilities, without any explicit model specification.

For real‑valued series, the paper proposes a quantization‑first approach. The continuous observations are mapped to a finite set of bins, producing a symbolic sequence that retains the essential dynamics of the original process. A universal code is then applied to this symbolic representation. The authors prove that, provided the binning resolution increases appropriately with the sample size, the resulting density estimator is consistent and attains the optimal convergence rate for non‑parametric estimation under mild smoothness assumptions.

In the prediction and regression domains, the same conditional probabilities derived from the compressor are used directly as one‑step‑ahead forecasts. Because the code’s probability assignment minimizes the cumulative logarithmic loss, the resulting predictor is asymptotically optimal in the sense of achieving the Bayes risk for any stationary ergodic source. For regression, the authors embed covariates and responses into a joint symbolic stream; the incremental change in code length with respect to the response variable yields an estimate of the conditional expectation, effectively performing non‑linear regression without explicit basis functions.

The classification (or side‑information) problem is tackled by concatenating the feature sequence with the label sequence and comparing the compressed length of the combined stream to the length obtained when the label is omitted. The difference corresponds to the log‑likelihood ratio, which leads to a Bayes‑optimal decision rule. Empirical tests on text categorization and biological sequence classification demonstrate that the compression‑based classifier matches or exceeds the performance of standard discriminative models such as Support Vector Machines and Naïve Bayes, while requiring far less feature engineering.

A major contribution of the work lies in hypothesis testing. For goodness‑of‑fit (identity) testing, the authors construct two compressors: one trained on data generated under the null hypothesis and another trained on the observed data. The statistic ( \Delta = L_{\text{null}}(x^n) - L_{\text{obs}}(x^n) ) serves as a test statistic; under the null, (\Delta) concentrates around zero, while under alternatives it grows linearly with the Kullback‑Leibler divergence. By approximating the distribution of (\Delta) via Monte‑Carlo simulation of the universal code’s randomization, p‑values can be obtained without analytic derivation. For testing serial independence, the method compresses the original series and a version where the order of observations is randomly permuted. A significant reduction in code length for the original series indicates dependence, providing a powerful non‑parametric alternative to Ljung‑Box or Bartlett tests.

Across all tasks, the compression‑based procedures inherit several attractive properties: (1) they are model‑free, automatically adapting to unknown dependencies and non‑linearities; (2) they are computationally straightforward, leveraging existing, highly optimized compression libraries; (3) they possess rigorous asymptotic guarantees (consistency, minimax optimality) rooted in information theory. The paper also discusses practical limitations. The convergence speed depends on the specific universal code and its internal model order; for high‑dimensional or high‑precision real‑valued data, quantization introduces bias, and the computational cost of sophisticated compressors (e.g., context‑tree weighting with large contexts) can become prohibitive.

In conclusion, the authors demonstrate that universal source coding is not merely a data‑compression tool but a versatile statistical engine capable of unifying estimation, prediction, classification, and hypothesis testing for both discrete and continuous time series. Their experimental results consistently show that, especially in modest‑sample regimes, the compression‑based methods often outperform traditional techniques, suggesting a promising direction for future research that blends information‑theoretic compression with modern statistical inference.