Joint universal lossy coding and identification of stationary mixing sources

The problem of joint universal source coding and modeling, treated in the context of lossless codes by Rissanen, was recently generalized to fixed-rate lossy coding of finitely parametrized continuous-alphabet i.i.d. sources. We extend these results to variable-rate lossy block coding of stationary ergodic sources and show that, for bounded metric distortion measures, any finitely parametrized family of stationary sources satisfying suitable mixing, smoothness and Vapnik-Chervonenkis learnability conditions admits universal schemes for joint lossy source coding and identification. We also give several explicit examples of parametric sources satisfying the regularity conditions.

💡 Research Summary

The paper tackles the long‑standing problem of performing universal source coding and simultaneous model identification for a broad class of stochastic processes. Building on Rissanen’s universal lossless coding‑and‑modeling framework, the authors extend the theory to variable‑rate lossy block coding of stationary ergodic sources that satisfy certain mixing, smoothness, and learnability conditions.

Problem setting.
A family of sources {Pθ : θ∈Θ} is considered, where Θ⊂ℝk is a compact, finite‑dimensional parameter set. For each θ the process is stationary, ergodic, and β‑mixing with exponentially decaying coefficients. The distortion measure d(·,·) is a bounded metric (max d≤dmax). The goal is, for any block length n, to design a single coding scheme that (i) achieves a rate‑distortion performance close to the optimal R(D;θ) for the true source, and (ii) produces an estimate (\hat θ_n) that converges to the true θ at a fast rate, all without prior knowledge of θ.

Assumptions.

1. Mixing: β(t)≤c e−αt for some α>0, guaranteeing rapid decay of temporal dependence and enabling concentration inequalities for dependent samples.
2. Smoothness: The log‑density ℓθ(x)=log pθ(x) is Lipschitz in θ and twice differentiable with a positive‑definite Fisher information matrix I(θ). This yields standard parametric convergence rates for maximum‑likelihood‑type estimators.
3. VC‑learnability: The collection of decision regions induced by the distortion (e.g., {x: d(x,y)≤τ}) has finite Vapnik‑Chervonenkis dimension. Consequently, empirical risk minimization over Θ enjoys uniform convergence at a rate O(√(VC·log n / n)).

Two‑stage universal scheme.
Stage 1 – Parameter estimation and lossless coding. From the observed block Xⁿ the encoder computes a parametric estimator (\hat θ_n) (e.g., MMSE or penalized MLE). The estimate is then encoded losslessly using a universal code such as Lempel‑Ziv or adaptive arithmetic coding. The extra bits required are O(log n), negligible compared with the main lossy block length. Under the mixing and smoothness assumptions the estimation error satisfies
\