Consistency of Feature Markov Processes

We are studying long term sequence prediction (forecasting). We approach this by investigating criteria for choosing a compact useful state representation. The state is supposed to summarize useful information from the history. We want a method that is asymptotically consistent in the sense it will provably eventually only choose between alternatives that satisfy an optimality property related to the used criterion. We extend our work to the case where there is side information that one can take advantage of and, furthermore, we briefly discuss the active setting where an agent takes actions to achieve desirable outcomes.

💡 Research Summary

The paper tackles the problem of long‑term sequence prediction by focusing on how to construct a compact state representation that captures all predictive information contained in the past. Traditional Markov models assume a fixed‑size state space, which often fails to summarize complex histories efficiently. To overcome this limitation, the authors introduce the notion of a Feature Markov Process (FMP). An FMP is defined by a set of candidate feature functions Φ = {φ₁, φ₂, …}. Each φ maps a history X₁:t to a latent state Z_t = φ(X₁:t). Given a particular φ, the process becomes a standard Markov chain on Z_t with transition probabilities P(Z_{t+1} | Z_t) and an emission model P(X_{t+1} | Z_{t+1}). The central question is how to select a φ that yields a “useful” state—i.e., one that preserves all information needed for optimal prediction—while guaranteeing that the selection method is asymptotically consistent.

Two statistical criteria are proposed for evaluating candidate φ’s. The first is based on the Minimum Description Length (MDL) principle: the total coding length consists of a model‑complexity term (the number of bits required to encode φ and its transition parameters) plus a data‑fit term (the negative log‑likelihood of the observed sequence). The φ that minimizes this sum is preferred. The second criterion adopts a Bayesian perspective, selecting the φ that maximizes the posterior probability (MAP estimate) under a suitably chosen prior over feature functions and transition parameters. Both criteria balance model simplicity against predictive accuracy.

The authors prove that, under mild regularity conditions, the selection procedure is asymptotically consistent: as the number of observations N → ∞, the chosen feature function φ̂_N converges in probability to the set Φ* of optimal feature functions that minimize the true expected coding length (or equivalently maximize the true posterior). The proof proceeds in two stages. First, they show that the empirical MDL (or MAP) objective converges uniformly to its population counterpart, which is minimized exactly by Φ*. Second, assuming the candidate set Φ is sufficiently rich (e.g., dense in a suitable function space) and each φ induces a finite‑dimensional parametrized Markov chain, they derive exponential concentration bounds that guarantee the probability of selecting a sub‑optimal φ decays exponentially with N. The analysis leverages tools from algebraic geometry (to handle the dimensionality of the parameter manifolds) and large‑deviation theory.

The framework is then extended to incorporate side information S_t (such as exogenous variables). The state is augmented to (Z_t, S_t) and the transition model becomes conditionally Markovian: P(Z_{t+1} | Z_t, S_t). The feature function φ still depends only on the past observations, while S_t is treated as an observed covariate. The authors demonstrate that the consistency results carry over unchanged, and that, when S_t carries substantial predictive power, the convergence of φ̂_N to Φ* can be significantly accelerated.

Finally, the paper briefly discusses the active setting where an agent selects actions A_t to influence future outcomes—a reinforcement‑learning scenario. In this case the policy π(A_t | Z_t) and the feature function φ must be learned jointly. The authors argue that if φ yields a sufficient Markovian state representation, standard RL convergence results (policy iteration, value iteration) remain applicable. Consequently, as φ̂_N converges to an optimal feature function, the induced policy π̂_N converges to the optimal policy π*.

In summary, the contribution of the paper is threefold: (1) it formalizes a principled method for learning compact, predictive state representations via feature functions; (2) it provides rigorous asymptotic guarantees that the learning procedure will eventually select only optimal or near‑optimal representations under MDL or Bayesian criteria; and (3) it extends these guarantees to settings with side information and to active decision‑making problems. By bridging the gap between classical Markov modeling and modern data‑driven sequence forecasting, the work offers a solid theoretical foundation for building scalable, reliable predictive systems.