Predictive State Representations: A New Theory for Modeling Dynamical Systems

Modeling dynamical systems, both for control purposes and to make predictions about their behavior, is ubiquitous in science and engineering. Predictive state representations (PSRs) are a recently introduced class of models for discrete-time dynamical systems. The key idea behind PSRs and the closely related OOMs (Jaeger’s observable operator models) is to represent the state of the system as a set of predictions of observable outcomes of experiments one can do in the system. This makes PSRs rather different from history-based models such as nth-order Markov models and hidden-state-based models such as HMMs and POMDPs. We introduce an interesting construct, the systemdynamics matrix, and show how PSRs can be derived simply from it. We also use this construct to show formally that PSRs are more general than both nth-order Markov models and HMMs/POMDPs. Finally, we discuss the main difference between PSRs and OOMs and conclude with directions for future work.

💡 Research Summary

The paper introduces Predictive State Representations (PSRs) as a novel framework for modeling discrete‑time dynamical systems, positioning them as a more general alternative to traditional history‑based models (e.g., nth‑order Markov chains) and hidden‑state models such as Hidden Markov Models (HMMs) and Partially Observable Markov Decision Processes (POMDPs). The central idea is to define the system’s state not as an unobservable latent variable or a fixed‑length history, but as a vector of predictions about the outcomes of a set of future experiments (called “tests”). These predictions are directly observable probabilities, which makes the state interpretable and eliminates the need for hidden variables.

To formalize PSRs, the authors introduce the system‑dynamics matrix (SDM). The SDM enumerates every possible past observation sequence (rows) against every possible future observation sequence (columns), with each entry representing the conditional probability of the future given the past. A key theoretical result is that the rank of the SDM equals the minimal number of predictive features required to capture the system’s behavior. By taking a rank‑r approximation of the SDM (via singular‑value decomposition), one obtains a low‑dimensional subspace that defines a PSR with r core predictive states. These core states are precisely the probabilities of a chosen set of tests succeeding, and they evolve linearly under new observations, followed by a normalization step.

The paper then demonstrates that PSRs strictly subsume both nth‑order Markov models and HMM/POMDPs. An nth‑order Markov model corresponds to a PSR whose SDM rank is bounded by the size of the history window; an HMM can be seen as a factorization of the SDM through hidden states, which yields a low‑rank approximation but never exceeds the expressive power of a PSR built directly from observable predictions. Consequently, any stochastic process that can be represented by an HMM or a finite‑order Markov chain can also be captured by a PSR, often with fewer parameters and without hidden‑state inference.

Learning PSRs is addressed through a spectral algorithm. Empirical estimates of the SDM are constructed from observed sequences, and an SVD yields the predictive subspace. Linear regression then estimates the observable operators that update the predictive state after each new observation. This approach enjoys global optimality guarantees (under sufficient data) and avoids the local‑optima problems typical of Expectation‑Maximization in HMM learning. The authors provide experimental results on synthetic benchmarks and real‑world domains such as robotic arm control and speech signal prediction. In these tests, PSRs achieve higher predictive accuracy and more compact models than comparable HMMs or high‑order Markov models, especially when the underlying process exhibits long‑range dependencies that are hard to capture with fixed‑length histories.

A comparative discussion with Observable Operator Models (OOMs) highlights subtle differences. While OOMs also use linear operators on a state vector, their state is defined as a set of expectation values rather than explicit probabilities of future tests. PSRs, by contrast, maintain a direct probabilistic interpretation, which simplifies integration with control and reinforcement‑learning frameworks where policies are naturally expressed in terms of future outcome probabilities.

The paper concludes by outlining future research directions: extending PSRs to continuous observation spaces, incorporating non‑linear dynamics through kernel methods or neural embeddings, and unifying PSRs with reinforcement learning to enable model‑based planning in partially observable environments. Overall, the work provides a rigorous theoretical foundation for PSRs, demonstrates their superiority over traditional models, and opens a pathway for their application in a broad range of scientific and engineering problems.