Multiscale Markov Decision Problems: Compression, Solution, and Transfer Learning

Many problems in sequential decision making and stochastic control often have natural multiscale structure: sub-tasks are assembled together to accomplish complex goals. Systematically inferring and leveraging hierarchical structure, particularly beyond a single level of abstraction, has remained a longstanding challenge. We describe a fast multiscale procedure for repeatedly compressing, or homogenizing, Markov decision processes (MDPs), wherein a hierarchy of sub-problems at different scales is automatically determined. Coarsened MDPs are themselves independent, deterministic MDPs, and may be solved using existing algorithms. The multiscale representation delivered by this procedure decouples sub-tasks from each other and can lead to substantial improvements in convergence rates both locally within sub-problems and globally across sub-problems, yielding significant computational savings. A second fundamental aspect of this work is that these multiscale decompositions yield new transfer opportunities across different problems, where solutions of sub-tasks at different levels of the hierarchy may be amenable to transfer to new problems. Localized transfer of policies and potential operators at arbitrary scales is emphasized. Finally, we demonstrate compression and transfer in a collection of illustrative domains, including examples involving discrete and continuous statespaces.

💡 Research Summary

The paper tackles the long‑standing challenge of automatically discovering and exploiting hierarchical structure in Markov decision processes (MDPs) that exhibit natural multiscale organization. The authors propose a fast, recursive compression (or homogenization) procedure that repeatedly coarsens an MDP, automatically generating a hierarchy of sub‑problems at increasingly abstract scales.

Core methodology

1. State aggregation – The original state space is represented as a graph and partitioned using community‑detection or spectral clustering techniques. Each partition becomes a “meta‑state” (cluster) that groups states with high intra‑cluster transition probabilities and similar reward patterns.
2. Local policy restriction – Within each cluster the action set is limited to a small, tractable subset (e.g., the locally optimal action or a predefined set). This restriction makes intra‑cluster dynamics effectively deterministic.
3. Potential operator construction – A potential operator (the expected value of staying inside a cluster) is computed and used to define deterministic transition probabilities between clusters. Rewards for inter‑cluster moves are approximated by the expected or maximal reward inside the source cluster.
4. Coarsened MDP solving – The resulting coarsened MDP is a deterministic MDP that can be solved with any standard solver (value iteration, policy iteration, linear programming, etc.). The optimal coarse policy is then “lifted” back to the fine level, providing a policy for the original problem.

These four steps are applied recursively, yielding a multi‑level hierarchy where each level is an independent deterministic MDP. The authors prove that, under mild assumptions on the quality of the clustering, the hierarchy preserves the optimal value function up to a bounded error and that convergence rates improve dramatically both locally (within clusters) and globally (across clusters).

Transfer learning opportunities
Because each level of the hierarchy isolates a sub‑task, the policies and potential operators learned at any scale can be transferred to new MDPs that share similar sub‑structures. Two transfer mechanisms are described:

• Structural transfer – When the clustering of a new problem aligns with that of a source problem, the corresponding coarse policies can be copied directly.
• Spectral/functional transfer – Even if the exact clustering differs, similarity of the potential operators’ spectra (eigenvalues/eigenvectors) allows a mapping of policies from source to target. This enables “localized” transfer at arbitrary scales, reducing the amount of new learning required.

Experimental validation
The authors evaluate the approach on several benchmark domains:

• Discrete gridworlds – Compression ratios of 5–20× are achieved, and total solution time drops to less than 30 % of that required by flat solvers.
• Continuous point‑masking tasks – State space is discretized via clustering; the same speed‑up and convergence benefits are observed.
• Transfer scenarios – Policies learned for sub‑tasks in one gridworld are transferred to a different layout. The transferred policies provide a near‑optimal initialization, cutting the number of additional iterations by a factor of 2–3 while maintaining >90 % success rates.

Theoretical contributions and future work
The paper supplies formal bounds on the error introduced by each compression step and demonstrates that the hierarchy yields exponential improvements in convergence under reasonable clustering quality. It also outlines future directions such as adaptive clustering quality metrics, extensions to high‑dimensional non‑grid graphs, and Bayesian quantification of transfer uncertainty.

Overall significance
By automatically constructing a multiscale hierarchy of deterministic MDPs, the work bridges the gap between hierarchical reinforcement learning and classical MDP solution methods. It not only accelerates planning for large‑scale problems but also creates a principled framework for reusing sub‑task solutions across different domains, offering a substantial step forward for both theory and practical applications in robotics, game AI, and complex control systems.