Decision-Theoretic Planning: Structural Assumptions and Computational Leverage

Planning under uncertainty is a central problem in the study of automated sequential decision making, and has been addressed by researchers in many different fields, including AI planning, decision analysis, operations research, control theory and economics. While the assumptions and perspectives adopted in these areas often differ in substantial ways, many planning problems of interest to researchers in these fields can be modeled as Markov decision processes (MDPs) and analyzed using the techniques of decision theory. This paper presents an overview and synthesis of MDP-related methods, showing how they provide a unifying framework for modeling many classes of planning problems studied in AI. It also describes structural properties of MDPs that, when exhibited by particular classes of problems, can be exploited in the construction of optimal or approximately optimal policies or plans. Planning problems commonly possess structure in the reward and value functions used to describe performance criteria, in the functions used to describe state transitions and observations, and in the relationships among features used to describe states, actions, rewards, and observations. Specialized representations, and algorithms employing these representations, can achieve computational leverage by exploiting these various forms of structure. Certain AI techniques – in particular those based on the use of structured, intensional representations – can be viewed in this way. This paper surveys several types of representations for both classical and decision-theoretic planning problems, and planning algorithms that exploit these representations in a number of different ways to ease the computational burden of constructing policies or plans. It focuses primarily on abstraction, aggregation and decomposition techniques based on AI-style representations.

💡 Research Summary

**
The paper “Decision-Theoretic Planning: Structural Assumptions and Computational Leverage” provides a comprehensive synthesis of decision‑theoretic planning (DTP) within the formalism of Markov decision processes (MDPs) and demonstrates how structural properties of MDPs can be exploited to obtain substantial computational savings. The authors begin by positioning DTP as a natural extension of classical AI planning: whereas traditional planners assume deterministic actions and a fully known world, DTP embraces stochastic action effects, partial observability, and utility‑based objectives. By casting virtually all sequential decision problems as MDPs, the paper makes explicit the deep connections among AI planning, decision analysis, operations research, control theory, and economics.

The core contribution is a taxonomy of structural assumptions that frequently appear in real‑world planning domains. These include (1) regularities in the reward or value function (e.g., additive, hierarchical, or factored forms), (2) conditional independencies in transition and observation models that allow factored or Bayesian‑network representations, (3) feature‑based (intensional) state descriptions rather than exhaustive enumerations, and (4) goal or success criteria that are probabilistic rather than binary. For each class, the authors describe how the structure can be made explicit and then leveraged algorithmically.

Three families of algorithmic techniques are examined in depth: abstraction, aggregation, and decomposition. Abstraction maps concrete states to a coarser set of abstract states, solves a reduced MDP, and then refines the abstract policy back to the original level. Aggregation groups together states that are “similar” with respect to transition dynamics and rewards, enabling the computation of a single value for an entire cluster. Decomposition splits a large MDP into independent or weakly coupled sub‑MDPs (e.g., separating mail‑handling from coffee‑delivery tasks) and recombines the sub‑policies. All three techniques mirror classic AI planning methods such as goal regression, planning graphs, and hierarchical task networks, but are reinterpreted in the language of dynamic programming and policy iteration.

The paper also discusses how traditional dynamic‑programming algorithms (value iteration, policy iteration, linear programming) can be adapted to work directly with factored or relational representations, avoiding the need to enumerate the full state space. By representing transition probabilities and rewards as functions over features, the Bellman backup becomes a symbolic manipulation rather than a table lookup, dramatically reducing memory and time requirements for high‑dimensional problems.

A detailed case study of a service robot in an office environment illustrates the ideas concretely. The robot’s state is described by five multi‑valued features (location, mail presence, robot‑held mail, coffee request, coffee held). Actions include moving, picking up mail, getting coffee, and delivering items; each action has stochastic outcomes (e.g., 90 % success for movement). The authors construct abstract states based solely on location, aggregate states that share the same mail/coffee status, and decompose the problem into two loosely coupled sub‑tasks (mail handling and coffee service). Experimental results show that the structured approach reduces planning time by an order of magnitude while preserving near‑optimal expected utility.

Finally, the authors outline future research directions: automatic discovery of structural regularities, integration with partial‑observable MDPs (POMDPs), online replanning where structure can be updated dynamically, and tighter theoretical bounds on the loss incurred by abstraction or aggregation. They argue that embracing structural assumptions bridges the gap between the scalability of AI planning and the rigorous optimality guarantees of operations‑research methods, positioning MDPs as a practical, unifying framework for modern planning systems.