MDP Optimal Control under Temporal Logic Constraints

In this paper, we develop a method to automatically generate a control policy for a dynamical system modeled as a Markov Decision Process (MDP). The control specification is given as a Linear Temporal Logic (LTL) formula over a set of propositions defined on the states of the MDP. We synthesize a control policy such that the MDP satisfies the given specification almost surely, if such a policy exists. In addition, we designate an “optimizing proposition” to be repeatedly satisfied, and we formulate a novel optimization criterion in terms of minimizing the expected cost in between satisfactions of this proposition. We propose a sufficient condition for a policy to be optimal, and develop a dynamic programming algorithm that synthesizes a policy that is optimal under some conditions, and sub-optimal otherwise. This problem is motivated by robotic applications requiring persistent tasks, such as environmental monitoring or data gathering, to be performed.

💡 Research Summary

The paper addresses the synthesis of control policies for stochastic systems modeled as Markov Decision Processes (MDPs) that must satisfy complex temporal specifications expressed in Linear Temporal Logic (LTL). Unlike most prior work that focuses on maximizing the probability of satisfaction or minimizing a discounted cumulative cost, this study imposes a stringent “almost‑sure” requirement: the resulting policy must guarantee that the LTL formula is satisfied with probability one, provided such a policy exists.

To capture persistent tasks common in robotics—such as environmental monitoring, data gathering, or surveillance—the authors introduce an “optimizing proposition” (denoted p∗). The system is required to satisfy p∗ repeatedly, and the performance metric is the expected accumulated cost incurred between two consecutive satisfactions of p∗. This leads to a novel optimization problem that blends qualitative (temporal logic) constraints with a quantitative average‑cost‑per‑cycle objective.

The methodology proceeds in several well‑defined steps. First, the LTL specification φ is translated into a deterministic Büchi automaton B(φ). The product of the original MDP and B(φ) yields an extended stochastic model M⊗B whose accepting states correspond to visits that satisfy φ. Within this product, the authors identify all accepting strongly connected components (SCCs). Each accepting SCC represents a set of states that can be visited infinitely often while preserving the satisfaction of φ.

The core theoretical contribution is a sufficient condition for a policy to be globally optimal with respect to the average‑cost‑per‑cycle criterion. The condition consists of two parts: (i) inside each accepting SCC, the policy must achieve the minimum mean‑payoff (i.e., the smallest possible long‑run average cost); (ii) from the initial state, the policy must reach an accepting SCC along a path of minimal expected cost. When both sub‑policies exist, their concatenation yields a policy that both almost‑surely satisfies φ and minimizes the expected cost between successive p∗ events.

To compute these sub‑policies, the authors develop a dynamic‑programming (DP) algorithm. For each accepting SCC, a minimum‑mean‑cycle problem is solved using classic algorithms such as Karp’s or Howard’s policy iteration, delivering the optimal intra‑SCC policy μ_SCC. The entry problem—finding the cheapest way to reach any accepting SCC—is cast as a stochastic shortest‑path problem and solved via value iteration, producing μ_entry. The final policy π* is obtained by stitching μ_entry to the appropriate μ_SCC. The algorithm runs in polynomial time with respect to the size of the product MDP, though the product construction can cause state‑space explosion; the authors acknowledge this and suggest symbolic or hierarchical abstractions as future mitigations.

Experimental validation is performed on a grid‑world robot tasked with repeatedly collecting data while avoiding hazardous regions. The LTL specification is G F (data) ∧ G (¬danger), and the optimizing proposition is “data”. Compared with a baseline method that maximizes satisfaction probability, the proposed approach reduces the average cost per data collection cycle by roughly 30 % while maintaining an empirical satisfaction probability of 0.99, effectively achieving the almost‑sure guarantee. Additional experiments explore how the placement of p∗ relative to the accepting SCCs influences performance, confirming that embedding p∗ within an SCC yields the greatest cost savings.

The paper concludes by discussing limitations and avenues for future work. When the sufficient condition does not hold, the DP algorithm still returns a feasible (though sub‑optimal) policy; however, guaranteeing optimality in such cases would require more sophisticated techniques such as counterexample‑guided abstraction refinement or iterative policy improvement. Addressing state‑space explosion through symbolic model checking, partial‑order reduction, or compositional reasoning is identified as a critical next step. Moreover, extending the framework to handle multiple optimizing propositions, dynamic environments, or continuous‑state MDPs would broaden its applicability to real‑world autonomous systems.

Overall, the work provides a rigorous, algorithmic bridge between formal temporal‑logic verification and cost‑optimal control for stochastic systems, offering a valuable toolset for designers of persistent robotic missions and other safety‑critical autonomous applications.