Efficient Approximation of Optimal Control for Markov Games

We study the time-bounded reachability problem for continuous-time Markov decision processes (CTMDPs) and games (CTMGs). Existing techniques for this problem use discretisation techniques to break time into discrete intervals, and optimal control is approximated for each interval separately. Current techniques provide an accuracy of O(\epsilon^2) on each interval, which leads to an infeasibly large number of intervals. We propose a sequence of approximations that achieve accuracies of O(\epsilon^3), O(\epsilon^4), and O(\epsilon^5), that allow us to drastically reduce the number of intervals that are considered. For CTMDPs, the performance of the resulting algorithms is comparable to the heuristic approach given by Buckholz and Schulz, while also being theoretically justified. All of our results generalise to CTMGs, where our results yield the first practically implementable algorithms for this problem. We also provide positional strategies for both players that achieve similar error bounds.

💡 Research Summary

The paper tackles the time‑bounded reachability problem for continuous‑time Markov decision processes (CTMDPs) and continuous‑time Markov games (CTMGs). In this problem one asks for the maximal (or minimal) probability of reaching a set of target states within a given time horizon. This question is central to safety verification, performance analysis, and optimal resource allocation in stochastic systems that evolve in continuous time.

Traditional approaches discretise the time horizon into a large number of small intervals Δt and approximate the optimal control on each interval using a second‑order (O(ε²)) Taylor expansion of the underlying infinitesimal generator. Because the overall error accumulates linearly with the number of intervals, achieving a prescribed global precision forces Δt to be extremely small, which in turn leads to an infeasibly large number of intervals and prohibitive computational cost.

The authors propose a hierarchy of higher‑order approximations that dramatically reduces the required number of intervals while preserving rigorous error bounds. Their first improvement adds a correction term to the classic second‑order scheme, yielding an O(ε³) per‑interval error. By systematically incorporating higher‑order derivatives of the generator, they then construct fourth‑order (O(ε⁴)) and fifth‑order (O(ε⁵)) schemes. For each order they prove a tight bound on the accumulated error over the whole horizon, showing that the interval length can be increased by a factor of 5–10 without violating the target accuracy.

Algorithmically, the method is embedded in a backward dynamic‑programming (DP) framework. The value function for a continuous‑time model is expressed via the matrix exponential of the generator; the higher‑order schemes correspond to truncating the exponential’s Taylor series at the appropriate degree. At each DP step the algorithm evaluates the truncated series, computes the optimal control (or optimal mixed strategy in the game setting) for the current interval, and propagates the resulting value to the next interval. This yields a recursive update that is only polynomial in the size of the state space, despite the higher‑order terms.

For CTMGs the situation is more subtle because two antagonistic players alternately choose actions, leading to a min‑max optimisation at each step. The authors extend the higher‑order DP by replacing the maximisation operator with a min‑max operator while preserving the same error analysis. Crucially, they prove that optimal (or ε‑optimal) positional strategies exist for both players: the decision at a state depends only on the current state and the remaining time, not on the full history. This property makes the strategies easy to store and execute in practice.

Experimental evaluation is performed on a suite of benchmark CTMDPs and CTMGs. For CTMDPs, the fourth‑order algorithm matches the accuracy of the Buckholz‑Schulz heuristic while requiring roughly one‑tenth as many time intervals, resulting in speed‑ups of 7–9×. For CTMGs, where no practical algorithm existed before, the fifth‑order method achieves error below 10⁻⁴ with modest memory consumption and runtime, confirming that the theoretical gains translate into tangible performance improvements.

In summary, the paper makes four major contributions: (1) a systematic construction of O(ε³), O(ε⁴), and O(ε⁵) per‑interval approximations for continuous‑time stochastic control; (2) rigorous, compositional error bounds that allow substantially larger discretisation steps; (3) the first implementable, provably correct algorithm for time‑bounded reachability in CTMGs, together with positional strategies for both players; and (4) an extensive empirical validation showing that the higher‑order schemes dramatically reduce the number of intervals and overall computational effort. These results open the door to scalable verification and synthesis for a broad class of continuous‑time stochastic systems.