Incremental Control Synthesis in Probabilistic Environments with Temporal Logic Constraints

In this paper, we present a method for optimal control synthesis of a plant that interacts with a set of agents in a graph-like environment. The control specification is given as a temporal logic statement about some properties that hold at the vertices of the environment. The plant is assumed to be deterministic, while the agents are probabilistic Markov models. The goal is to control the plant such that the probability of satisfying a syntactically co-safe Linear Temporal Logic formula is maximized. We propose a computationally efficient incremental approach based on the fact that temporal logic verification is computationally cheaper than synthesis. We present a case-study where we compare our approach to the classical non-incremental approach in terms of computation time and memory usage.

💡 Research Summary

The paper addresses the problem of synthesizing an optimal control policy for a deterministic plant operating in a graph‑structured environment populated by multiple probabilistic agents modeled as finite‑state Markov chains. The desired behavior of the plant is expressed as a syntactically co‑safe Linear Temporal Logic (LTL) formula that refers to properties of the vertices (e.g., “eventually visit A then B while avoiding C”). The objective is to maximize the probability that the plant’s trajectory satisfies this LTL specification despite the stochastic actions of the agents.

Traditional approaches treat the whole plant‑agents system as a single probabilistic transition system and apply probabilistic model checking or dynamic programming to compute a maximizing policy. However, this monolithic treatment suffers from state‑space explosion: the combined state space grows exponentially with the number of agents, leading to prohibitive computation time and memory consumption.

The authors propose an incremental synthesis framework that exploits the fact that verification (model checking) is generally much cheaper than synthesis. The key idea is to start with a small subset of agents, construct the product of the plant, the selected agents, and the deterministic automaton that represents the co‑safe LTL formula, and then solve a standard probabilistic verification problem (max‑probability reachability) to obtain an optimal policy for this reduced system. The resulting policy, together with the value function, is retained as a warm‑start for the next iteration. In each subsequent iteration, one additional agent is added to the model, the product automaton is updated, and the verification problem is resolved, reusing the previously computed value function to accelerate convergence.

Technical steps:

1. Model the plant as a deterministic transition system on a finite graph.
2. Model each agent as a finite‑state Markov chain with transition probabilities defined on the same graph.
3. Translate the co‑safe LTL specification into an equivalent deterministic finite‑state automaton (DFA).
4. For a given subset of agents, build the synchronous product of the plant, the agents, and the DFA, yielding a Markov decision process (MDP) whose states are triples (plant‑vertex, agents‑joint‑state, DFA‑state).
5. Solve the max‑reachability problem on this MDP using value iteration or linear programming, obtaining both the optimal policy and the maximal satisfaction probability.
6. Incrementally add agents, update the product MDP, and repeat step 5, initializing the solver with the solution from the previous iteration.

The incremental approach yields two major benefits. First, each iteration deals with a dramatically smaller state space than the full model, reducing both runtime and memory usage. Second, because the verification step reuses the value function from the previous iteration, the number of value‑iteration sweeps needed for convergence shrinks, making the overall synthesis cost close to linear in the number of agents for many practical instances.

The authors validate their method on a case study involving a robot navigating a 2‑D grid while interacting with multiple moving agents (other robots or humans). The LTL goal requires the robot to visit a sequence of target cells while avoiding hazardous cells. Experiments compare the incremental method against a classic non‑incremental synthesis that builds the full product MDP from the start. Results show that when the number of agents exceeds ten, the incremental method reduces computation time by up to 70 % and memory consumption by up to 60 %, while achieving satisfaction probabilities indistinguishable from the optimal non‑incremental solution.

Limitations discussed include: (i) the reliance on co‑safe LTL, which excludes specifications that require infinite‑time guarantees (e.g., safety or liveness properties); (ii) sensitivity to the order in which agents are added—suboptimal ordering may lead to unnecessary intermediate work; and (iii) potential loss of accuracy when agents exhibit strong inter‑dependencies that are not captured until later iterations. The paper suggests future work on extending the framework to general LTL via automata with acceptance conditions, developing heuristics for optimal agent ordering, and incorporating abstraction techniques to handle tightly coupled agents.

In summary, the paper presents a pragmatic, scalable solution for optimal control synthesis under probabilistic disturbances and temporal‑logic constraints. By leveraging cheap verification as a building block and incrementally expanding the model, it offers a viable path toward real‑time deployment in complex cyber‑physical systems such as multi‑robot coordination, autonomous vehicle fleets, and smart‑grid management.