Stochastic Optimal Control in Continuous Space-Time Multi-Agent Systems

Recently, a theory for stochastic optimal control in non-linear dynamical systems in continuous space-time has been developed (Kappen, 2005). We apply this theory to collaborative multi-agent systems. The agents evolve according to a given non-linear dynamics with additive Wiener noise. Each agent can control its own dynamics. The goal is to minimize the accumulated joint cost, which consists of a state dependent term and a term that is quadratic in the control. We focus on systems of non-interacting agents that have to distribute themselves optimally over a number of targets, given a set of end-costs for the different possible agent-target combinations. We show that optimal control is the combinatorial sum of independent single-agent single-target optimal controls weighted by a factor proportional to the end-costs of the different combinations. Thus, multi-agent control is related to a standard graphical model inference problem. The additional computational cost compared to single-agent control is exponential in the tree-width of the graph specifying the combinatorial sum times the number of targets. We illustrate the result by simulations of systems with up to 42 agents.

💡 Research Summary

The paper extends the path‑integral formulation of stochastic optimal control introduced by Kappen (2005) to collaborative multi‑agent systems operating in continuous space‑time. Each agent follows a nonlinear dynamics with additive Wiener noise and can apply its own control input. The overall objective is to minimize a cumulative cost that comprises a state‑dependent term, a quadratic control‑effort term, and a terminal cost that depends on the specific assignment of agents to a set of targets.

Focusing on the case of non‑interacting agents, the authors show that the optimal control for the whole system can be expressed as a weighted sum of independent single‑agent‑single‑target optimal controls. The weight for each agent‑target pair is proportional to the exponential of the negative terminal cost for that assignment, effectively turning the combinatorial assignment problem into a probabilistic inference problem on a graphical model. By constructing a factor graph whose variables represent possible agent‑target assignments, standard message‑passing algorithms (e.g., belief propagation or variational inference) can be used to compute the weights efficiently.

The computational complexity of the method scales as O(|V|·|G|·2^tw), where |V| is the number of agents, |G| the number of targets, and tw the tree‑width of the assignment graph. Consequently, when the graph has low tree‑width (e.g., chain‑like or tree‑structured assignments), the approach remains tractable even for large numbers of agents. The authors validate the theory with simulations involving up to 42 agents and 6 targets. In these experiments, the proposed method achieves the same or lower total cost compared with brute‑force dynamic programming, while reducing computation time dramatically; for tree‑widths of 2–3 the optimal control signals are computed in well under a second.

The key insight is the equivalence between stochastic optimal control in multi‑agent settings and graphical‑model inference. This equivalence allows the exploitation of a rich toolbox of approximate inference techniques to handle the combinatorial explosion inherent in agent‑target assignments. The work opens avenues for real‑time, scalable control of robot swarms, unmanned aerial vehicle formations, and distributed sensor networks where agents must self‑organize to cover multiple objectives under uncertainty.