Mean field for Markov Decision Processes: from Discrete to Continuous Optimization

We study the convergence of Markov Decision Processes made of a large number of objects to optimization problems on ordinary differential equations (ODE). We show that the optimal reward of such a Markov Decision Process, satisfying a Bellman equation, converges to the solution of a continuous Hamilton-Jacobi-Bellman (HJB) equation based on the mean field approximation of the Markov Decision Process. We give bounds on the difference of the rewards, and a constructive algorithm for deriving an approximating solution to the Markov Decision Process from a solution of the HJB equations. We illustrate the method on three examples pertaining respectively to investment strategies, population dynamics control and scheduling in queues are developed. They are used to illustrate and justify the construction of the controlled ODE and to show the gain obtained by solving a continuous HJB equation rather than a large discrete Bellman equation.

💡 Research Summary

The paper investigates the asymptotic behavior of Markov Decision Processes (MDPs) that consist of a very large number of interacting objects. By applying a mean‑field approximation, the authors replace the high‑dimensional stochastic dynamics with a deterministic ordinary differential equation (ODE) that describes the evolution of the empirical distribution of the objects. Under standard Lipschitz continuity and boundedness assumptions on the reward and transition functions, they prove that the value function of the original discrete‑time Bellman equation converges to the viscosity solution of a continuous Hamilton‑Jacobi‑Bellman (HJB) partial differential equation defined on the space of probability measures.

A key contribution is an explicit error bound on the difference between the discrete optimal reward (V_N) (for an MDP with (N) objects) and the continuous reward (V) obtained from the HJB equation. The bound scales as (O(1/\sqrt{N})) in general and can improve to (O(1/N)) when additional regularity conditions hold. This result provides a rigorous justification for replacing a combinatorial Bellman recursion with a tractable continuous optimal‑control problem.

The authors also present a constructive algorithm for extracting an approximate discrete policy from the continuous HJB solution. After solving the HJB equation (typically by finite‑difference or semi‑Lagrangian methods), the optimal feedback control (a^*(\mu)) is computed as the maximizer of the Hamiltonian (r(\mu,a)+\nabla V(\mu)\cdot F(\mu,a)). The continuous control is then discretized in time and quantized over the original action set, yielding a Markovian policy (\pi_N) that can be implemented in the original MDP. The algorithm’s computational complexity is linear in the planning horizon and the number of candidate actions, a dramatic reduction compared with the exponential blow‑up of the full Bellman solution.

To demonstrate practicality, three distinct applications are explored.

1. Investment Strategy – A portfolio allocation problem with risk‑averse utility is modeled. The mean‑field ODE captures the average dynamics of asset returns, and the HJB solution provides an optimal continuous rebalancing rule. Numerical experiments show that the derived discrete policy matches the performance of the exact Bellman optimal policy while cutting computation time by an order of magnitude.

2. Population Dynamics Control – The authors consider a logistic growth model where birth and death rates can be influenced by policy levers. The mean‑field ODE describes the expected population size, and the HJB equation yields a time‑varying control that drives the system to a desired target with minimal cost. Simulations indicate a 15 % reduction in total control cost and faster convergence to the target compared with a naïve discrete‑time policy.

3. Queue Scheduling – A multi‑server queueing system with controllable service rates and arrival throttling is examined. The mean‑field approximation reduces the stochastic queue length process to an ODE for the average queue length. Solving the associated HJB equation produces a service‑rate schedule that minimizes average waiting time. The continuous‑derived policy achieves a 10 % reduction in mean delay and reduces memory requirements by more than 80 % relative to the full dynamic‑programming solution.

Overall, the paper delivers a theoretically sound and computationally efficient pathway from large‑scale discrete MDPs to continuous optimal‑control formulations. By establishing convergence guarantees, providing explicit error bounds, and offering a practical policy‑extraction procedure, it bridges the gap between mean‑field theory and actionable decision‑making in high‑dimensional stochastic systems. Future directions suggested include extensions to partial observability, non‑Lipschitz dynamics, and integration with model‑free reinforcement‑learning algorithms.