Sub-optimality bounds for certainty equivalent policies in partially observed systems

In this paper, we present a generalization of the certainty equivalence principle of stochastic control. One interpretation of the classical certainty equivalence principle for linear systems with output feedback and quadratic costs is as follows: the optimal action at each time is obtained by evaluating the optimal state-feedback policy of the stochastic linear system at the minimum mean square error (MMSE) estimate of the state. Motivated by this interpretation, we consider certainty equivalent policies for general (non-linear) partially observed stochastic systems that allow for any state estimate rather than restricting to MMSE estimates. In such settings, the certainty equivalent policy is not optimal. For models where the cost and the dynamics are smooth in an appropriate sense, we derive upper bounds on the sub-optimality of certainty equivalent policies. We present several examples to illustrate the results.

💡 Research Summary

The paper extends the classic certainty‑equivalence principle—well known from linear‑quadratic‑Gaussian (LQG) control—to general partially observable Markov decision processes (POMDPs) with possibly nonlinear dynamics and non‑Gaussian noise. The authors consider a “certainty‑equivalent” (CE) policy that first computes an arbitrary state estimate (E_t) from the observation‑action history (the estimator may be MMSE, MAP, a linear filter, or any heuristic) and then applies the optimal state‑feedback policy (\pi_{M,t}) of a fully observed MDP (M) to this estimate: (\mu_{E,t}(h_t)=\pi_{M,t}(E_t(h_t))). While this construction is optimal for LQG when (E_t) is the conditional mean, it is generally sub‑optimal for nonlinear or non‑Gaussian systems.

To quantify the performance loss, the authors impose three technical assumptions. First, the MDP (M) must satisfy a measurable‑selection condition guaranteeing the existence of an optimal deterministic policy. Second, both the transition kernel and the per‑step cost are required to be “smooth”: for any two states (s,s’) and any action (a), the Wasserstein‑1 distance between the resulting next‑state distributions is bounded by a concave, non‑decreasing function (F_{P,t}) of the state distance (d_S(s,s’)); similarly, the cost difference is bounded by another function (F_{c,t}). When these functions are linear, the assumption reduces to standard Lipschitz continuity. Third, the estimator’s worst‑case conditional error is defined as (\eta_t=\sup_{h_t}\mathbb{E}