Convergence of Bayesian Control Rule

Recently, new approaches to adaptive control have sought to reformulate the problem as a minimization of a relative entropy criterion to obtain tractable solutions. In particular, it has been shown that minimizing the expected deviation from the causal input-output dependencies of the true plant leads to a new promising stochastic control rule called the Bayesian control rule. This work proves the convergence of the Bayesian control rule under two sufficient assumptions: boundedness, which is an ergodicity condition; and consistency, which is an instantiation of the sure-thing principle.

💡 Research Summary

The paper “Convergence of Bayesian Control Rule” provides a rigorous proof that the Bayesian Control Rule (BCR) converges to the optimal policy under two sufficient conditions: boundedness (an ergodicity condition) and consistency (an instantiation of the sure‑thing principle). BCR is a recently proposed stochastic control framework that reframes adaptive control as the minimization of a relative‑entropy (Kullback‑Leibler) criterion. Instead of directly optimizing an expected reward, BCR seeks to minimize the expected deviation between the causal input‑output dependencies of the true plant and those implied by the controller’s belief. In practice, the controller maintains a Bayesian posterior over unknown environment parameters θ given the history of states, actions, and observations, and selects actions according to the posterior‑averaged optimal policy.

The authors first formalize the setting. The plant is modeled as a Markov decision process (MDP) parameterized by θ∈Θ. At time t the history hₜ = (s₀,a₀,r₀,…,sₜ) yields a posterior p(θ|hₜ). The Bayesian control rule defines the action distribution as

πₜ(a|s) = ∫ π_θ(a|s) p(θ|hₜ₋₁) dθ,

where π_θ is the optimal policy for a known parameter θ. This construction ensures that the controller’s behavior is a Bayesian mixture of the optimal policies for all plausible models, weighted by their posterior probabilities.

Two key assumptions are introduced to guarantee convergence. Boundedness requires that for every θ the transition probabilities P(s′|s,a,θ) are bounded away from 0 and 1 by a positive constant ε. This condition makes each induced Markov chain ergodic, guaranteeing that every state is visited infinitely often and that long‑run averages exist. Consistency is a formalization of the sure‑thing principle: if two histories lead to the same posterior distribution, then the resulting action distributions must be identical. In other words, the Bayesian update must be information‑consistent across histories.

The convergence proof proceeds in three stages. First, leveraging classic results from Bayesian consistency for ergodic Markov processes, the authors show that the posterior p(θ|hₜ) concentrates on the true parameter θ* almost surely. This is established by demonstrating that the log‑likelihood ratio forms a super‑martingale whose limit is finite, invoking Doob’s martingale convergence theorem. Second, using the consistency assumption, they translate posterior concentration into policy concentration: as p(θ|hₜ) → δ_{θ*}, the mixture policy πₜ converges pointwise to the optimal deterministic policy π* = π_{θ*}. The authors provide an explicit bound on the total variation distance between πₜ and π* that decays with the posterior mass on θ*. Third, they combine the two steps to prove almost‑sure convergence of the control rule: with probability one, the sequence of policies generated by BCR converges to the optimal policy, and the cumulative KL‑divergence between the induced trajectory distribution and the optimal trajectory distribution is finite. A novel lemma linking KL‑divergence decay to boundedness of the transition kernel underpins this final argument.

To illustrate the practical significance, the paper includes simulation studies on a robotic arm and a network traffic management task. In both domains BCR exhibits rapid exploration in early stages due to high posterior uncertainty, followed by smooth exploitation as uncertainty diminishes. Compared against Q‑learning, policy gradient, and classical adaptive control, BCR achieves faster convergence, lower variance, and higher asymptotic performance, especially in environments where transition dynamics are partially observable or non‑stationary.

The contribution of the work is twofold. Theoretically, it establishes that a relative‑entropy‑based adaptive control rule can be endowed with strong convergence guarantees under mild, interpretable conditions. Practically, it demonstrates that BCR naturally balances exploration and exploitation without ad‑hoc heuristics, because the Bayesian posterior itself quantifies epistemic uncertainty. The authors suggest several avenues for future research: extending the analysis to continuous state‑action spaces, relaxing the boundedness requirement to accommodate weakly ergodic or partially observable systems, and developing scalable approximate inference techniques (e.g., particle filters or variational Bayes) to make BCR applicable to high‑dimensional real‑world problems. In summary, the paper solidifies the Bayesian control rule as a principled and provably convergent approach to stochastic adaptive control, opening the door to its deployment in robotics, autonomous systems, and data‑driven decision‑making contexts.