Distributional Reinforcement Learning with Diffusion Bridge Critics

Recent advances in diffusion-based reinforcement learning (RL) methods have demonstrated promising results in a wide range of continuous control tasks. However, existing works in this field focus on the application of diffusion policies while leaving the diffusion critics unexplored. In fact, since policy optimization fundamentally relies on the critic, accurate value estimation is far more important than policy expressiveness. Furthermore, given the stochasticity of most reinforcement learning tasks, it has been confirmed that the critic is more appropriately depicted with a distributional model. Motivated by these points, we propose a novel distributional RL method with Diffusion Bridge Critics (DBC). DBC directly models the inverse cumulative distribution function (CDF) of the Q value. This allows us to accurately capture the value distribution and prevents it from collapsing into a trivial Gaussian distribution owing to the strong distribution-matching capability of the diffusion bridge. Moreover, we further derive an analytic integral formula to address discretization errors in DBC, which is essential in value estimation. To our knowledge, DBC is the first work to employ the diffusion bridge model as the critic. Notably, DBC is also a plug-and-play component and can be integrated into most existing RL frameworks. Experimental results on MuJoCo robot control benchmarks demonstrate the superiority of DBC compared with previous distributional critic models.

💡 Research Summary

The paper addresses a critical gap in contemporary diffusion‑based reinforcement learning (RL): while most recent works have leveraged diffusion models to improve policy expressiveness, the critic—a component that fundamentally drives policy updates—has remained largely untouched. Recognizing that accurate value estimation is more important than policy expressiveness, especially in stochastic environments where a distributional view of the value function is beneficial, the authors propose a novel framework called Diffusion Bridge Critics (DBC).

Problem Identification – Gaussian Degradation
The authors first formalize the “Gaussian degradation” phenomenon. When a diffusion model is used directly as a critic and repeatedly updated via the Bellman backup operator, approximation errors accumulate. By the Central Limit Theorem, the learned distribution converges toward a Gaussian regardless of the true underlying return distribution. Theorem 4.1 provides a theoretical proof, and a toy example illustrates how standard diffusion or flow‑matching critics collapse to a trivial Gaussian after a few bootstrapping steps. This collapse eliminates multimodality and asymmetry, severely limiting the usefulness of the critic for policy improvement.

Core Idea – Modeling the Inverse CDF with a Diffusion Bridge
To overcome this, DBC does not model the Q‑value distribution directly. Instead, it learns the inverse cumulative distribution function (inverse CDF or quantile function) of the return distribution. For a given state‑action pair ((s,a)) and a quantile level (\tau\in(0,1)), the network predicts the corresponding return quantile (F^{-1}_{Z}(s,a,\tau)). This formulation aligns naturally with quantile‑based distributional RL (e.g., QR‑DQN, IQN) but removes the need for discrete quantile approximations.

The diffusion bridge is a stochastic process that connects a start point (z_{\text{start}}) (often a simple prior) to an endpoint (z_{\text{end}}) (the target return) over a fixed interval (t\in