Diffusion Bridge Variational Inference for Deep Gaussian Processes

Deep Gaussian processes (DGPs) enable expressive hierarchical Bayesian modeling but pose substantial challenges for posterior inference, especially over inducing variables. Denoising diffusion variational inference (DDVI) addresses this by modeling the posterior as a time-reversed diffusion from a simple Gaussian prior. However, DDVI’s fixed unconditional starting distribution remains far from the complex true posterior, resulting in inefficient inference trajectories and slow convergence. In this work, we propose Diffusion Bridge Variational Inference (DBVI), a principled extension of DDVI that initiates the reverse diffusion from a learnable, data-dependent initial distribution. This initialization is parameterized via an amortized neural network and progressively adapted using gradients from the ELBO objective, reducing the posterior gap and improving sample efficiency. To enable scalable amortization, we design the network to operate on the inducing inputs, which serve as structured, low-dimensional summaries of the dataset and naturally align with the inducing variables’ shape. DBVI retains the mathematical elegance of DDVI, including Girsanov-based ELBOs and reverse-time SDEs,while reinterpreting the prior via a Doob-bridged diffusion process. We derive a tractable training objective under this formulation and implement DBVI for scalable inference in large-scale DGPs. Across regression, classification, and image reconstruction tasks, DBVI consistently outperforms DDVI and other variational baselines in predictive accuracy, convergence speed, and posterior quality.

💡 Research Summary

This paper introduces Diffusion Bridge Variational Inference (DBVI), a novel variational framework for posterior inference in Deep Gaussian Processes (DGPs). DGPs stack multiple Gaussian‑process layers to obtain highly expressive, hierarchical Bayesian models, but inference is notoriously difficult because of non‑conjugate likelihoods, strong inter‑layer dependencies, and the large number of inducing variables required for scalability. Traditional sparse variational methods approximate the inducing‑point posterior with a factorised Gaussian, which is often too restrictive to capture the complex, multimodal distributions that arise in deep models.

A recent method, Denoising Diffusion Variational Inference (DDVI), models the variational posterior as the terminal distribution of a reverse‑time stochastic differential equation (SDE). While DDVI provides a flexible, score‑based representation, it starts the reverse diffusion from a fixed unconditional Gaussian U₀ ∼ N(0,σ²I). Because this initial distribution is typically far from the true posterior, the reverse SDE must traverse a long, intricate trajectory, leading to high variance, slow convergence, and poor sample efficiency.

DBVI addresses these shortcomings in two complementary ways. First, it replaces the fixed prior with a data‑dependent Gaussian pθ₀(U₀|x)=N(μθ(x),σ²I), where the mean μθ(x) is produced by an amortized neural network. Crucially, the network does not ingest raw inputs; instead it operates on the inducing locations Z(l) at each layer, which are low‑dimensional, structured summaries of the dataset and match the dimensionality of the inducing variables. This design yields a scalable amortization scheme that can be trained jointly with the DGP hyper‑parameters.

Second, DBVI reinterprets the diffusion process as a Doob‑h‑bridge. By applying a Doob h‑transform that conditions the forward SDE on the learned start point, the forward dynamics acquire an additional drift term g(t)² h(Uₜ,t,U₀), while the reverse‑time SDE uses a conditional score s_cond = s + h. This bridge formulation effectively “bends” the diffusion toward the posterior endpoint, dramatically shortening the reverse trajectory and reducing the KL gap between the initial distribution and the target posterior.

The authors provide a rigorous mathematical foundation. Proposition 1 derives the forward and reverse bridge SDEs under the h‑transform. Proposition 2 shows that, for a linear forward drift with the bridge correction, the marginal at any time t remains Gaussian, with mean mₜ and variance κₜ governed by a pair of coupled ODEs that depend on μθ(x). Proposition 3 then expresses the pathwise KL between the variational reverse bridge Qϕ and the reference bridge in a score‑matching form, leading to a tractable evidence lower bound (ELBO):

ℓ_DBVI = 𝔼_{U₀:₁∼Qϕ}