Geometric Regularity in Deterministic Sampling Dynamics of Diffusion-based Generative Models

Diffusion-based generative models employ stochastic differential equations (SDEs) and their equivalent probability flow ordinary differential equations (ODEs) to establish a smooth transformation between complex high-dimensional data distributions and tractable prior distributions. In this paper, we reveal a striking geometric regularity in the deterministic sampling dynamics of diffusion generative models: each simulated sampling trajectory along the gradient field lies within an extremely low-dimensional subspace, and all trajectories exhibit an almost identical boomerang shape, regardless of the model architecture, applied conditions, or generated content. We characterize several intriguing properties of these trajectories, particularly under closed-form solutions based on kernel-estimated data modeling. We also demonstrate a practical application of the discovered trajectory regularity by proposing a dynamic programming-based scheme to better align the sampling time schedule with the underlying trajectory structure. This simple strategy requires minimal modification to existing deterministic numerical solvers, incurs negligible computational overhead, and achieves superior image generation performance, especially in regions with only 5 - 10 function evaluations.

💡 Research Summary

The paper investigates the deterministic sampling dynamics of diffusion‑based generative models, focusing on the probability‑flow ordinary differential equation (PF‑ODE) that underlies the backward generation process. While diffusion models are typically described by a forward stochastic differential equation (SDE) that gradually adds Gaussian noise to data and a backward SDE that removes it using a learned score function, the authors concentrate on the deterministic ODE obtained by setting the stochasticity parameter to zero. This ODE enables a purely deterministic trajectory from a white‑noise sample to a data point, driven solely by the learned score (or equivalently a denoising model).

The central empirical discovery is a striking geometric regularity: every sampling trajectory, regardless of the random seed, model architecture, or conditioning, follows an almost identical “boomerang” shape. When projected onto a low‑dimensional subspace, each high‑dimensional path can be faithfully represented by just three orthogonal directions: the displacement vector from start to end, and two orthogonal deviation bases. In this three‑dimensional subspace the trajectories are fully described by Frenet‑Serret formulas, exhibiting nearly the same curvature and torsion across samples. Consequently, the entire high‑dimensional dynamics are confined to an extremely low‑dimensional manifold.

To explain this phenomenon theoretically, the authors derive a closed‑form solution for the denoising trajectory under a kernel density estimate (KDE) of the data distribution with varying bandwidths. They show that the KDE‑based solution is mathematically equivalent to the classical mean‑shift algorithm, and that as the bandwidth shrinks it converges to the optimal trajectory defined by the true data distribution. Although the KDE solution is not directly usable for large‑scale sampling, it provides a solid analytical foundation for the observed regularity. Under a Gaussian data assumption, they further bound the trajectory deviation and prove that the path consists of a linear‑nonlinear‑linear “mode‑seek” segment of roughly constant length, confirming the empirical linear‑nonlinear‑linear pattern reported in prior works.

Building on these insights, the paper proposes a practical acceleration technique: a dynamic‑programming (DP) algorithm that optimally selects the time‑step schedule to align with the intrinsic trajectory geometry. The DP objective minimizes the cumulative geometric distortion of the projected path, effectively allocating larger steps where the trajectory is nearly straight and finer steps near the high‑curvature region (the “turn” of the boomerang). Implementation requires only a minor modification to existing deterministic ODE solvers and incurs negligible computational overhead.

Extensive experiments on state‑of‑the‑art diffusion models (Stable‑Diffusion, Imagen, DALL·E‑2) demonstrate that the DP‑optimized schedule dramatically improves sample quality when the total number of function evaluations is very low (≤ 10). Metrics such as FID improve by 5–10 % and PSNR/SSIM increase by 0.3–0.7 dB compared to standard uniform or heuristic schedules, while the wall‑clock time remains essentially unchanged. Visual inspection confirms that the generated images retain fine details and exhibit fewer artifacts, especially in the early stages of generation where most existing acceleration methods struggle.

In summary, the authors reveal that deterministic diffusion sampling is not a high‑dimensional chaotic process but is governed by a simple, low‑dimensional geometric structure. This insight unifies several empirical heuristics (shared schedules, large early steps) under a common theoretical framework and opens new avenues for efficient sampling, better interpretability, and controlled generation in diffusion models.