Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models

Score-based generative models (SGMs) have emerged as one of the most popular classes of generative models. A substantial body of work now exists on the analysis of SGMs, focusing either on discretization aspects or on their statistical performance. In the latter case, bounds have been derived, under various metrics, between the true data distribution and the distribution induced by the SGM, often demonstrating polynomial convergence rates with respect to the number of training samples. However, these approaches adopt a largely approximation theory viewpoint, which tends to be overly pessimistic and relatively coarse. In particular, they fail to fully explain the empirical success of SGMs or capture the role of the optimization algorithm used in practice to train the score network. To support this observation, we first present simple experiments illustrating the concrete impact of optimization hyperparameters on the generalization ability of the generated distribution. Then, this paper aims to bridge this theoretical gap by providing the first algorithmic- and data-dependent generalization analysis for SGMs. In particular, we establish bounds that explicitly account for the optimization dynamics of the learning algorithm, offering new insights into the generalization behavior of SGMs. Our theoretical findings are supported by empirical results on several datasets.

💡 Research Summary

This paper, titled “Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models,” presents a novel theoretical framework for analyzing the generalization performance of score-based generative models (SGMs/diffusion models). It identifies a key limitation in prior theoretical work, which primarily adopted an approximation-theoretic viewpoint. Such approaches, while providing polynomial convergence rates, are often pessimistic and coarse because they ignore the crucial role of the practical optimization algorithm used to train the score network and the specific properties of the training dataset.

To bridge this gap, the authors first provide empirical evidence (Figure 1) showing that optimizer hyperparameters like learning rate and batch size in Adam significantly affect generation quality, as measured by Wasserstein distance and Fréchet Inception Distance (FID) on various datasets. This motivates the need for an analysis that accounts for algorithmic choices.

The core theoretical contribution is a refined decomposition of the score approximation error term (ε_s), which appears in existing high-level generalization bounds for SGMs. The authors show that for any network parameter θ, this error can be expressed as: ε_s(θ) = L_ESM(θ) + Δ_s + G_l(θ).

1. L_ESM(θ) is the explicit score-matching loss actively minimized during training.
2. Δ_s is a data-dependent concentration term that captures the statistical fluctuation between the empirical dataset and the true data distribution in the context of the forward process. The paper bounds this term, linking it to the smooth Wasserstein distance and showing it scales as O(1/√n) plus the discretization error.
3. G_l(θ) is termed the “score generalization gap,” representing the difference between the true risk and the empirical risk of the score estimator at θ.

This decomposition is pivotal because it isolates the components influenced by different factors. The term G_l(θ) is shown to be amenable to analysis using existing algorithm-dependent generalization bounds from learning theory. The authors invoke bounds based on gradient norms and optimization path properties, suggesting that the characteristics of the training trajectory (not just the final parameter) contain valuable information about generalization.

Consequently, the paper derives high-probability bounds on the KL divergence between the true and generated distributions that take the form: L_ESM(θ) + O(1/√n) + E_i + E_d, where E_i and E_d are initialization and discretization errors, respectively. This result explicitly shows how the final generalization error depends jointly on the optimization quality (reflected in L_ESM), the amount of data (1/√n), and inherent modeling approximations.

In summary, this work provides the first algorithmic- and data-dependent generalization analysis for diffusion models. By moving beyond pure approximation theory and incorporating the dynamics of practical learning algorithms, it offers more nuanced and potentially less pessimistic explanations for the empirical success of SGMs, opening new avenues for theoretically understanding and improving their training.