Preserving Spectral Structure and Statistics in Diffusion Models

Standard diffusion models (DMs) rely on the total destruction of data into non-informative white noise, forcing the backward process to denoise from a fully unstructured noise state. While ensuring diversity, this results in a cumbersome and computationally intensive image generation task. We address this challenge by proposing new forward and backward process within a mathematically tractable spectral space. Unlike pixel-based DMs, our forward process converges towards an informative Gaussian prior N(mu_hat,Sigma_hat) rather than white noise. Our method, termed Preserving Spectral Structure and Statistics (PreSS) in diffusion models, guides spectral components toward this informative prior while ensuring that corresponding structural signals remain intact at terminal time. This provides a principled starting point for the backward process, enabling high-quality image reconstruction that builds upon preserved spectral structure while maintaining high generative diversity. Experimental results on CIFAR-10, CelebA and CelebA-HQ demonstrate significant reductions in computational complexity, improved visual diversity, less drift, and a smoother diffusion process compared to pixel-based DMs.

💡 Research Summary

The paper introduces Preserving Spectral Structure and Statistics (PreSS), a diffusion modeling framework that operates in the Fourier spectral domain rather than the conventional pixel space. Traditional Denoising Diffusion Probabilistic Models (DDPMs) progressively corrupt images until they become isotropic white noise N(0, I), forcing the reverse process to reconstruct all structural information from an uninformative prior. This leads to high computational cost, long sampling trajectories, and a phenomenon known as “sampling drift,” especially for high‑resolution data.

PreSS observes that natural images exhibit a power‑law distribution (≈ 1/K^α) in their Fourier spectra, with heavy‑tailed amplitude statistics. Empirically, the mean μ̂ and diagonal covariance Σ̂ of the spectral coefficients across a dataset capture these statistics well. The authors therefore define an informative Gaussian prior N(μ̂, Σ̂) as the target of the forward diffusion.

The forward process is formulated as a closed‑form Gaussian Markov chain in spectral space:
\