Latent Structure Emergence in Diffusion Models via Confidence-Based Filtering

Diffusion models rely on a high-dimensional latent space of initial noise seeds, yet it remains unclear whether this space contains sufficient structure to predict properties of the generated samples, such as their classes. In this work, we investigate the emergence of latent structure through the lens of confidence scores assigned by a pre-trained classifier to generated samples. We show that while the latent space appears largely unstructured when considering all noise realizations, restricting attention to initial noise seeds that produce high-confidence samples reveals pronounced class separability. By comparing class predictability across noise subsets of varying confidence and examining the class separability of the latent space, we find evidence of class-relevant latent structure that becomes observable only under confidence-based filtering. As a practical implication, we discuss how confidence-based filtering enables conditional generation as an alternative to guidance-based methods.

💡 Research Summary

This paper investigates whether the high‑dimensional latent space of diffusion models encodes class‑level information that can be accessed before generation. Focusing on deterministic DDIM sampling, the authors define a label and a confidence score for each generated image using a pretrained classifier. By composing the classifier with the inverse diffusion map, they obtain label and confidence functions on the initial noise seeds. The key hypothesis is that low‑confidence seeds lie near class boundaries and correspond to low‑density regions of the data distribution, where the score term ∇log p_t can become unstable, making label prediction from the seed difficult. Conversely, high‑confidence seeds reside in high‑density regions, yielding stable flow dynamics and preserving class information in the latent space.

To test this, the authors partition random seeds into confidence bands and train a latent classifier gℓ on each band. Cross‑band evaluation shows that classifiers trained on high‑confidence seeds achieve strong accuracy even on seeds from other bands, whereas those trained on low‑confidence seeds perform near chance. This demonstrates that class information is concentrated in the high‑confidence subset of the latent space.

The structural analysis uses a two‑step visualization pipeline: Linear Discriminant Analysis (LDA) to extract class‑aligned linear directions, followed by Uniform Manifold Approximation and Projection (UMAP) for 2‑D embedding. High‑confidence seeds form clearly separated clusters after LDA‑UMAP, while low‑confidence seeds appear mixed, indicating that latent class structure emerges only after confidence filtering. Direct UMAP on raw seeds fails to reveal any class separation, confirming that LDA is essential for exposing the latent geometry.

Building on these findings, the authors propose a confidence‑based filtering method for conditional generation. By selecting seeds that both have high classifier confidence and belong to a desired class, and then feeding only these seeds to the unchanged diffusion model, one can generate class‑conditioned samples without modifying the model or using gradient‑based guidance. Experiments show that this approach yields high‑quality conditional samples, especially when the proportion of high‑confidence seeds is sufficient.

The paper’s contributions are threefold: (1) empirical evidence that diffusion model seeds can predict class labels, especially for high‑confidence seeds; (2) demonstration that latent space exhibits class‑separable structure only after confidence filtering; (3) a practical, model‑agnostic conditional generation technique based on confidence filtering. Limitations include reduced sampling efficiency when high‑confidence seeds are scarce, the need to tune confidence thresholds for different datasets, and validation only on relatively small benchmark datasets. Future work is suggested on improving the prevalence of high‑confidence seeds, formalizing the relationship between confidence and flow stability, scaling to large‑scale and multimodal data, and combining filtering with existing guidance methods. Overall, the study reveals that diffusion models do embed class‑relevant latent structure, which becomes accessible through confidence‑based filtering.