Alignment of Diffusion Models: Fundamentals, Challenges, and Future

Diffusion models have emerged as the leading paradigm in generative modeling, excelling in various applications. Despite their success, these models often misalign with human intentions and generate results with undesired properties or even harmful content. Inspired by the success and popularity of alignment in tuning large language models, recent studies have investigated aligning diffusion models with human expectations and preferences. This work mainly reviews alignment of diffusion models, covering advancements in fundamentals of alignment, alignment techniques of diffusion models, preference benchmarks, and evaluation for diffusion models. Moreover, we discuss key perspectives on current challenges and promising future directions on solving the remaining challenges in alignment of diffusion models. To the best of our knowledge, our work is the first comprehensive review paper for researchers and engineers to comprehend, practice, and research alignment of diffusion models.

💡 Research Summary

This survey paper provides a comprehensive overview of the emerging field of aligning diffusion models with human intentions and preferences. Diffusion models have become the dominant paradigm for generative AI, achieving state‑of‑the‑art results in image, video, text, audio, 3D, and molecular generation. However, their standard training objective—maximizing likelihood of the data distribution—does not guarantee that the outputs satisfy nuanced human expectations regarding aesthetics, safety, or domain‑specific constraints. Inspired by the success of alignment techniques in large language models (LLMs), recent work has begun to adapt similar strategies for diffusion models.

The authors first situate diffusion‑model alignment within the two‑stage training pipeline that has proven effective for LLMs: a large‑scale pre‑training phase followed by a post‑training alignment phase. They outline the fundamental components of alignment: (1) preference data consisting of prompts, model responses, and human feedback; (2) preference modeling, typically using pairwise comparisons encoded via Bradley‑Terry or Plackett‑Luce probabilistic models to produce scalar reward signals; and (3) alignment algorithms that optimize the model to maximize these rewards while controlling deviation from the original policy.

Two broad families of alignment methods are examined. Training‑time alignment modifies the model parameters using reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO) variants. RLHF approaches such as REINFORCE, PPO, RAFT, and RRHF are adapted to the multi‑step denoising process of diffusion models, where the policy is defined over the sequence of noise levels. DPO‑style methods directly minimize a loss that combines a preference‑based log‑sigmoid term with a KL‑divergence regularizer, yielding algorithms like Diffusion‑DPO, D3PO, and Diffusion‑KTO. These techniques have already been incorporated into high‑profile text‑to‑image systems such as Stable Diffusion 3 (SD3) and SD3‑Turbo, leading to measurable gains on human‑preference benchmarks.

Test‑time alignment, in contrast, leaves the model weights unchanged and instead injects guidance during generation. Strategies include prompt optimization, initial‑noise manipulation, attention control, and reward‑guided decoding or sampling. Such methods enable real‑time adaptation to user feedback or to enforce specific attributes (e.g., style, safety) without costly retraining.

The paper surveys the landscape of alignment datasets. Scalar preference collections such as HPD‑v1/v2, Pick‑a‑Pic, and ImageRewardDB provide single‑score human judgments. Multi‑dimensional feedback datasets like MHP and RichHF‑18K capture richer signals (style, ethical considerations, social impact). Benchmark suites (e.g., GenEval, VPEval, HEIM) and evaluation metrics span traditional image quality measures (IS, FID) and human‑alignment metrics (CLIP‑Score, Aesthetic, PickScore, ImageReward, VP‑Score). Fine‑grained evaluations assess alignment across dimensions such as textual fidelity, aesthetic quality, and societal values.

Key challenges are identified: (1) the high cost and potential bias of collecting large‑scale human preference data for high‑dimensional image spaces; (2) the reliability and safety of reward models, which can inherit dataset biases or be vulnerable to adversarial prompts; (3) stability and sample efficiency of RL‑based alignment given the long denoising trajectories; and (4) broader ethical concerns, including generation of harmful content, copyright violations, and reinforcement of societal biases.

To address these issues, the authors propose several research directions. Integrating multi‑modal and multi‑feedback signals can provide a more holistic view of human intent. Quantifying uncertainty in reward predictions (e.g., Bayesian or ensemble methods) can improve safety and enable calibrated decision‑making. Developing more efficient sampling and policy‑update algorithms—leveraging fast solvers like DDIM or DPM‑Solver—can reduce computational overhead. Finally, domain‑specific alignment (e.g., drug discovery, 3D modeling, motion synthesis) requires tailored constraints and expert feedback loops.

In conclusion, the survey positions diffusion‑model alignment as a nascent but rapidly growing field, drawing heavily on lessons from LLM alignment while confronting unique challenges posed by continuous, high‑dimensional generative spaces. By systematically organizing the components of data collection, preference modeling, algorithmic optimization, and evaluation, the paper offers a roadmap for researchers and engineers aiming to build safer, more controllable, and human‑aligned diffusion systems.