General and Efficient Steering of Unconditional Diffusion

Guiding unconditional diffusion models typically requires either retraining with conditional inputs or per-step gradient computations (e.g., classifier-based guidance), both of which incur substantial computational overhead. We present a general recipe for efficiently steering unconditional diffusion {without gradient guidance during inference}, enabling fast controllable generation. Our approach is built on two observations about diffusion model structure: Noise Alignment: even in early, highly corrupted stages, coarse semantic steering is possible using a lightweight, offline-computed guidance signal, avoiding any per-step or per-sample gradients. Transferable concept vectors: a concept direction in activation space once learned transfers across both {timesteps} and {samples}; the same fixed steering vector learned near low noise level remains effective when injected at intermediate noise levels for every generation trajectory, providing refined conditional control with efficiency. Such concept directions can be efficiently and reliably identified via Recursive Feature Machine (RFM), a light-weight backpropagation-free feature learning method. Experiments on CIFAR-10, ImageNet, and CelebA demonstrate improved accuracy/quality over gradient-based guidance, while achieving significant inference speedups.

💡 Research Summary

The paper tackles the problem of steering unconditional diffusion models without requiring any gradient computation at inference time, a limitation that has hampered the practicality of existing guidance methods such as classifier‑based guidance, classifier‑free guidance (CFG), and recent training‑free gradient approaches. The authors propose a two‑stage, gradient‑free steering framework that leverages two empirical observations about diffusion models. First, even at very early, highly noisy timesteps, coarse semantic information can be extracted from class‑conditional statistics (means and covariances) of the data distribution. By pre‑computing these statistics and applying a linear PCA‑based denoiser (called “noise alignment”) to the noisy latent, the generation can be nudged toward a target class without any per‑step computation. Second, strong discriminative directions exist in the activation space of the UNet (especially in encoder and bottleneck layers) and these directions, once learned at a low‑noise timestep, remain effective when applied across a wide range of intermediate timesteps. The authors identify these directions using Recursive Feature Machines (RFMs), a lightweight, back‑propagation‑free method that trains a linear classifier on forward‑noised activations to obtain a class‑specific steering vector.

During inference, the process proceeds as follows: starting from pure Gaussian noise, the sampler runs a standard DDIM schedule. In the early high‑noise region, the pre‑computed noise‑alignment operator is applied to each latent, providing a coarse alignment toward the desired class. After a predefined transition timestep, the fixed RFM steering vector is injected into the chosen activation layer at every subsequent step, effectively guiding the denoising trajectory toward finer semantic details of the target concept. Both operations are simple vector arithmetic; no gradients are computed, and the only overhead is a few additional forward passes.

The authors conduct extensive experiments on CIFAR‑10, ImageNet‑256×256, and CelebA (multi‑attribute) datasets. On CIFAR‑10, their method achieves 96.6 % guidance accuracy, surpassing training‑free gradient guidance (77.1 %) and noise‑conditioned classifier guidance (86.0 %). Per‑class FID improves from 73.9 (TFG) and 47.0 (classifier guidance) down to 41.4. Inference speed is accelerated by 16.4× relative to TFG and roughly 10× relative to CFG. Similar gains are reported on ImageNet and CelebA, including out‑of‑distribution fine‑grained concepts. Ablation studies confirm that (i) RFM directions learned from forward‑noised activations at low noise are as discriminative as those collected from reverse sampling, (ii) these directions transfer well across timesteps, and (iii) the early‑stage noise alignment is essential because forward‑noised activations become nearly random at high noise levels.

The key contributions are: (1) a principled, offline computation of class‑conditional PCA statistics for high‑noise steering; (2) adaptation of RFM to diffusion models for efficient discovery of transferable concept vectors; (3) a hybrid two‑stage guidance scheme that combines coarse statistical alignment with fine‑grained activation steering; and (4) empirical evidence that this gradient‑free approach outperforms or matches state‑of‑the‑art gradient‑based methods while offering orders‑of‑magnitude speedups. The work opens the door to fast, post‑hoc controllable generation from large unconditional diffusion models, potentially extending to multimodal settings where gradient‑based guidance is even more costly.