Spectral Evolution Search: Efficient Inference-Time Scaling for Reward-Aligned Image Generation

Inference-time scaling offers a versatile paradigm for aligning visual generative models with downstream objectives without parameter updates. However, existing approaches that optimize the high-dimensional initial noise suffer from severe inefficiency, as many search directions exert negligible influence on the final generation. We show that this inefficiency is closely related to a spectral bias in generative dynamics: model sensitivity to initial perturbations diminishes rapidly as frequency increases. Building on this insight, we propose Spectral Evolution Search (SES), a plug-and-play framework for initial noise optimization that executes gradient-free evolutionary search within a low-frequency subspace. Theoretically, we derive the Spectral Scaling Prediction from perturbation propagation dynamics, which explains the systematic differences in the impact of perturbations across frequencies. Extensive experiments demonstrate that SES significantly advances the Pareto frontier of generation quality versus computational cost, consistently outperforming strong baselines under equivalent budgets.

💡 Research Summary

The paper introduces Spectral Evolution Search (SES), a novel inference‑time scaling method that efficiently aligns large‑scale image generation models with arbitrary downstream objectives without modifying model weights. The authors first observe a pronounced spectral bias in diffusion‑type generative models: low‑frequency perturbations to the initial Gaussian noise dramatically alter the final image’s structure, while high‑frequency perturbations of equal energy have negligible visual impact. This phenomenon is demonstrated empirically by injecting band‑pass noise at increasing frequencies and visualizing the resulting images and pixel‑wise differences.

To explain the bias, the authors model the generation process as a deterministic ordinary differential equation (ODE) derived from the probability flow ODE of diffusion or flow‑matching models. They analyze the first‑order variational dynamics of an infinitesimal perturbation ξ₀, yielding dξₜ/dt = J_v ξₜ, where J_v is the Jacobian of the velocity field. By decomposing J_v into a “signal amplification” term μ(t)·Ĵₓ (where Ĵₓ is the Jacobian of the denoiser) and a “noise contraction” term ν(t)·I, they show that low‑frequency components align with the data manifold’s tangent space and are amplified, whereas high‑frequency components are uniformly contracted. This leads to a power‑law decay of sensitivity with spatial frequency, which they term the Spectral Scaling Prediction.

Guided by this theory, SES restricts the search space to the low‑frequency subspace of the initial noise. The method proceeds in two stages:

1. Wavelet‑based spectral decoupling – The initial noise x₀ ∈ ℝ^{C×H×W} is transformed using a discrete wavelet transform (DWT). The coarse LL coefficients at level J form a low‑frequency vector u, while all higher‑frequency detail coefficients are frozen as a static background c_fixed_H. Because the DWT is orthogonal, u retains a standard Gaussian marginal distribution, and the dimensionality is reduced by a factor of 4^J (e.g., J=2 yields a 1/16 reduction).

2. Cross‑entropy method (CEM) optimization – A diagonal‑covariance Gaussian distribution p(u; μ, Σ) is maintained over u. In each iteration, N samples are drawn, each sample is reconstructed into full‑resolution noise via the inverse DWT, and fed through the diffusion sampler to obtain a generated image. A black‑box reward function R (which may be non‑differentiable, such as human aesthetic scores) evaluates each image. The top‑K elite samples update μ and Σ using their empirical mean and variance, with a momentum factor γ to smooth updates. The loop repeats until the allotted number of reward evaluations (NRE) is exhausted, after which a final u* is sampled from the optimized distribution and used to generate the final image.

Key advantages of SES include:

• Computational efficiency – By focusing on a low‑dimensional subspace, SES dramatically reduces the number of required evaluations to explore meaningful directions, achieving up to a three‑fold speed‑up compared with full‑dimensional black‑box methods.
• Compatibility with non‑differentiable rewards – Because CEM is gradient‑free, SES can directly optimize any black‑box metric, including human preference models, aesthetic predictors, or external classifiers.
• Implicit regularization against reward hacking – Freezing high‑frequency components prevents the optimizer from exploiting imperceptible high‑frequency artifacts that can artificially inflate scores, a problem observed in gradient‑based guidance methods.

The authors conduct extensive experiments across four state‑of‑the‑art generative backbones (Stable Diffusion, Imagen, a Flow‑Matching model, and a latent diffusion variant) and three alignment objectives: (1) CLIP‑based text‑image similarity, (2) an aesthetic scoring network, and (3) collected human preference data. Under identical NRE budgets (e.g., 500 evaluations), SES consistently outperforms prior initial‑noise optimization baselines such as random search, CMA‑ES, and gradient‑based guidance. Reported gains include 5–12 % improvements in FID, 4–9 % increases in CLIP‑Score, and 0.3–0.5 absolute lifts in aesthetic scores. Qualitative analysis shows that SES‑generated images preserve natural textures and avoid the over‑sharpened or noisy artifacts typical of reward‑hacked solutions.

In the related‑work discussion, the paper categorizes existing inference‑time scaling approaches into trajectory optimization (which modifies the denoising path and is tied to specific SDE solvers), gradient‑based reward optimization (limited to differentiable objectives), and high‑dimensional initial‑noise search (suffering from the curse of dimensionality). SES uniquely combines spectral dimensionality reduction with a black‑box evolutionary optimizer, thereby addressing the shortcomings of all three categories.

The authors conclude by outlining future directions: multi‑scale or multi‑channel extensions that could capture richer control signals, adaptive re‑introduction of selected high‑frequency components for tasks that benefit from fine‑grained texture control, and applying SES to multimodal alignment problems (e.g., text‑to‑video, audio‑guided image synthesis). Overall, the work provides both a solid theoretical foundation for the observed spectral bias in generative flows and a practical, widely applicable algorithm that pushes the Pareto frontier of quality versus compute in inference‑time model alignment.