CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration

Text-to-image diffusion models, such as Stable Diffusion, can produce high-quality and diverse images but often fail to achieve compositional alignment, particularly when prompts describe complex object relationships, attributes, or spatial arrangements. Recent inference-time approaches address this by optimizing or exploring the initial noise under the guidance of reward functions that score text-image alignment without requiring model fine-tuning. While promising, each strategy has intrinsic limitations when used alone: optimization can stall due to poor initialization or unfavorable search trajectories, whereas exploration may require a prohibitively large number of samples to locate a satisfactory output. Our analysis further shows that neither single reward metrics nor ad-hoc combinations reliably capture all aspects of compositionality, leading to weak or inconsistent guidance. To overcome these challenges, we present Category-Aware Reward-based Initial Noise Optimization and Exploration (CARINOX), a unified framework that combines noise optimization and exploration with a principled reward selection procedure grounded in correlation with human judgments. Evaluations on two complementary benchmarks covering diverse compositional challenges show that CARINOX raises average alignment scores by +16% on T2I-CompBench++ and +11% on the HRS benchmark, consistently outperforming state-of-the-art optimization and exploration-based methods across all major categories, while preserving image quality and diversity. The project page is available at https://amirkasaei.com/carinox/.

💡 Research Summary

Text‑to‑image diffusion models such as Stable Diffusion generate high‑quality images but frequently fail to respect complex compositional constraints—missing objects, incorrect attributes, wrong spatial relations, and counting errors. Existing remedies fall into two categories. Fine‑tuning approaches modify model weights (UNet, text encoder, or attention modules) but are computationally expensive and lack flexibility for new prompts. Inference‑time techniques avoid retraining and can be divided further into (i) continuous noise‑optimization methods (e.g., ReNO, InitNO) that iteratively adjust the initial latent using a differentiable reward, and (ii) discrete noise‑exploration methods (e.g., ImageSelect, SeedSelect, SemI) that sample many seeds and pick the highest‑scoring result. Optimization alone is sensitive to the initial seed and can get stuck in poor local minima; exploration alone requires a large number of samples because well‑aligned seeds are sparse in the high‑dimensional latent space. Moreover, the choice of reward function is critical. Most prior work relies on a single CLIP‑based similarity score or ad‑hoc combinations, which correlate poorly with human judgments on specific compositional aspects such as attribute binding, numeracy, or spatial reasoning.

CARINOX (Category‑Aware Reward‑based Initial Noise Optimization and Exploration) addresses these shortcomings with a unified framework. First, it interleaves a broad exploration of the initial noise space with a lightweight gradient‑based refinement. A modest pool of candidate seeds (e.g., 32) is generated, each undergoes a few (e.g., 5) optimization steps guided by a reward, and the best refined seed is selected for final diffusion. This combination preserves the diversity‑seeking advantage of exploration while mitigating its inefficiency, and it reduces the risk of optimization stagnation by starting from multiple promising points. Second, CARINOX introduces a data‑driven reward selection pipeline. The authors evaluate seven candidate reward metrics (CLIPScore, HPS, PickScore, ImageReward, VQAScore, TIFA, B‑VQA) against human‑annotated alignment scores on the T2I‑CompBench++ and HRS benchmarks, measuring Pearson and Spearman correlations. For each compositional category (object presence, attribute binding, spatial relation, numeracy) the top‑correlating rewards are automatically chosen and combined via a weighted average. This principled selection ensures that the guidance signal aligns with human notions of compositional quality, avoiding the bias of any single metric.

Extensive experiments on three state‑of‑the‑art backbones—Stable Diffusion‑Turbo, SD‑XL‑Turbo, and PixArt‑α—demonstrate the effectiveness of CARINOX. On T2I‑CompBench++, average alignment scores rise from 0.39 to 0.57 for SD‑Turbo (+16 %), from 0.41 to 0.57 for SD‑XL‑Turbo (+16 %), and from 0.35 to 0.58 for PixArt‑α (+16 %). Gains are especially pronounced in texture, numeracy, and spatial reasoning sub‑tasks. On the HRS benchmark, CARINOX improves mean scores by +0.18 (SD‑Turbo), +0.16 (SD‑XL‑Turbo), and +0.23 (PixArt‑α), setting new records in creativity, style, and visual writing. Importantly, image fidelity (FID) and diversity (IS) remain comparable to the original models, indicating that compositional improvements do not come at the expense of realism. Ablation studies confirm that (1) pure optimization suffers from poor initialization, (2) pure exploration requires prohibitive sample counts, and (3) using a single, non‑correlated reward degrades performance across categories. The combined approach and reward‑selection strategy each contribute independently, and together they yield a synergistic boost.

The paper also discusses limitations: computational overhead grows linearly with the number of seeds and optimization steps, though the chosen settings keep the cost only ~1.3× that of standard exploration methods. The reward selection is limited to the evaluated metrics, so novel compositional phenomena (e.g., abstract emotional cues) may need additional reward engineering. Future work is proposed on meta‑learning reward functions and distributed multi‑GPU exploration to further reduce latency.

In summary, CARINOX provides a practical, training‑free inference‑time solution that unifies noise exploration and gradient‑based refinement while grounding the guidance in human‑aligned rewards. It substantially narrows the compositional gap of diffusion models, delivering higher alignment without sacrificing image quality or diversity.