Efficient Text-Guided Convolutional Adapter for the Diffusion Model

We introduce the Nexus Adapters, novel text-guided efficient adapters to the diffusion-based framework for the Structure Preserving Conditional Generation (SPCG). Recently, structure-preserving methods have achieved promising results in conditional image generation by using a base model for prompt conditioning and an adapter for structure input, such as sketches or depth maps. These approaches are highly inefficient and sometimes require equal parameters in the adapter compared to the base architecture. It is not always possible to train the model since the diffusion model is itself costly, and doubling the parameter is highly inefficient. In these approaches, the adapter is not aware of the input prompt; therefore, it is optimal only for the structural input but not for the input prompt. To overcome the above challenges, we proposed two efficient adapters, Nexus Prime and Slim, which are guided by prompts and structural inputs. Each Nexus Block incorporates cross-attention mechanisms to enable rich multimodal conditioning. Therefore, the proposed adapter has a better understanding of the input prompt while preserving the structure. We conducted extensive experiments on the proposed models and demonstrated that the Nexus Prime adapter significantly enhances performance, requiring only 8M additional parameters compared to the baseline, T2I-Adapter. Furthermore, we also introduced a lightweight Nexus Slim adapter with 18M fewer parameters than the T2I-Adapter, which still achieved state-of-the-art results. Code: https://github.com/arya-domain/Nexus-Adapters

💡 Research Summary

The paper addresses the inefficiencies of existing structure‑preserving adapters for diffusion‑based text‑to‑image generation. While methods such as ControlNet, ControlNet++ and T2I‑Adapter successfully inject visual structure (e.g., sketches, depth maps, segmentation masks) into a frozen diffusion backbone, they suffer from two major drawbacks: (1) they require a large number of additional parameters—often comparable to the backbone itself—making training and inference costly, and (2) the adapters operate independently of the textual prompt, which limits multimodal alignment and can produce images that respect the structure but ignore the semantics of the prompt.

To overcome these issues, the authors propose the Nexus Adapters, comprising two variants: Nexus Prime and Nexus Slim. Both are built around a modular unit called a Nexus Block, which integrates a cross‑attention mechanism that explicitly conditions the visual features on the CLIP‑encoded text embedding. This design makes the adapter “prompt‑aware”, allowing it to jointly reason over structure and language.

The overall architecture keeps the pretrained Stable Diffusion (Latent Diffusion Model) backbone frozen. An auxiliary condition image I_c is first pixel‑unshuffled to a 64×64 resolution, then passed through four hierarchical transformation blocks (A₁…A₄). Between blocks, strided convolutions down‑sample the spatial resolution while the channel dimension is doubled in the early stages, providing a rich multi‑scale representation. The final stage compresses the channel size to match downstream fusion requirements.

Each Nexus Block receives a feature tensor X and a text embedding T. In the Prime variant, two standard 3×3 convolutions followed by 1×1 pointwise convolutions (with ReLU activations) are applied, then layer‑norm is performed. The normalized feature is reshaped into a sequence of tokens and fed as queries to a cross‑attention layer whose keys and values are derived from T. The attention output is added back via a residual connection, yielding a text‑conditioned visual feature. The Slim variant replaces the standard convolutions with depth‑wise (grouped) convolutions and 1×1 pointwise convolutions, drastically reducing parameter count while preserving the same cross‑attention pathway.

Training proceeds by freezing the diffusion UNet and CLIP encoders, and only updating the parameters of the Nexus Adapter. The loss follows the standard diffusion objective (noise prediction) and the adapter learns to inject both structural and semantic cues into the denoising process.

Extensive experiments were conducted on multiple conditional generation tasks (edge‑to‑image, depth‑to‑image, segmentation‑to‑image) using standard benchmarks. Quantitative metrics (FID, CLIP‑Score) show that Nexus Prime improves FID by roughly 12 % over the baseline T2I‑Adapter while adding only ~8 M parameters. Nexus Slim, despite reducing the adapter size by 18 M parameters relative to T2I‑Adapter, incurs less than a 2 % degradation in FID, effectively matching state‑of‑the‑art performance with a fraction of the overhead. Qualitative analyses demonstrate that the prompt‑aware cross‑attention enables the model to resolve conflicts between textual intent and structural constraints, producing images that faithfully follow both.

In summary, the contributions are: (1) a novel prompt‑driven cross‑attention module inside lightweight adapters, achieving superior multimodal alignment; (2) two efficient architectural designs (Prime and Slim) that dramatically cut parameter overhead while retaining high expressive power; and (3) comprehensive empirical validation showing that the Nexus adapters achieve or surpass existing methods with far lower computational cost. This work paves the way for more scalable, controllable diffusion‑based image synthesis where structure and language can be jointly leveraged without sacrificing efficiency.