ContextGen: Contextual Layout Anchoring for Identity-Consistent Multi-Instance Generation

Multi-instance image generation (MIG) remains a significant challenge for modern diffusion models due to key limitations in achieving precise control over object layout and preserving the identity of multiple distinct subjects. To address these limitations, we introduce ContextGen, a novel Diffusion Transformer framework for multi-instance generation that is guided by both layout and reference images. Our approach integrates two key technical contributions: a Contextual Layout Anchoring (CLA) mechanism that incorporates the composite layout image into the generation context to robustly anchor the objects in their desired positions, and Identity Consistency Attention (ICA), an innovative attention mechanism that leverages contextual reference images to ensure the identity consistency of multiple instances. To address the absence of a large-scale, high-quality dataset for this task, we introduce IMIG-100K, the first dataset to provide detailed layout and identity annotations specifically designed for Multi-Instance Generation. Extensive experiments demonstrate that ContextGen sets a new state-of-the-art, outperforming existing methods especially in layout control and identity fidelity.

💡 Research Summary

ContextGen tackles the long‑standing challenge of multi‑instance image generation (MIG) by simultaneously addressing two critical shortcomings of current diffusion models: precise layout control and identity preservation across multiple distinct subjects. Built on the Diffusion Transformer (DiT) architecture, the framework introduces two novel attention‑based mechanisms—Contextual Layout Anchoring (CLA) and Identity Consistency Attention (ICA)—and a large‑scale, purpose‑built dataset called IMIG‑100K.

CLA embeds a composite layout image (either user‑provided or automatically synthesized) into the unified token sequence fed to the transformer. By constructing a global attention mask that connects layout tokens with text and latent image tokens, the layout image acts as an “anchor” that forces each generated object to occupy its prescribed spatial coordinates. This design dramatically reduces positional drift and improves spatial metrics such as mean Intersection‑over‑Union (mIoU).

ICA addresses the complementary problem of preserving fine‑grained visual identity for each instance. For every bounding box Bₙ defined in the layout, ICA creates a specialized mask that restricts attention from the tokens inside Bₙ to the corresponding reference image tokens (t_refₙ) and the shared textual context. Consequently, detailed attributes—color, texture, pose, lighting—are transferred from high‑resolution reference images to the exact locations dictated by the layout, even when objects overlap or occlude each other. Tokens outside any bounding box fall back to the CLA mask, ensuring a smooth transition from global layout to local identity injection.

A refined ternary position‑encoding scheme further disambiguates tokens from different modalities. The base latent image tokens retain (0,i,j) coordinates, while layout and reference image tokens receive unique indices of the form (1, offset_i, offset_j). This eliminates positional ambiguity in the unified attention space and enables the model to differentiate between multiple conditioning images.

To train such a system, the authors curate IMIG‑100K, a 100‑K sample dataset that provides (1) high‑resolution ground‑truth images, (2) precise layout bounding boxes, and (3) identity‑matched reference crops captured under varied viewpoints, lighting, and poses. The dataset is split into three sub‑sets: (a) Basic Instance Composition (50 K samples) generated by FLUX.1‑Dev with minimal post‑processing, (b) Complex Instance Interaction (50 K samples) featuring up to eight instances per scene with intentional occlusions and pose changes, and (c) Flexible Composition with References (10 K samples) where reference instances are independently generated and then composited using subject‑driven models, followed by rigorous identity‑consistency filtering. Text prompts are synthesized by large language models to ensure linguistic diversity.

Extensive evaluation on three benchmarks demonstrates that ContextGen sets a new state‑of‑the‑art. On COCO‑MIG, it improves instance‑level success rate by +3.3 % and spatial accuracy (mIoU) by +5.9 % over prior work. On LayoutSAM‑Eval, it achieves the highest texture and color fidelity scores, indicating superior detail preservation. Most strikingly, on LAMICBench++ it outperforms all open‑source competitors by +1.3 % average and surpasses commercial systems such as GPT‑4o in identity retention by +13.3 %. Ablation studies confirm that removing CLA degrades layout metrics, while removing ICA harms identity scores, validating the complementary nature of the two modules.

In summary, ContextGen offers a unified, attention‑driven solution that bridges the gap between layout‑conditioned and reference‑conditioned image synthesis. Its novel CLA and ICA mechanisms, together with a carefully engineered position‑encoding strategy and the IMIG‑100K dataset, enable high‑fidelity, multi‑subject generation with both accurate composition and consistent identity. The framework opens avenues for further research, including integration with richer textual conditioning, extension to 3‑D scene generation, and application to video synthesis where temporal identity consistency becomes crucial.