MCIE: Multimodal LLM-Driven Complex Instruction Image Editing with Spatial Guidance

Recent advances in instruction-based image editing have shown remarkable progress. However, existing methods remain limited to relatively simple editing operations, hindering real-world applications that require complex and compositional instructions. In this work, we address these limitations from the perspectives of architectural design, data, and evaluation protocols. Specifically, we identify two key challenges in current models: insufficient instruction compliance and background inconsistency. To this end, we propose MCIE-E1, a Multimodal Large Language Model-Driven Complex Instruction Image Editing method that integrates two key modules: a spatial-aware cross-attention module and a background-consistent cross-attention module. The former enhances instruction-following capability by explicitly aligning semantic instructions with spatial regions through spatial guidance during the denoising process, while the latter preserves features in unedited regions to maintain background consistency. To enable effective training, we construct a dedicated data pipeline to mitigate the scarcity of complex instruction-based image editing datasets, combining fine-grained automatic filtering via a powerful MLLM with rigorous human validation. Finally, to comprehensively evaluate complex instruction-based image editing, we introduce CIE-Bench, a new benchmark with two new evaluation metrics. Experimental results on CIE-Bench demonstrate that MCIE-E1 consistently outperforms previous state-of-the-art methods in both quantitative and qualitative assessments, achieving a 23.96% improvement in instruction compliance.

💡 Research Summary

The paper tackles the problem of editing images according to complex, compositional textual instructions while preserving the integrity of untouched background regions. Existing diffusion‑based instruction‑guided editors perform well on simple, single‑object edits but struggle when multiple sub‑instructions and spatially distinct edits are required. The authors identify two core deficiencies: insufficient instruction compliance and background inconsistency. To address these, they introduce three major contributions.

First, they construct a new dataset called MCIE, specifically designed for complex instruction image editing. Starting from 20 K multi‑turn editing sequences in SEED‑Data‑Edit, they expand each sequence into multiple complex‑instruction samples through a systematic augmentation strategy. A powerful multimodal large language model (Qwen2.5‑VL‑72B) is employed to automatically detect conflicts among sub‑instructions, and 20 human annotators perform fine‑grained validation on instruction consistency, image quality, and scenario complexity. The final dataset comprises 90 K high‑resolution (≥1024 px) samples, each containing the full instruction, a bounding‑box that localizes the region to be edited, the source image, and the edited target image.

Second, they propose MCIE‑E1, a diffusion‑based editing model augmented with two novel cross‑attention mechanisms. An MLLM first decomposes a complex instruction into a set of sub‑instructions and their associated bounding boxes. Each sub‑instruction is encoded independently with CLIP, avoiding the interference that arises when a single embedding represents the whole prompt. The Spatial‑Aware Cross‑Attention (SA‑CA) module injects the bounding‑box mask into the attention computation, ensuring that each sub‑instruction influences only its designated region during the denoising steps. In parallel, the Background‑Consistent Cross‑Attention (BCCA) module preserves pixel‑level features of unedited regions by directly passing original visual features through a masked attention pathway. Together, these modules enable precise, region‑specific edits while keeping the rest of the image unchanged.

Third, they introduce CIE‑Bench, a benchmark for evaluating complex‑instruction editing. CIE‑Bench contains 400 carefully curated test cases and defines two metrics: Instruction Compliance, which measures how faithfully the output reflects each sub‑instruction (using a combination of automatic MLLM scoring and human judgment), and Background Consistency, which quantifies the similarity of non‑edited areas between source and result via CLIP similarity.

Extensive experiments on CIE‑Bench demonstrate that MCIE‑E1 outperforms prior state‑of‑the‑art methods such as IP2P, AnyEdit, and FOI. It achieves a 23.96 % improvement in Instruction Compliance and shows markedly higher Background Consistency scores. Qualitative examples illustrate that MCIE‑E1 correctly applies multiple edits (e.g., color changes, object additions, removals) in the right locations while leaving the surrounding scenery intact.

The work’s strength lies in its holistic approach: a high‑quality, spatially annotated dataset, a model architecture that explicitly aligns language sub‑instructions with image regions, and a benchmark that captures both compliance and preservation aspects. Limitations include reliance on bounding‑box guidance, which may be coarse compared to pixel‑level masks, and the computational overhead introduced by the MLLM during inference. Future directions could explore finer mask generation, lightweight language‑vision models for real‑time deployment, and interactive user feedback loops. Overall, the paper makes a significant step toward practical, reliable complex‑instruction image editing.